- Python 58.9%
- Rust 41.1%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
| docs | ||
| drawio | ||
| figures | ||
| mcp | ||
| rust | ||
| tests | ||
| .gitignore | ||
| BENCHMARK.md | ||
| engram_demo_v1.py | ||
| Engram_paper.pdf | ||
| engram_pq.py | ||
| LICENSE | ||
| README.md | ||
conditional memory on the p/q · BFP-64 · MEM|8 substrate
The unified kernel (kernel8 · ayeOS · cordis · mem8 · GAIA-MLX-QUANT) treats precision as a property of the information's requirements, not a fixed tax paid by every operation. In DeepSeek's Engram architecture, each $N$-gram lookup head has dimension D_{\text{head}} = 512 / 8 = 64 — the exact block size of our Omni BFP-64 representation (64 × i8 mantissas + 1 × i8 shared dyadic exponent = 65 bytes/row). Retrieving 16 heads (16 × 64 = 1024) from a BFP-64 table yields a pre-quantized 16-block vector with zero runtime activation quantization overhead, ready for 16-wide i16 SIMD ternary accumulation (|S_{64}| \le 64 \times 127 = 8,128 \le 32,767), i64 cross-head integer exponent alignment, and a 100% zero-float p/q ("Counting Divided") forward step.
DeepSeek Engram — 8b-is Sovereign Edition
8b-is/Engram is the sovereign p/q + BFP-64 + MEM|8 edition of deepseek-ai/Engram (Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models), aligned with 8b-is/deepsiper-enthea and 8b-is/alexiai.
Fork Enhancements (8b-is/Engram)
- Exact 64-Dim Omni
BFP-64Head Table (3.94×Memory Compression): Stores each 64-dimensional $N$-gram head row as64 × i8mantissas in[-127, 127]+1 × i8shared power-of-two exponent (65 bytes/rowvs256 bytes/rowFP32). Shrinks the DeepSeek Engram demo table (10.34Mrows) from2.65 GBto0.67 GBand the DeepSeek-V4.1-Flash table (384.0Mrows) from98.31 GBto24.96 GB. - Zero-Quantization
\mathcal{O}(1)Gather +i16 -> i64Integer-Aligned Projection: Because retrieved Engram heads are already quantized into 64-elementBFP-64blocks in the table, all 5 downstream projections (value_projandkey_projs[0..hc_mult]) skip activation quantization completely, accumulating 64-element blocks ini16(|S_64| <= 8,128) and aligning cross-head exponents in purei64integer arithmetic. - 100% Zero-Float Pure-Integer
p/q("Counting Divided") Engram Layer Step (6.92×Speedup):engram_step_pq_no_floats_intoexecutes head gather, 5 linear projections (2.62Mweights), Fused BilinearRMSNormgating (via exact integer Newton-Raphsonisqrt_u128), signed-sqrt, rational Padé[2/2]sigmoid, and ShortConv SiLU in pure integerQ16/Q20arithmetic with zero IEEE-754 floating-point instructions and0.010%relative error (rel_err = 0.00010). - MEM|8 Adaptive Precision (
BFP-i8 -> BFP-i16Promotion): Frequent syntactic $N$-gram heads stay in compactBFP-i8(65 B/row,i16accumulator), while high-salience or outlier factual $N$-gram heads whose reconstruction error exceeds\tauautomatically promote toBFP-i16(129 B/row,i32accumulator), cutting projection error by3.8×(rel_err = 0.00083). - Lemire
p/q(q = 2^{64}) Fast Reciprocal Prime Hashing: Replaces hardware% primedivision across all(ngram, head)prime buckets with precomputed 64-bit dyadic reciprocals (ceil(2^64 / prime)), achieving 100% bit-exact bucket equivalence. - Community Upstream PR Integrations (
#13,#14,#15,#27):- PR #13: Multi-layer
_HASH_MAPPING_CACHE(get_or_create_hash_mapping) avoids rebuildingCompressedTokenizerand prime tables across layers. - PR #14: Vectorized hyper-connection gating (
torch.einsum("bshd,bshd->bsh", ...)). - PR #15: Bounds-safe
CompressedTokenizer._compress, device-aware CPU/GPU tensor placement, and a zero-networkSovereignOfflineTokenizer. - PR #27: Pluggable
embedding_factoryhook ("dense","bfp64","mem8_adaptive", or custom callable) + offlineunittestsuite (tests/test_engram_demo_v1.py).
- PR #13: Multi-layer
- Dual Zero-Dependency Substrates (
rust/+engram_pq.py): Includes both a#![no_std]-first,#![forbid(unsafe_code)], zero-dependency Rust crate (rust/) and a 100% Python 3 stdlib engine (engram_pq.py) alongside the PyTorch implementation (engram_demo_v1.py).
Empirical Benchmark Summary
Full reproduction details and commands are documented in BENCHMARK.md.
| Benchmark Stage | Baseline (Dense FP64 / % prime) |
8b-is Sovereign (p/q + BFP-64) |
Speedup / Compression | Relative Error |
|---|---|---|---|---|
Multi-Head Prime Hashing (16,384 hashes) |
20.90 µs (% prime) |
15.00 µs (Lemire p/q, q=2^{64}) |
1.39× – 3.25× faster (x86-64 Linux) |
Bit-Exact (0) |
Engram Demo Table (10.34M × 64) |
2.65 GB (FP32) |
0.67 GB (BFP-64, 65 B/row) |
3.94× smaller |
0.31% |
DeepSeek-V4.1 Table (384.0M × 64) |
98.31 GB (FP32) |
24.96 GB (BFP-64, 65 B/row) |
3.94× smaller |
0.31% |
Isolated Head Projection (1024 -> 512) |
208.70 µs |
35.65 µs (i16 -> i64 -> Q16) |
5.85× faster |
0.00317 |
MEM|8 Adaptive Projection (1024 -> 512) |
208.70 µs |
48.05 µs (BFP-i8 -> BFP-i16) |
4.34× faster |
0.00083 |
Full Engram Layer Step (2.62M weights + Gate + Conv) |
1,282.36 µs |
190.11 µs (Int-Aligned BFP-64) |
6.75× faster |
0.00021 |
100% Zero-Float p/q Engram Step (Q16) |
1,282.36 µs |
185.41 µs (Zero Float Ops!) |
6.92× faster |
0.00010 |
Hashing speedups are platform-dependent: on Apple Silicon the
% primepath wins (~0.9×forp/q). Bucket equivalence and the projection/step error numbers are platform-independent; see BENCHMARK.md.
1. Introduction
This repository contains the implementation for the paper: Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models.
Abstract: While Mixture-of-Experts (MoE) scales capacity via conditional computation, Transformers lack a native primitive for knowledge lookup. To address this, we explore conditional memory as a complementary sparsity axis, instantiated via Engram, a module that modernizes classic $N$-gram embeddings for
\mathcal{O}(1)lookup.
Key Contributions:
- Sparsity Allocation: We formulate the trade-off between neural computation (MoE) and static memory (Engram), identifying a U-shaped scaling law that guides optimal capacity allocation.
- Empirical Verification: Under strict iso-parameter and iso-FLOPs constraints, the Engram-27B model demonstrates consistent improvements over MoE baselines across knowledge, reasoning, code and math domains.
- Mechanistic Analysis: Our analysis suggests that Engram relieves early layers from static pattern reconstruction, potentially preserving effective depth for complex reasoning.
- System Efficiency: The module employs deterministic addressing, enabling the offloading of massive embedding tables to host memory with minimal inference overhead.
2. Architecture
The Engram module augments the backbone by retrieving static $N$-gram memory and fusing it with dynamic hidden states. The architecture is shown below (drawio provided):
3. Evaluation
Scaling Law
Large Scale Pre-training
Long-context Training
4. Case Study of Engram
5. Quick Start
A. Sovereign Zero-Dependency Rust (rust/)
cargo test --manifest-path rust/Cargo.toml --release
cargo run --manifest-path rust/Cargo.toml --release --bin bench
B. Sovereign Zero-Dependency Python 3 (engram_pq.py & engram_demo_v1.py)
Works out-of-the-box on bare Python 3 standard library (no network calls or third-party packages required) and automatically uses PyTorch when torch is installed:
python3 engram_demo_v1.py
python3 -m unittest discover -s tests -v
C. Full PyTorch + HuggingFace Environment
pip install torch numpy transformers sympy
python3 engram_demo_v1.py
6. License
The use of Engram models is subject to the Model License.