an experimental open-source long-term memory graph for AI agents using Jev
  • Python 62%
  • TeX 26.4%
  • BibTeX Style 7.7%
  • HTML 3.5%
  • Makefile 0.4%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Rishabh Sharma 2080752aaa Paper v3 on arXiv (2609.34227): README badge and BibTeX, CITATION.cff, Space page, dataset card tag
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-09-29 09:49:50 -04:00
bench Hugging Face: v3 dataset configs, dataset card and project page (prepared; not uploaded) 2026-09-26 21:30:32 -04:00
demo Retrieval: query_relation pull (p >= 0.85, unioned) before history expansion; live retrieval regression; known-since display for undated facts 2026-09-23 04:32:24 -04:00
docs Paper v3: final revision pass (claims, readability, system names, Appendix C recomputation) 2026-09-26 17:28:38 -04:00
hf_dataset Paper v3 on arXiv (2609.34227): README badge and BibTeX, CITATION.cff, Space page, dataset card tag 2026-09-29 09:49:50 -04:00
hf_space Paper v3 on arXiv (2609.34227): README badge and BibTeX, CITATION.cff, Space page, dataset card tag 2026-09-29 09:49:50 -04:00
paper Merge v1.1 (Jev-Mem wording, tag v1.1-preprint) into v2 2026-09-24 22:07:13 -04:00
paper_v3 Paper v3 on arXiv (2609.34227): README badge and BibTeX, CITATION.cff, Space page, dataset card tag 2026-09-29 09:49:50 -04:00
src/engram POST-HOC EXPLORATORY: T0R-wide (own ledger bench/results/v3_posthoc, capped at $1.00 Jev and $0.50 OpenAI; not a registered test). 150-turn cosine shortlist, Jev relevance on every shortlisted turn (30 per request), top k by Jev score with no 0.5 cut; k=47 matched to Jev-Mem k=40 (2,000 vs 1,987 tokens). 81.5% on the 778 scored questions (Jev-Mem k=40 80.3%, engram v2 k=20 82.4%, mem0 k=20 78.7%, full context 78.3%); read $0.00089 per query, live p50 720 ms; shortlist recall all 91.9%, any 97.9%, top 47 drops all shortlisted evidence for 0.9%. New read-path options retrieval_shortlist and rerank_keep default to the registered behaviour. Blinded H1 human-audit sheet (142 discordant questions, 284 rows, shuffled with seed 0) with its key in a separate file 2026-09-26 12:47:55 -04:00
tests Paper v3: engram attribution and §3.1 setup and notation 2026-09-26 20:10:25 -04:00
.env.example Release housekeeping: .env.example, make reproduce-v3, README for the v3 paper, CITATION.cff 2026-09-26 20:49:22 -04:00
.gitignore docs/JEVMEM_COMPARISON.md: code-level comparison of engram and Jev-Mem (commit 81574eb), the v1.1 paper's claims checked against its code, and its feasibility and cost as an exploratory baseline; bench/external/ (read-only clones) is gitignored 2026-09-24 22:05:29 -04:00
.python-version Step 2: models, store, question schemas, mock backend, offline tests 2026-09-23 03:32:46 -04:00
CITATION.cff Paper v3 on arXiv (2609.34227): README badge and BibTeX, CITATION.cff, Space page, dataset card tag 2026-09-29 09:49:50 -04:00
LICENSE Step 2: models, store, question schemas, mock backend, offline tests 2026-09-23 03:32:46 -04:00
Makefile Hugging Face: v3 dataset configs, dataset card and project page (prepared; not uploaded) 2026-09-26 21:30:32 -04:00
pyproject.toml Lean extraction arms L0-L2 (dev, conv-26): conversation units instead of LLM-extracted facts, behind flags (extraction=lean). L0 raw turns '[date] speaker: text' with cosine retrieval and no Jev; L1 spaCy sentences with relative dates resolved in code against the session date and written into the unit; L2 + a Jev worth_sentence Noul per sentence, only units above 0.5 indexed (the rest stored without an embedding). pipeline/lean.py (LeanWriter subclasses WritePipeline), render=lean, ledger stage 'lean' with cumulative caps ($2 OpenAI, $1 Jev). conv-26, k=3 / k=20: L0 63.2% / 82.2%, L1 55.3% / 70.4%, L2 63.8% / 75.0%; write cost per 1k messages $0.001, $0.001, $0.028 (Jev); lean spend so far OpenAI $0.145, Jev $0.012. spaCy and en_core_web_sm pinned in the bench extra 2026-09-25 21:06:53 -04:00
README.md Paper v3 on arXiv (2609.34227): README badge and BibTeX, CITATION.cff, Space page, dataset card tag 2026-09-29 09:49:50 -04:00
sample.txt Step 5: temporal_status question, extraction and write pipeline, sample ingest; throughput bench; 50-pair regression test 2026-09-23 04:08:04 -04:00
uv.lock Lean extraction arms L0-L2 (dev, conv-26): conversation units instead of LLM-extracted facts, behind flags (extraction=lean). L0 raw turns '[date] speaker: text' with cosine retrieval and no Jev; L1 spaCy sentences with relative dates resolved in code against the session date and written into the unit; L2 + a Jev worth_sentence Noul per sentence, only units above 0.5 indexed (the rest stored without an embedding). pipeline/lean.py (LeanWriter subclasses WritePipeline), render=lean, ledger stage 'lean' with cumulative caps ($2 OpenAI, $1 Jev). conv-26, k=3 / k=20: L0 63.2% / 82.2%, L1 55.3% / 70.4%, L2 63.8% / 75.0%; write cost per 1k messages $0.001, $0.001, $0.028 (Jev); lean spend so far OpenAI $0.145, Jev $0.012. spaCy and en_core_web_sm pinned in the bench extra 2026-09-25 21:06:53 -04:00

engram

arXiv DOI

Current paper: When Does Selection Replace Extraction? A Pre-Registered Test of Agent Memory with a Typed Decision Model (Rishabh Sharma and Rishika Lall, 2026). arXiv: 2609.34227; DOI: 10.5281/zenodo.22985242 (release tag paper-v3-preprint-r3). PDF: paper_v3/when-does-selection-replace-extraction.pdf.

Does conversational memory need LLM-extracted facts, or is it enough to select the right raw turns? Published results disagree. This study tests the question under a pre-registered plan, on LoCoMo conversations never used for development and on LongMemEval. Raw turns selected by one request to Jev, TypeSafe's typed decision model ("Turns + Jev"), are compared with engram v2, an LLM-extraction memory we built earlier, at matched context. Within the study, the context budget decides the answer: when the answer model reads few items, selection matters and raw turns hold their own; when it reads many, extraction is more accurate.

Headline results (all from paper_v3/numbers.json, which records the result file behind every number):

Result Value
H1 (registered): Turns + Jev at k=6 (265 tokens per question) vs engram v2 at k=3 (251), 778 held-out LoCoMo questions 77.0% vs 77.5%; difference −0.5 points, one-sided 95% bound −3.0 against a −5-point margin: non-inferior
H1 under blind human grading of the discordant questions difference −1.7 to −2.6 points; still non-inferior (worst bound −4.7)
H1 with a second answer model (Llama 3.3 70B) still non-inferior (bound −2.9)
Write cost raw turns are 3,061× cheaper to write than engram v2
Rerank gain over cosine similarity at k=3 (3 of 30 candidates read) +17.4 points on LoCoMo, +9.1 on LongMemEval
Same at k=20 (generous budget) +1.5 and +1.1; engram v2 at k=20 is the most accurate system measured (82.4%)
Jev against a gpt-4o-mini listwise reranker at matched context (S4) non-inferior (bound −2.0) at about a third of the latency
Reranking and abstention reranking lowers correct abstention (54.1% vs 63.6% at k=3)

The paper labels every result as registered, exploratory or post-hoc; the cross-paper reading of the budget result is an interpretation, not a tested claim.

Status: research code. It reproduces the papers; it is not a maintained library, and its thresholds are tuned against one decision-model version (jev-1.13.0).

Reproduce this paper

Requires uv, Python 3.12, and Docker (or a local pdflatex with pdfcrop).

make reproduce-v3

This rebuilds paper_v3/numbers.json, every table and figure, and the PDF from the committed result files in bench/results/v3/ and bench/results/v3_posthoc/, then runs paper_v3/check.py, which fails on any number in the paper that does not trace to a result file. It makes no model or API calls (the API keys are removed from its environment). It downloads LoCoMo and LongMemEval at pinned revisions, because the numbers count their categories and question ids. tests/test_paper_v3_thresholds.py checks that the constants the paper states equal the code's.

The runs themselves are recorded in docs/V3_PLAN.md and docs/V3_PROGRESS.md; re-running them calls the APIs (OpenAI, TypeSafe, OpenRouter) with a spend guard in bench/run.py.

Secret scanners (gitleaks, trufflehog, GitHub push protection) flag a few strings in the LongMemEval result files, such as a GitHub token; they come verbatim from the public LongMemEval dataset, pasted there by its users, and are not our credentials.

A warning for anyone benchmarking mem0. mem0 2.1.0 silently sends its OpenAI calls to OpenRouter whenever OPENROUTER_API_KEY is set in the environment. It happened in this study (paper, Appendix G); bench/run.py now removes the key from mem0's process.

Project history

The work ran in three phases. Each has its own tag, pre-registration and paper.

v1: the preprint. Typed Decisions in Agent Memory: Where They Help, Where They Don't, and What It Costs (paper/typed-decisions-in-agent-memory.pdf). engram, an LLM-extraction memory in which a typed decision model makes every later decision about the facts, measured against mem0 2.1.0 on LoCoMo. DOI (all versions) 10.5281/zenodo.22941757; v1 10.5281/zenodo.22941758, v1.1 10.5281/zenodo.22948964. Tags v1-preprint, v1.1-preprint, v1-article. make reproduce-dev replays its Table 1 from the shipped call cache at no cost; make -C paper paper rebuilds it. Where the v3 paper revises a v1 claim, the v3 paper is the current statement.

v2: paused. A pre-registered confirmatory study of engram v2 (session-dated extraction and a same-attribute gate), plan docs/V2_PLAN.md: DOI (all versions) 10.5281/zenodo.22948854, at the v2-frozen tag 10.5281/zenodo.22953496. Paused on 2026-09-26 before any held-out run, for the v3 design; no paper. engram v2 at v2-frozen is the comparator of the v3 paper.

v3: the current paper. Plan docs/V3_PLAN.md, deposited before any run (10.5281/zenodo.22970745, tag v3-frozen), and its amendment, deposited before any primary-test result (10.5281/zenodo.22977848, tag v3-amended). Paper in paper_v3/, arXiv 2609.34227, DOI 10.5281/zenodo.22985242; release tag paper-v3-preprint-r3. The earlier tags paper-v3-preprint, -r1 and -r2 hold the same study and results; r1 to r3 revise only the text, references and authorship.

Repository map

src/engram/          the library: store, write pipeline, retrieval, decision backends (Jev, LLM), CLI, server
bench/               experiments: bench/run.py (every arm), reports, the v3 scripts (bench/v3_*.py)
bench/results/v3/    v3 result files, ledgers and the human audit (human_audit/); v3_posthoc/ holds the post-hoc run
bench/results/       v1 and v2 result files
paper_v3/            v3 paper: main.src.md, make_numbers.py, check.py, figures.py and diagram.py, build scripts, PDF
paper/               v1 paper and its build
docs/                V3_PLAN.md, V3_OUTCOMES.md, V3_PROGRESS.md, V3_LITERATURE.md; V2_PLAN.md; v1 notes
tests/               unit tests, the plan guards (test_v2_plan.py, test_v3_plan.py) and the paper-threshold checks
hf_dataset/          the released dataset (ris3abh-11/engram-eval on the Hugging Face Hub)
hf_space/            the project page
demo/                the page served by `engram serve`

Citation

See CITATION.cff. This paper:

@misc{sharma2026selection,
  author        = {Sharma, Rishabh and Lall, Rishika},
  title         = {When Does Selection Replace Extraction? A Pre-Registered Test of Agent Memory with a Typed Decision Model},
  year          = {2026},
  eprint        = {2609.34227},
  archivePrefix = {arXiv},
  doi           = {10.5281/zenodo.22985242},
  url           = {https://arxiv.org/abs/2609.34227}
}

The v1 preprint:

@misc{sharma2026typed,
  author    = {Sharma, Rishabh},
  title     = {Typed Decisions in Agent Memory: Where They Help, Where They Don't, and What It Costs},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.22948964},
  note      = {Version 1.1. Version 1: doi:10.5281/zenodo.22941758}
}

Licences

  • Code: MIT (LICENSE).
  • LoCoMo-derived data (the excerpts in bench/slices/, the prompts recorded in bench/cache/dev_calls.sqlite, and the LoCoMo turns and questions inside result files): CC BY-NC 4.0, LoCoMo's licence (snap-research/locomo). LoCoMo itself is downloaded, not redistributed.
  • LongMemEval-derived data: MIT, LongMemEval's licence (xiaowu0162/longmemeval-cleaned).
  • mem0's prompts in src/engram/llm/prompts_mem0.py: Apache 2.0 (© mem0.ai).