RAP: Retrieval-Augmented Planning with Contextual Memory for Multimodal LLM Agents
  • Python 95.7%
  • Rust 4.3%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
dependabot[bot] 4b495b1f8a
build(deps): bump werkzeug from 2.0.3 to 3.1.9 in /webshop (#9)
Bumps [werkzeug](https://github.com/pallets/werkzeug) from 2.0.3 to 3.1.9.
- [Release notes](https://github.com/pallets/werkzeug/releases)
- [Changelog](https://github.com/pallets/werkzeug/blob/main/CHANGES.rst)
- [Commits](https://github.com/pallets/werkzeug/compare/2.0.3...3.1.9)

---
updated-dependencies:
- dependency-name: werkzeug
  dependency-version: 3.1.9
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-11-09 02:47:45 -05:00
alfworld fix: keep small memory banks from drawing a blank 2026-10-28 07:56:53 +09:00
figure First commit 2024-06-19 09:16:23 +08:00
tests fix: keep small memory banks from drawing a blank 2026-10-28 07:56:53 +09:00
webshop build(deps): bump werkzeug from 2.0.3 to 3.1.9 in /webshop (#9) 2026-11-09 02:47:45 -05:00
.gitignore First commit 2024-06-19 09:16:23 +08:00
LICENSE Update LICENSE 2024-06-19 13:03:46 +09:00
rap_mem8_bfp64_pq.py docs: give memory claims a reality byte 2026-10-28 08:55:00 +09:00
README.md docs: give memory claims a reality byte 2026-10-28 08:55:00 +09:00

RAP: Retrieval-Augmented Planning with Contextual Memory for Multimodal LLM Agents

This repository is the implementation of RAP.

RAP figure

Get Started

Please refer to the following README's for each benchmark.

Standalone BFP-64 memory prototype

rap_mem8_bfp64_pq.py is an experimental, process-local memory bank. It accepts float embeddings, quantizes them in blocks of up to 64 values, and ranks episodes using approximate integer cosine scores. The ALFWorld and WebShop scripts use their own sentence-transformer retrieval paths and do not import this prototype. It provides no persistence, update or deletion API, and its stored reward_pq field does not scale retrieval scores.

raw_block_bytes and compression_ratio_pq() estimate a hypothetical packed payload: one byte per mantissa plus one byte per block exponent. For 384 values, that is 390 bytes versus 1536 bytes of FP32 data. The implementation stores Python integer tuples and cached norm metadata; it does not serialize a 390-byte row or measure actual memory savings. Query preparation uses floating-point operations and an integer square root; only scoring of prepared rows is integer arithmetic.

Run the prototype's standard-library tests with:

python -m unittest discover -s tests -p 'test_rap_mem8_bfp64_pq.py'

With NumPy and PyTorch installed from the benchmark dependencies, run all focused Python tests with:

python -m unittest discover -s tests -p 'test_*.py'
python -O -m unittest discover -s tests -p 'test_*.py'

The benchmark retrieval tests extract only the relevant function definitions, using synthetic embeddings and a test cosine function. They skip when NumPy or PyTorch is unavailable. They do not import benchmark startup, download models, contact a model API, or validate full benchmark outcomes. The standalone Rust files under tests/ are not part of this Python suite; no Cargo project or GPU integration is defined here.

Citation

If you find RAP helpful in your research, please consider citing.

@misc{kagaya2024rap,
      title={RAP: Retrieval-Augmented Planning with Contextual Memory for Multimodal LLM Agents}, 
      author={Tomoyuki Kagaya and Thong Jing Yuan and Yuxuan Lou and Jayashree Karlekar and Sugiri Pranata and Akira Kinose and Koki Oguri and Felix Wick and Yang You},
      year={2024},
      eprint={2402.03610},
      archivePrefix={arXiv},
      primaryClass={cs.LG}
}

License

MIT license