Memorilla Open Source: 16 Tokens for Frozen Qwen3-8B

A memory module from Stanford University compresses entire collections into 16 tokens injected into a frozen LLM. Gains are observed on FactKG and other test sets, with unfavorable cases like PubMedQA. The code and checkpoints are released under Apache 2.0, but execution targets high-end GPUs, and the figures are self-reported.

Heterogeneous Results and Self-Reported Figures

Memorilla ranks lower on three benchmarks. On PubMedQA, the module achieves 59.00% while LLMLingua-2 shows 69.00%. This result is also among the lowest recorded, with 59.00% reported compared to 69.00% for LLMLingua-2. The team explains that the summaries of PubMedQA total only 324 tokens, leaving little room for compression. MemoRAG maintains an advantage over PersonalizationV4 (67.88% vs. 66.65%) and on NarrativeQA in ROUGE-L (20.51 vs. 19.31), although it uses approximately 690 times and 50 times more document tokens, respectively. All these scores and comparisons are provided by the research team, which specifies that it evaluated the baselines on the same frozen Qwen3-8B.

Code Released Under Apache 2.0 and Available Resources

The project is accessible under the Apache 2.0 license, with the code, training recipes, and checkpoints uploaded, including checkpoints of 194 million parameters hosted on Hugging Face. Installation is done via uv on Linux before loading with MemoryVLLM.from_pretrained, with an initial launch downloading about 25 GB. PersonalizationV4, a synthetic corpus of 149 users and 200 conversations each, is released under the CC BY 4.0 license.

What Limited Latent Memory of 16 Tokens Changes

Memorilla scans a complete document storage and delivers a compact summary of 16 tokens to the frozen LLM, a fixed size regardless of the collection. By transforming the collection into 16 vectors, the goal is to replace the inclusion of entire documents in context for a frozen Qwen3-8B. On FactKG, they report 84.33% compared to 80.43% for the full context, with 108 times fewer document tokens. On an H100, the system responds in 0.74 seconds with 512 documents (about 128K tokens), while the full context stretches from 0.75 seconds to 19.9 seconds, making it 27 times slower in this case; the memory module itself takes about 2 ms.

The Three-Step Pipeline and Training

The processing follows three phases. First, a frozen Qwen3-Embedding-4B encodes each document once, without re-encoding previous ones during additions. Subsequently, sixteen learned slots connect to the question and memory via two layers of cross-attention with eight heads. In the next step, these slots are transformed into the input embedding space of Qwen3-8B and added as a prefix to the query. Only the memory module is trained, comprising 194,123,776 parameters related to the 8B decoder. The training is organized into three phases: approximately 1.98 million examples of paraphrase reconstruction from Wikipedia, followed by about 1.34 million examples drawn from ten question-answering and summarization datasets, and finally, fine-tuning performed on eight benchmarks. The entire recipe takes about 19 hours on four H100 GPUs.

Quantitative Summary: Averages, Winners, and Losers by Cases

Beyond FactKG, Memorilla achieves 44.60% on PersonaMem-v2 compared to 30.90% for xRAG, and in another reported comparison, 44.60% against 27.92% for the full context. On TriviaQA, the exact match score of 57.00 slightly surpasses MemoRAG at 56.60. On LaMP-4 and LaMP-7, scores of 18.15 and 48.37 in ROUGE-L are recorded, the best scores. On average across eight benchmarks, Memorilla shows 49.68 compared to 41.92 for xRAG, 40.75 for LLMLingua-2, and 38.02 for Mem0. Comparative tables indicate an average geometric consumption per query of 16 document tokens for Memorilla, 78 for MemoRAG, 42 for Mem0, and 121 for xRAG. The accuracies on FactKG are distributed at 84.33, 80.38, 72.79, and 78.98 respectively; on PersonaMem-v2 at 44.60, 28.76, 27.48, and 30.90; on PubMedQA at 59.00, 67.00, 24.00, and 36.00; and on NarrativeQA in ROUGE-L at 19.31, 20.51, 13.80, and 16.92. Compression peaks when 16 tokens replace 67,385 document tokens on NarrativeQA (reported reduction of 4,212x). In agent context, eight memory tokens raise the average success of a Qwen3-4B TextWorld agent from 18.6% to 24.2%.

Execution, Hardware Constraints, and Acceleration Options

The system targets Linux, CUDA-compatible GPUs, and Python 3.11. Quick startup requires an 80 GB GPU, with recipes conducted on an 80 GB H100, and no quantified variant is provided. In execution, a latency of 0.74 seconds is noted with 512 documents on an H100, while the full context rises to 19.9 seconds. An optional memory cache pre-computes attention keys and values: on a B200, a query lasts 8.1 ms at 1M documents and 33 ms at 4M, and drops to 5.0 ms at 1M with torch.compile; the cache requires 80 GiB at 4M documents. For integration, installation is done via uv and checkpoints are loaded with MemoryVLLM.from_pretrained; each checkpoint must be associated with the corresponding decoder, and custom data is presented in parquet files with collection_id, question, answer, and documents.