Adding knowledge to an LLM usually forces a choice: retrieve documents at inference time, or pay a large offline cost to teach the model a parametric memory. This work challenges the assumption that memory targets must be built for every token: a single predicted memory can supervise a whole chunk, then be selectively used by each token inside the model.
Key Findings
- Chunk-level supervision cuts retrieval queries by 32× and retrieved items by 2,048× compared with the token-level parametric-memory approaches discussed in the paper, making offline memory construction substantially lighter.
- Gated cross-attention injects predicted memory into intermediate hidden states rather than only blending it into the final output. Later decoder layers can therefore use memory as they continue processing context.
- On six QA tasks, the method exceeds the strongest baseline by 12.71% on Mistral-7B-v0.3 and 11.19% on Qwen3-8B-Base. Domain adaptation also reports average gains of 6.51 and 4.38 points on Qwen3-4B-Base and Qwen3-8B-Base, respectively.
How It Works
A causal predictor uses the preceding chunk to produce a set of memory vectors, trained against embeddings of corpus chunks retrieved offline. Those vectors are shared across predictions in the next chunk, while gated cross-attention lets each token use them according to its context. The backbone stays frozen; the added predictor and fusion modules are trained end to end.
Who It’s For—and the Trade-Off
A good fit for researchers seeking to add corpus-derived knowledge without per-request retrieval, especially when inference latency or retrieval infrastructure is a concern. It is less suitable when knowledge must be freshly updated or individually traceable to retrieved documents: the approach shifts retrieval to offline training and relies on learned memory, and the reported results cover specific models and benchmarks.