Most fixes or skills provided through context disappear once that context is removed. This paper reframes that problem as Memory Consolidation: internalize context-conditioned improvements into the model itself while preserving unrelated capabilities. The central insight is to anticipate how context will be used (via prospective queries) and distill the resulting behavior into compact, mergeable parameter updates so the model retains the benefit without needing persistent context.
Key Findings
- Proposes a consolidation objective that unifies context-specified editing, skill learning, and behavioral steering under a common formalization; consolidation is performed by distilling context-conditioned outputs into low-rank or structural feed-forward updates. This makes updates mergeable and compatible with model maintenance workflows.
- Evaluated on a five-task-family testbed (single, batch, sequential updates) across dense and mixture-of-experts backbones from 4B to 120B, including text-only and multimodal settings. Cortex lies on the empirical consolidation–retention Pareto frontier in most settings, demonstrating favorable trade-offs compared to baselines.
- Quantitative example: after 50 sequential factual edits on a 14B model, the Cortex-Structural variant recovers 88.9% of the context-induced gain while retaining 99.8% of original non-target performance — illustrating strong consolidation with minimal collateral degradation.
- Introduces practical mechanisms: prospective query generation to anticipate downstream uses of context, novelty gating to avoid redundant writes, and options for low-rank vs. structural updates to trade capacity, mergeability, and compute cost.
Who it's for and trade-offs
Great fit if you need to make context-driven changes persistent in deployed LLMs (factual edits, new skills, policy steering) without keeping expensive retrieval or prompt scaffolding. The method is especially relevant for teams managing model updates at scale who value mergeable parameter edits and controlled retention of legacy capabilities.
Look elsewhere if you require guaranteed lossless preservation of all behaviors (the method favors empirical Pareto-optimal trade-offs, not absolute invariance), if your deployment forbids any parameter modification, or if you only need ephemeral, session-scoped context where retrieval-based approaches suffice.
Where it sits methodologically
Cortex complements retrieval and fine-tuning by occupying the middle ground: it anticipates likely uses of context (like retrieval would) but internalizes behavior via compact parametric edits (like fine-tuning) while emphasizing mergeability and minimal non-target impact. Benchmarks and ablations in the paper quantify consolidation vs. forgetting dynamics and compare low-rank and structural update families across model sizes and modalities.