Most continual-learning studies target short sequences of tasks; this paper asks whether composing mechanisms that protect data, function, and weights can scale memorization to much longer horizons. The core insight is that different forgetting modes are complementary and that combining anchors with low-rank allocation (LoRA) yields far larger retention gains than any single method.
Key Findings
- Composition matters: combining data anchor (generative replay), function anchor (self-distillation), weight anchor (synaptic-style regularization) and merged LoRA produces the strongest retention across datasets.
- Large numeric gains: naive sequential fine-tuning yields ~1.2% final retention; the best compositional method raises this to 34.9% (≈28× improvement). The best individual mechanism reaches only ~8.1%.
- Synergy: data anchor and merged LoRA provide the largest average gains and interact super-additively across all three testbeds.
- Extended memory half-life: task accuracy half-life increases from ~1–2 tasks under naive fine-tuning to roughly 19–44 tasks under top compositions, showing multi-order-of-magnitude shifts in forgetting timescales.
- Robustness tests: evaluated on three 100-task memorization datasets (Symbol-QA, LLM-QA, Real-QA) and selected configurations via task-level successive halving to manage the large experimental space.
Who it's for and tradeoffs
Great fit if you study continual learning, memory for LMs, or deployment scenarios where models receive many sequential updates without storing past data. The work shows practical mechanism combinations rather than a single silver bullet. Look elsewhere if you need a turnkey production system—compositional methods increase complexity (more losses, tuning, and LoRA management) and do not fully eliminate forgetting.
Methods snapshot
The paper organizes designs along two axes: (1) anchors that specify what the update should preserve (data replay, self-distillation on current inputs, and weight regularizers) and (2) low-rank allocation rules that determine where updates are stored (LoRA variants, merged LoRA). Experiments use task-level successive halving to search ~90 configurations across three distinct 100-task datasets, measuring final retention and temporal accuracy matrices to quantify forgetting.