Why this matters
Large language models are increasingly trained or adapted on many tasks in stages, but SFT-style pipelines often suffer catastrophic interference across tasks while RL-based adaptation appears to accumulate capabilities. The core insight: RL produces much smaller, sparser, and near-orthogonal parameter updates across tasks, which empirically and theoretically reduces gradient interference and enables stable multi-stage multi-task training.
Key Findings
-
Parameter-scale and sparsity: RL update vectors have a much smaller average L2 norm (~3×10^-2) versus SFT (~7.4), and only ~20% of parameters exceed a small magnitude threshold in RL compared to ~93% in SFT — implying RL changes are minimal and localized. This reduces destructive overwriting across tasks.
-
Directional decorrelation: Pairwise cosine similarity of parameter updates across RL tasks is effectively negligible (on the order of 10^-5), whereas SFT updates show high similarity (10^-1 to 1.0) and occasional opposing directions. Near-orthogonality explains why RL updates interfere far less.
-
Theoretical view of interference: SFT interference scales with absolute gradient norms (norm-limited), so large overlapping updates cause catastrophic forgetting; RL interference is variance-limited due to advantage normalization and on-policy optimization, yielding a small bound on cross-task interference and hence robustness in sequential training.
-
Empirical consequences & method: Multi-stage SFT can produce large average performance drops (reported average decline ~23%), while RL yields cumulative gains and robustness (reported gains ~25% in some settings). Building on these mechanisms, the paper proposes Parallel-RL, a training paradigm that decouples tasks to improve efficiency and flexibility.
Who it's for and tradeoffs
Great fit if you are researching or engineering multi-task adaptation of LLMs and need to understand failure modes of SFT versus RL, or if you want a principled approach (Parallel-RL) to reduce interference in staged pipelines. Look elsewhere if your environment cannot afford RL's practical costs: on-policy RL methods can be sample- and compute-inefficient, require careful reward/advantage design, and may need more infrastructure than straightforward SFT. The paper offers concrete empirical metrics (norms, sparsity, cosine similarities) and a theoretical framing to decide which approach suits your constraints.