A Local Perturbation Theory for Cross-Domain Interference and Recovery in Multi-Domain RL

When training large language models (LLMs) through reinforcement learning (RL) to master different skills—like math, coding, or creative writing—one skill often breaks the others. This is a pervasive issue in multi-domain post-training. As you refine the model for a new task, you watch the performance on previously mastered domains degrade.

Existing wisdom suggests this is due to "catastrophic forgetting" or massive "gradient conflict" (where updates for different tasks pull parameters in opposing directions). However, this paper argues that the problem is more subtle. Even when the model's global gradients appear nearly orthogonal (meaning they are not fighting in any obvious way), substantial interference still occurs. The researchers propose that this damage is not a global rewrite of the model. Instead, it is a localized strike on specific "shared active routes" (the paths of neurons activated during inference) that the model uses.

The Problem

The status quo assumes that to avoid interference, we must train on all domains simultaneously via joint training. The logic is that if gradients from different domains point in opposite directions, they will cancel each other out.

But the authors find this explanation is incomplete. In their experiments following a sequential curriculum (Code $\rightarrow$ Math $\rightarrow$ QA $\rightarrow$ CW), they observe that while the global gradient cosine between domains like Math and QA stays near zero [Figure 1a], Math performance still collapses from 66.49 to 57.66 after subsequent training stages. This suggests that interference occurs "under the radar" of global gradient monitoring. Furthermore, the updates themselves are remarkably sparse and small in magnitude .

This means the model is not undergoing a wholesale transformation. If the updates are small and the gradients aren't clashing globally, why does the performance drop? The paper posits that we are watching the whole forest when the fire is actually burning specific trees.

How It Works

The authors move the analysis from the full-model gradient level to the "active route" level. Their approach rests on three core observations:

1. Sparse Edits: Single-domain RL produces very small, sparse parameter changes. There is almost no overlap in the specific neurons being heavily modified by different domains [Figure 3a].
2. Shared Routes: Even though the edits do not overlap, the activations do. Different domains frequently trigger the same sets of neurons during inference. These are the "active computation routes" [Figure 3b].
3. Directional Conflict: Interference happens when a later update moves along a direction that is highly sensitive to the earlier domain's objective.

The authors formalize this with a local perturbation theory. They argue that because the model is already near a local optimum for the first domain, the first-order (linear) impact of a new update is negligible. Instead, the damage is driven by a second-order term (a mathematical term involving the Hessian, or the landscape's curvature). Essentially, the update pushes the model into a region of high curvature for the original task. This damage is concentrated in a low-dimensional "shared active conflict subspace."

Because this damage is localized to a specific subspace, the authors propose two recovery mechanisms: * Domain Refresh: A brief period of RL training on the damaged domain. This geometrically "contracts" the harmful component in the conflict subspace. It restores performance without needing to retrain the entire model from scratch. * Sparse Proxy Rollback: A training-free method that identifies specific parameters responsible for the conflict. It uses a composite score of shared activation, update magnitude, and directional conflict to revert them.

Numbers

The results suggest that the "refresh" strategy is highly efficient. The authors report that after the full sequential training sequence (where Math dropped to 57.66), a brief "Re-Math" refresh recovered Math performance to 66.04 [Table 2]. This brings the model nearly back to its original expert level. Crucially, this recovery was selective. While Math improved, performance on Code, QA, and CW remained largely stable . This allowed the model to reach an average score of 66.39, outperforming both naive joint training (JT) and existing methods like CGPO [Table 2].

On the intervention side, the "rollback" method shows that you do not need a massive budget to see gains. The authors show that reverting only 2% of MLP neurons—selected via their $A \times M \times C$ proxy—could recover 20.4% of the lost Math performance [Table 3]. If you increase the budget to 32% and include attention units, the recovery jumps to 73.6% [Table 3].

What's Missing

While the theoretical grounding is rigorous, there are gaps that a production engineer should note:

• Automation Gap: The paper demonstrates that refreshes work. However, it does not provide a closed-loop algorithm to decide when to refresh or how long to do it. In a production pipeline, you would need an automated way to detect these "silent" performance drops.
• Proxy Imprecision: The rollback method relies on a "coarse proxy" for the conflict subspace. The authors admit this is a basis-aligned heuristic rather than a true projection onto the latent subspace. Consequently, you might need a larger-than-optimal budget to achieve full recovery.
• Generalization Limits: The study focuses strictly on multi-domain RL. It is unclear if these second-order damage mechanics apply identically to other post-training regimes, such as supervised fine-tuning (SFT) or on-policy distillation.

Should You Prototype This

Yes, but as a maintenance strategy, not a training replacement.

If you are building a multi-capability model and find that sequential RL stages are degrading your specialized reasoning capabilities, do not immediately jump to expensive joint training. Instead, prototype a "refresh" schedule. The math suggests that adding a short, targeted training burst for your most critical domains is a computationally cheap way to maintain performance. The "rollback" method is a great diagnostic tool for debugging why a specific domain failed. For a live deployment, the "refresh" is the more robust path.

Figures from the paper