Rethinking Continual Experience Internalization for Self-Evolving LLM Agents

When AI agents try to learn from their own past mistakes to get better, they often actually get worse over time. This happens because the process of converting experiences—past interactions and successes—into permanent model weights is surprisingly unstable. Most current approaches attempt to turn "notes" from previous runs into part of the model's core knowledge. However, as the agent tries to evolve through multiple rounds of training, its performance tends to collapse rather than compound.

This paper investigates why this "capability collapse" occurs and provides a recipe to stop it. The researchers argue that for an agent to truly self-evolve, we must move beyond simply feeding it raw logs. Instead, we need to distill high-level principles, inject them only when relevant to the current step, and train the model on high-quality demonstrations rather than its own flawed, real-time attempts.

The collapse of the self-evolution loop

The goal of experience internalization is to move from in-context learning (using past examples in the prompt) to parametric capability (having that knowledge baked into the model weights). Currently, most researchers focus on single-iteration transfers. They take a set of successful trajectories, use them to supervise a model, and finish. This works well for a one-off upgrade.

However, the dream is a closed-loop system. The agent acts, learns from those actions, updates its weights, and then uses that new version to generate even better experiences. As shown in, this is where current methods fail.

When using iterative on-policy context-distillation—where the model learns from its own generated trajectories—performance doesn't climb. It drops. The agent enters a cycle where it reinforces its own errors. This leads to a progressive loss of capability across successive iterations.

A three-dimensional recipe for stability

The authors identify three specific levers to stabilize this loop. To solve the collapse, they propose a shift in how experience is represented, delivered, and learned.

1. Granularity (The "What"): Instead of "instance-level" experience, which preserves specific, brittle details like URLs or exact numbers, they use "principle-level" experience. This involves extracting abstract strategies and failure patterns (e.g., "always verify the source before answering") rather than raw logs. This prevents the model from overfitting to the specific quirks of a single training trajectory.
2. Injection Pattern (The "When"): Rather than "global injection," where the entire experience pool is dumped into the prompt at the start of a task, they use "step-wise injection." An LLM-based selector looks at the current interaction history. It then pulls in only the relevant piece of experience for the immediate next decision. This ensures the supervision is aligned with the agent's current state. This is critical for long-horizon tool use (tasks involving many sequential steps).
3. Regime (The "How"): Finally, they replace "on-policy" distillation with "off-policy" distillation. In the on-policy setup, the model learns from its own trajectories. In the off-policy setup, the model is trained on high-quality trajectories generated by an experience-aware teacher. These are often refined via rejection sampling (a method of keeping only the best outputs).

Sustaining gains across iterations

The results suggest that these changes are foundational for long-term stability. The authors report that while instance-level experience might offer a quick boost in the first round, it quickly degrades. It often falls below the base model's performance .

The impact of the injection pattern is even more stark. Under global injection, the model develops a massive "premature-answer" problem. The authors note that in the third iteration, global-injection models triggered an answer without any preceding tool calls in 63.82% of cases [Table 2]. Step-wise injection reduced this failure rate to 0%. This preserves the agent's ability to seek evidence .

When combining all three elements—principle-level experience, step-wise injection, and off-policy distillation—the authors demonstrate a successful self-evolution loop. As seen in, the model maintains robust performance gains across multiple iterations.

Crucially, the model doesn't just perform better "internally." It also retains its "experience-use ability." This means it can still effectively utilize the experience pool if provided in-context during inference.

Implementation blind spots

While the results are compelling, there are gaps a practitioner should note. First, the evaluation is centered on web-reasoning tasks. Whether this "principle-over-instance" abstraction holds up in other domains is not demonstrated here.

Second, the complexity of the "step-wise" approach introduces a new dependency: the selector. The effectiveness of the system hinges on the quality of the LLM used to pick the right experience. The paper uses DeepSeek-V4 for these tasks. The performance delta might change if you use a much smaller, local selector.

Finally, the paper does not quantify the total compute overhead of the off-policy regime. Generating teacher trajectories and performing rejection sampling is likely more expensive than simply rolling out the student model. For a team worried about the cost of continuous fine-tuning, this cost remains unquantified.

The verdict

If you are building an agent meant to improve itself over time, the status quo of on-policy distillation is a trap. You may eventually hit a wall where the model begins to skip essential steps.

The transition to principle-level abstraction and off-policy training is a necessary evolution for self-improving systems. The "step-wise" injection is a key takeaway for engineers. Stop treating experience as a static document. Start treating it as a dynamic lookup table. The code and data are available at https://github.com/RUCBM/ExpInternalization. If you have the budget for the extra rollouts required by the off-policy regime, this is a solid architecture to prototype.

Figures from the paper