OPD-Evolver: Cultivating Holistic Agent Evolver via On-Policy Distillation

Can Agents Truly Learn to Evolve?

Most AI agents can remember things. But they struggle to learn how to get better from their mistakes. Current systems can store past experiences or retrieve helpful tips. However, they often lack the holistic competence to decide which memories are actually worth keeping. They also struggle to turn a failure into a permanent skill. The OPD-Evolver paper proposes a solution. It uses a "fast loop" for immediate action and a "slow loop" to distill long-term wisdom into the agent's actual brain.

Beyond mere memory retention

The researchers wanted to solve a specific deficiency in current agentic foundation models (large-scale models designed to act as autonomous agents). Most existing work treats "self-evolution" as the ability to augment a model with a memory buffer. This essentially gives a smart model a notebook to write in. However, the authors argue that storing a trajectory (a record of actions and observations) is not the same as possessing the competence to evolve. They frame the central research question as: How can we train an agent to acquire the holistic competence of evolving through experience?

To be a "qualified evolver," the authors posit that an agent must master four tightly coupled capabilities. First, it must perform experience selection to find useful memories in a noisy repository. Second, it needs experience-grounded execution to act on those memories. Third, it must perform experience writing to extract knowledge from new trials. Finally, it must perform experience management to consolidate or retire memories. This prevents long-term degradation (the gradual decline in performance as a memory bank grows).

The cracks in the augmentation paradigm

Before this work, many believed that memory-augmented agents could achieve lifelong learning. These systems use retrieval-augmented generation (RAG; a technique to pull external data into a prompt) to pull past experiences into a prompt. The logic was simple. If the agent encounters a similar task, it retrieves the relevant past trajectory and follows it.

But the authors identify a fundamental crack in this approach. Memory is merely the substrate, not the driver, of evolution. If the agent's selection mechanism is weak, it amplifies retrieval noise by pulling in irrelevant items. If its writing mechanism is poor, it pollutes its own future context with generic, unhelpful advice. Essentially, the field has been optimizing fragments of the lifecycle. Researchers have focused on retrieval, execution, or storage separately. They have not addressed how these parts interact. Without supervising memory management, an agent's repository eventually becomes a cluttered graveyard of useless information.

A slow-fast approach to distillation

The authors propose a "slow-fast co-evolution framework." The "fast loop" is the agent's test-time behavior (its behavior during actual use). It interacts with a four-level hierarchy of trajectories, tips, skills, and tools. During this loop, the agent performs the work of reading, using, writing, and maintaining its memory.

The "slow loop" is where the actual learning happens. The researchers introduce a mechanism called outcome-calibrated memory attribution. Instead of just rewarding an agent for a successful task, they estimate the specific value of each memory. They do this by comparing task outcomes in groups where a memory was selected versus where it was merely retrieved but ignored. This creates a "privileged hindsight" signal.

In the slow loop, a "teacher" model uses this enriched data. It performs on-policy self-distillation (a process where a model learns from its own generated outputs) on the "student" model. As shown in, the teacher observes things the student cannot see at deployment. This includes the future utility of a newly written memory or repository-level diagnostics. The goal is to distill these high-level lifecycle competencies directly into the student's policy weights (the internal parameters that govern behavior).

Internalizing the lifecycle

The findings are quite striking regarding the efficiency of the resulting model. The authors report that OPD-Evolver-9B can challenge significantly larger models. It even surpasses the massive QWEN3.5-397B-A17B on several benchmarks. For example, it beats the larger model on SQL tasks (64.01% vs. 62.74%) and State Abstraction tasks (52.92% vs. 51.41%).

Crucially, the paper demonstrates that the agent is actually internalizing the experience. Through ablation studies (tests where specific components are removed to measure their impact), the authors show that removing "slow evolution" or "memory attribution" leads to significant performance drops [Table 3]. Furthermore, the analysis of memory scores in and shows that distillation successfully shifts the distribution of both selected and written memories toward higher utility.

The most compelling evidence of "internalization" comes from .

The researchers found that after distillation, the agent does not just succeed more often. It also becomes more efficient. It reduces the number of execution steps required to solve tasks. This suggests the agent has moved from "looking up instructions" to "knowing how to act."

From augmentation to autonomy

If this framework generalizes, it marks a shift in how we build agents. We move from building agents that use memory to building agents that possess expertise. The implication chain is clear. By treating memory management as a trainable skill, we can create compact models that compete with much larger architectures.

There are two major implications for the field. First, for practitioners, it suggests that the bottleneck in agent performance may not be raw reasoning power. It may be the quality of "meta-cognitive" loops (the processes used to monitor and manage one's own cognition). Second, for theorists, it validates that on-policy self-distillation can bridge the gap between test-time context-use and parameter-level intelligence.

However, I notice the paper does not explicitly address whether this scales indefinitely. It also does not show if the "slow loop" might eventually suffer from feedback loop corruption. The paper does not show generalization to entirely novel modalities, such as moving from text-based agents to robotic control. A necessary follow-up would be to test if an agent trained on digital tool-use can transfer its evolutionary competence to a physical environment where feedback signals are much noisier.

Figures from the paper