Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents

Harness Updating is Not Harness Benefit: Disentangling Evolution in LLM Agents

When AI agents try to improve themselves by updating their own instruction manuals—the prompts, skills, and memories that guide them—we assume that smarter models will always build better manuals. However, a new paper suggests this is a mistake. While even small models can write highly effective instructions, the real bottleneck is that weaker models often fail to actually read or follow those new instructions once they are provided.

In the growing field of agentic systems, we are moving away from static prompts toward "harness self-evolution." This is the process where an agent performs a task, learns from its mistakes, and then automatically updates its external harness (the collection of tools, skills, and prompts it uses) to avoid those mistakes in the future. Historically, we have evaluated these systems end-to-end. We measure if the agent gets better. We then assume the improvement came from a smarter "evolver" (the model responsible for writing updates) creating better instructions. This paper breaks that assumption apart.

The Problem

The current status quo in agent research conflates two distinct capabilities. It mixes the ability to generate a better harness with the ability to utilize that harness. Most self-evolving agent methods focus on the former. They use an LLM "evolver" to write new skills or prompts based on execution evidence. But end-to-end scores hide the source of truth. If an agent's performance climbs, is it because the evolver is getting smarter? Or is it because the agent is getting better at following directions?

As shown in, the harness is an external, non-parametric layer (meaning the model weights stay fixed while the environment changes).

Current evaluations fail to disentangle these roles. This leads practitioners to potentially waste significant compute scaling up the evolver when the actual bottleneck is the agent. This lack of clarity makes it impossible to know if a performance plateau is caused by a mediocre optimizer or a model that simply cannot adhere to long-horizon instructions (complex tasks requiring many sequential steps).

How It Works

The authors propose a framework to decouple these two roles. They define two specific metrics: harness-updating (the evolver's ability to produce useful updates) and harness-benefit (the agent's ability to actually gain performance from those updates).

The evolution protocol operates in an iterative loop: 1. Execution: A frozen LLM agent attempts a batch of tasks using its current harness $H_t$. It generates execution trajectories $\tau$ and final outputs $y$. 2. Evidence Collection: The system collects this execution evidence $D_t$. This includes successes, failures, and tool-use traces. 3. Evolution: An "evolver" model processes this evidence to propose a harness update $\Delta H_t$. This update might involve writing a new "skill" (a reusable procedural module) or revising a prompt. 4. Application: The update is applied to the harness. This creates $H_{t+1}$ for the next cycle.

To analyze this, the researchers varied the evolver and the agent independently across three benchmarks: SWE-bench Verified (software engineering), MCP-Atlas (tool orchestration), and Skills-Bench (skill-based execution). This allowed them to isolate whether a model's base capability predicted its effectiveness in either role.

Numbers

The findings reveal a massive decoupling between model scale and utility. The authors report that harness-updating is flat in base capability. Surprisingly, even a tiny 9B model (Qwen3.5-9B) can produce harness updates that yield gains comparable to a frontier model like Claude Opus 4.6 .

In fact, the case study in shows that a 9B evolver can produce a skill that is "procedurally isomorphic." This means it prescribes the exact same logic and steps as a much larger model.

However, harness-benefit is non-monotonic. The authors find that mid-tier models benefit the most. Conversely, weak-tier models see almost no gain. This isn't because weak models can't understand the updates. Instead, it is due to two specific failure modes identified in : * Harness Activation Failure: Weak models fail to even load the new tools or skills.

On Skills-Bench, the authors report a Skill-Load Rate (SLR) of only 0.251 for Qwen3-32B. This is significantly lower than the $\approx$0.96 seen in strong models [Table 2]. * Harness Adherence Failure: Even when a skill is loaded, weak models fail to follow it. The authors measured a Harness-Following Rate (HFR) of just 0.142 for Qwen3-32B [Table 2].

Furthermore, the authors highlight a "drift" problem. Adherence decays sharply over long-horizon tasks. For example, Qwen3-32B's adherence score drops from 0.52 immediately after loading a harness to just 0.13 by the final turn [Table 3].

What's Missing

The paper outlines several specific limitations regarding its scope: * Parametric vs. Non-Parametric: The study strictly looks at updating the external harness. It does not address whether combining harness evolution with parametric fine-tuning (updating the actual model weights) would overcome the adherence bottlenecks seen in weak models. * Model Coverage: The authors note that their model set is representative but not exhaustive. A broader grid of model families and scales might clarify these findings further. * Hybrid Adaptation: The work focuses on harness updates rather than hybrid methods that combine weight updates and harness updates.

Should You Prototype This

Depends on your model tier.

If you are building agentic systems using frontier models (Claude 3.5/4 series, GPT-4 class), the research suggests that spending extra effort on a sophisticated, high-parameter "evolver" provides diminishing returns. You can likely get away with a much smaller, cheaper model to manage your prompts and skills.

However, if you are trying to deploy smaller, local, or specialized models, do not rely on self-evolution to fix their performance. Simply giving them better instructions won't work if they can't reliably invoke them or follow them over a long trajectory. Instead of building an evolver, your engineering priority should be training or fine-tuning those smaller models specifically for harness invocation and long-horizon instruction following.

Code is reportedly available; see paper for the canonical link.

Figures from the paper