Rethinking the Evaluation of Harness Evolution for Agents

Most work on LLM agents focuses on building better "scaffolds." These are the prompts, tools, and memory structures that allow a model to interact with a computer. This field of "harness evolution" aims to automate this process. It uses an AI to iteratively design better environments for other AIs. But a new paper argues that we have been measuring this progress incorrectly. Researchers found that trying to automatically improve these tools doesn't actually work better than simply letting the agent try a task multiple times. Furthermore, the supposed improvements often fail to work on any new, unseen tasks. This happens because the systems often just memorize specific fixes for the training set.

The flaw in current harness evaluation

The status quo in automatic harness evolution involves a search loop. An agent analyzes past failures. It then proposes modifications to the harness (the collection of prompts, tools, and control logic). It evaluates those changes on a benchmark and repeats the cycle. However, the authors identify a fundamental flaw in how these methods are reported. Most studies use the same public benchmark—such as Terminal-Bench—both to guide the search and to report final performance.

This creates two major epistemic problems. First, harness evolution is itself an iterative search procedure. If you spend a large amount of compute searching for a better harness, you are performing "search" at the meta-level. The authors argue that this must be compared against simple task-level search baselines. These baselines spend that same compute budget on the task itself rather than the harness. This determines if gains come from better design or just from having more chances to guess correctly. Second, because the search and evaluation share a benchmark, the reported gains risk being nothing more than overfitting (adjusting a model to fit a specific dataset too closely) to that specific set of tasks. As illustrated in, there is a critical distinction between "Harness Evolution," which seeks a reusable design across a distribution, and "Harness Scaling," which adapts a harness for a single instance.

Current literature often conflates the two.

Comparing search versus design

To untangle these effects, the authors implement a unified budget framework. This framework categorizes four distinct ways to spend inference compute (the computational resources used during a model's response generation). They focus on what the agent updates and what feedback it observes:

1. Parallel Sampling: A test-time scaling method where the agent draws $K$ independent trajectories (sequences of actions and observations) using a fixed harness. It expands the "width" of exploration to increase the chance of hitting a correct solution.
2. Sequential Refinement: A test-time scaling method that allocates budget to "depth." Each subsequent attempt is conditioned on the reasoning or failures of the previous one, using a fixed harness.
3. Harness Evolution: The target method, which uses a meta-agent to optimize a shared harness across a batch of tasks. Unlike the scaling methods, it attempts to create a reusable artifact.
4. Harness Scaling: A novel baseline introduced by the authors. This is an instance-guided adaptation. The harness is revised specifically for a single task. It acts as a middle ground between task-level search and global evolution.

By matching the total inference and feedback budgets across these four approaches, the authors aim to isolate whether "evolving" a harness actually produces a superior, generalizable tool.

Evidence of overfitting and inefficiency

The empirical results are a sobering reality check. The authors evaluate these methods on Terminal-Bench 2.1 using Claude Opus 4.6, GPT-5.4, and GPT-5.4 mini.

In settings where unit test cases (automated scripts that verify if a task was completed correctly) are unavailable, harness evolution actually underperforms simple test-time scaling. As shown in, the average pass@1 (the probability that the first randomly sampled trajectory is correct) for harness evolution is 67.4%.

This is lower than the 72.3% achieved by Parallel Sampling. The authors note that for strong models like GPT-5.4, iterative harness revision can actually degrade performance. This occurs when the agent is forced to rely on its own noisy, self-generated feedback.

Even when unit test cases are provided, the hierarchy of effectiveness remains unchanged. In Table 2, the authors report that Parallel Sampling (86.0% average pass@1) and Sequential Refinement (91.8% average pass@5) consistently outperform Harness Evolution. Note that pass@5 measures the fraction of tasks solved by at least one of five attempts. Crucially, the authors observe that the benefits of these methods mostly materialize when they can select from multiple successful trajectories. This suggests the methods are finding solutions via repetition rather than through improved harness design.

The most damning evidence comes from the generalization test. When the evolved harness was tested on a held-out set of tasks, the gains evaporated. The tasks used to "train" the harness were entirely different from the ones used to evaluate it. The paper reports that Harness Evolution yielded only a marginal 0.6% average improvement over the initial harness on the test set. This is a stark contrast to the much larger gains seen during the in-distribution evaluation.

Why evolution fails to generalize

There are several reasons why these results might hold. First, the authors suggest that most "rational" edits made by the meta-agent are actually just memorizing fixes. By looking at case studies, they show that the meta-agent often embeds specific command sequences, file paths, or variable names directly into the prompt. This helps bypass a specific error. However, this is "knowledge injection" rather than "strategy distillation." This technique offers zero utility for a new task with a different file structure.

Second, there is the problem of "context bloat." As the meta-agent adds more rules, warnings, and templates to the harness, the prompt becomes increasingly dense. This extra noise can eventually offset the benefits of the added instructions. It may confuse the underlying model.

Finally, the authors suggest a possible ceiling effect. If the bottleneck for an agent is the underlying model's capacity for deep reasoning, then no amount of clever prompting will bridge the gap. The authors note that if the tasks are already "solved" by the model's inherent capabilities, the harness becomes a secondary factor.

The verdict: Don't evolve your harness yet

If you are currently building an agentic workflow, my advice is: probably not.

The evidence suggests that if you have extra compute, you are almost always better off spending it on parallel sampling or sequential refinement. Trying to build an automated pipeline to "evolve" your prompts and tools currently offers a lower return on investment. The current state of harness evolution functions primarily as a sophisticated way to overfit to your specific benchmark.

Until we develop methods that can distill generalizable heuristics—rather than just memorizing specific command strings—automatic harness evolution remains a research curiosity. If you want to pursue this, the authors' suggestion is clear. Stop evaluating on the same tasks you use for optimization. Instead, start building benchmarks that are specifically designed to be sensitive to scaffolding. Code for this critique is available at https://github.com/rethinking-harness-evolution.