Adaptive Auto-Harness: Sustained Self-Improvement for Agentic System Deployment on Open-Ended Task Streams

When we deploy AI agents into the wild, we often expect them to get smarter over time. In practice, we frequently see the opposite. Agents perform brilliantly on day one, but their effectiveness craters as the world changes.

Current research focuses on "auto-harness" systems. These frameworks automatically optimize an agent's prompts, tools, and skills based on feedback. Most of these systems are trained on static, offline benchmarks. Real-world deployment is different. It is a continuous, unpredictable stream of tasks. Rules shift, topics change, and history grows indefinitely. When an agent uses one massive, "dense" toolkit for everything, it suffers from cognitive overfitting. It accumulates specialized skills that help with yesterday's problems but sabotage today's.

The collapse of the dense harness

The fundamental problem with current auto-harnessing is the assumption of a stationary environment. This is a world where the distribution of tasks stays the same. As the authors demonstrate, repeatedly evolving a single, monolithic harness leads to performance degradation. In their testing on prediction markets, they show that as an agent undergoes more evolution cycles, its prompt size and skill count balloon. However, accuracy eventually peaks and then declines .

This happens because a single harness becomes a "jack of all trades, master of none." It becomes cluttered with "toxic artifacts." These are rules or tools that were effective for a specific past task but are now harmful. For instance, a specialized skill for analyzing sports data might cause an agent to misinterpret political news . The authors identify three critical dimensions of deployment: unbounded streams where history grows forever, task heterogeneity where tasks require different expertise, and distributional non-stationarity where the nature of tasks shifts over time .

Decomposing regret into evolution and adaptation

To solve this, the authors propose a framework that separates the construction of capabilities from the selection of those capabilities. They formalize this using a regret decomposition. They split the gap between an agent's performance and an ideal "oracle" harness (the best possible harness) into two parts. These are Evolution Loss ($L_{evo}$) and Adaptation Loss ($L_{adapt}$).

$L_{evo}$ represents the structural limit of the system. It is the gap caused by the evolver's inability to build certain complex capabilities. $L_{adapt}$ represents the failure to match the right harness to the specific task at hand. The Adaptive Auto-Harness system attacks these two losses through three core architectural choices:

1. Stateful Multi-Agent Evolution: Instead of one agent managing a growing history in one context window, the system uses a four-phase pipeline (Analyze $\rightarrow$ Research $\rightarrow$ Build $\rightarrow$ Verify) .

Dedicated agents handle each stage. They use a persistent workspace to maintain cross-cycle memory (information preserved across different rounds of improvement). 2. Harness-Tree Routing: To minimize $L_{adapt}$, the system does not use one big toolkit. Instead, the evolver builds a "harness tree." This is a collection of specialized branches managed via a git-like structure . At solve-time, a lightweight "router" agent inspects the task. It then directs the task to the most appropriate branch. 3. Human-in-the-Loop (HITL) Hooks: Some signals cannot be learned from history alone. This includes things like proprietary API credentials. The system includes triggers for human intervention . Humans can then provide necessary external information.

Evidence of sustained performance

The authors validate this approach across three distinct, open-ended streams: PolyBench (prediction markets), CTF-Dojo (cybersecurity), and FutureX (event forecasting). The results suggest that separating these concerns is effective.

In the PolyBench task, the "Full System" achieved an accuracy of 80.9%. It also saw a +352% increase in Return compared to a no-evolution baseline [Table 2]. This means the agent became significantly more profitable in its trades. Ablation studies show that the multi-agent structure and persistent memory are essential. Removing either leads to significant performance drops .

The system's ability to handle heterogeneity is shown by the routing analysis. In CTF-Dojo, the "Adapt" variant (using the router) closed a significant portion of the adaptation gap [, Table 11]. This gap is the difference between the router's choice and the best possible branch. Furthermore, the bottleneck analysis shows the system targets the specific "binding capability" (the most critical limiting factor) for each domain .

It improves calibration for prediction markets and enhances web-retrieval for forecasting.

Identifying the remaining gaps

There are clear boundaries to what this system achieves. First, the authors admit that $L_{evo}$ and $L_{adapt}$ are analytical constructs used for diagnosis. They are not direct measurements of an absolute oracle. Therefore, we cannot mathematically guarantee how much "optimal" performance remains.

Second, the efficacy of the routing mechanism depends on the task. In the FutureX forecasting benchmark, the router did not provide as much lift as it did in other domains [, Table 11]. This is because the primary bottleneck there was source acquisition. The system needed better data access rather than better tool selection.

Finally, the human-in-the-loop component is highly specific. The data suggests that human intervention is most effective when it injects "missing external signals" . For example, providing an API key is more effective than providing general strategic advice. This implies the system still requires humans to identify exactly what the agent is missing.

Verdict: A blueprint for production agents

If you are building an agent meant to run autonomously in a changing environment, this paper is a mandatory read. Moving from "optimizing a prompt" to "managing a tree of specialized capabilities" is a necessary step. This shift helps move LLM agents from research toys to reliable production infrastructure.

The core insight is that we must decouple capability growth from task execution. This prevents "knowledge dilution," where new information washes out old, useful skills. While managing a multi-agent evolver and a git-based harness tree is complex, the trade-off in sustained performance appears worthwhile. The code is available at https://github.com/A-EVO-Lab/AdaptiveHarness.

Figures from the paper