Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systems

Can We Trust AI Agents in the Real World?

Current AI tests are like short quizzes. They do not show how an AI performs during a long, complex job. We have seen impressive leaps in how models answer questions or write code snippets. However, these "static" evaluations often fail to capture the messy reality of professional engineering. Ramp, a new research project, attempts to bridge this gap. It tests AI agents on long, multi-step projects—like building a compiler from scratch. This reveals if they can handle real-world errors and stay efficient over time.

Does benchmark performance imply production readiness?

The central question driving this research is whether high scores on current coding benchmarks actually translate to reliable performance. The authors of the Ramp paper argue there is a fundamental mismatch. This mismatch exists between how we evaluate agents today and how they are used in industry. In a production environment, an engineer does not just solve a single, isolated problem. They manage long-horizon tasks. These involve multi-file reasoning, dependency management, and iterative debugging.

The researchers wanted to know how far today’s agentic models can actually go. They tested them against a continuous, stateful, and serial development pipeline. They are not just asking if a model can write a function. They ask if it can maintain a coherent mental model of a growing system. Can it do this without collapsing under its own mistakes or a massive execution history?

The cracks in the isolated task paradigm

Until now, the field has largely relied on a "benchmark-driven paradigm." In this view, evaluation consists of isolated, short-horizon tasks. Each instance is executed independently. The agent's state is reset after every task. This approach assumes that if a model can solve Task A and Task B separately, it can handle a workflow. But in a workflow, Task B depends on the output of Task A.

The authors identify several critical cracks in this assumption. First, traditional benchmarks lack support for "serial dependency." This is the requirement that one task's output serves as the next task's input. Second, they are almost entirely "outcome-centric." They focus on whether the final code works. They ignore the "process," such as cost, time, and stability. This creates a blind spot regarding "cascading failure propagation." This happens when a tiny error in an early stage, like a lexer (a component that breaks code into tokens), silently corrupts everything downstream. This makes it impossible to tell if the model is failing at the later stage or just working with broken tools.

Testing through the resurrection of failed states

To investigate these cracks, the authors developed Ramp. This is an infrastructure built on the YatCC platform. It uses realistic compiler-construction workloads. Instead of isolated snippets, they organized tasks into a serial dependency chain (T0 through T5) [Table 1]. This forces the agent to evolve a single system over time. This mimics the persistent state of a real repository.

The most striking part of their design is the "resurrection protocol." When an agent fails a task (scoring below 60%), the orchestrator does not just stop. Instead, it intercepts the failure. It replaces the agent's broken output with a "golden artifact" (a perfect reference version). It then allows the agent to attempt the next task in the chain .

This increases diagnostic granularity. It separates "the agent cannot do the current task" from "the agent cannot do the next task because the previous one was broken."

The researchers ran this gauntlet across 15 mainstream models. These included frontier models like Claude-Opus-4.7 and GPT-5.5. They also tested open-weight and lightweight models [Table 3]. They did not just look at pass/fail rates. They tracked "process metrics" like token consumption, wall-clock time, and monetary cost. This provided a multidimensional view of agentic utility.

Capability collapse in long-horizon workflows

The results are sobering. The authors report that task completion rates progressively collapse. This happens as the workflow moves through its serial stages. Models achieved 100% completion on the initial environment setup (T0). However, the rate plummeted to just 20.0% by the final assembly generation stage (T5) [Table 4]. This 20% rate means only one in five models could finish the final step. Crucially, none of the 15 evaluated models successfully completed the entire six-task pipeline.

Even the top performer, Opus-4.7, stalled at the IR Optimization stage (T4) [Table 4]. Here, "IR" refers to Intermediate Representation, a middle-stage version of the code. The model achieved a Mean Reward (MR) of 93.39 but could not finish the whole chain. The data suggests the decline is not just due to tasks getting "harder." Since T5 actually showed a slightly higher completion rate than T4, the decline likely comes from cumulative error propagation.

Perhaps most surprising is the "efficiency gap." Computational costs varied by up to 2,525x among models with comparable performance [Abstract]. For example, Opus-4.7 delivered the highest reward. However, it did so at a "prohibitive cost" compared to more efficient models like DeepSeek-v4-Pro [Section 4.3]. This led the researchers to propose the Agent Efficiency Index (AEI). This composite metric penalizes models that achieve high scores through brute-force token consumption or excessive latency [Section 3.3.3].

Implications for the future of agent deployment

If these findings generalize, the implications for AI deployment are significant. The authors note the current workload is limited to compiler construction.

First, the "best" model for production might not be the one with the highest benchmark accuracy. As the paper demonstrates through the AEI, a model like GPT-5.5 might be a superior choice. It balances reasonable performance with much higher resource discipline than a "reward-heavy" model like Opus-4.7 .

Second, the research highlights that "Context Failure" is a primary bottleneck. This is the exhaustion of the model's memory window. It accounted for 60% of hard-stop failures .

This suggests that solving long-horizon agency is a systems-engineering problem. It requires better memory management, state compression, and tool integration.

Finally, the "resurrection" methodology proves we can measure "latent downstream capability." We should not assume an agent is useless just because it broke a dependency chain. We need to know if it could have succeeded if the environment had been stable.

Figures from the paper