Your Agents Are Aging Too: New Benchmark Reveals Reliability Decay in Long-Lived AI Systems
Just like humans, AI agents get "worse" the longer they are used. Their memory gets cluttered or they forget important details. This paper introduces a new way to test and fix these "aging" problems. It aims to keep agents reliable across months of use.
Current evaluations for AI agents focus on "day-one" performance. We treat agents like freshly initialized models. We test their ability to reason or recall facts in a static environment. But in production, agents are persistent operational systems. They accumulate interaction history and compress old logs into summaries. They also revise facts as the world changes. This creates a massive blind spot. We have no standardized way to measure how long an agent remains reliable after deployment.
The Problem
The status quo fails because it ignores the "agent harness." This is the complex loop of memory writing, storage, retrieval, and utilization surrounding the base model. Even if you freeze the model weights, the effective system state drifts. As seen in, an agent might suffer from "compression aging." This happens when a summarization step drops a critical detail.
It might also suffer from "interference aging." This occurs when similar names cause the agent to retrieve the wrong identity.
Most current benchmarks treat reliability as an end-to-end score. They ask, "did the agent answer correctly?" For a long-lived system, that is insufficient. A single accuracy drop doesn't tell you the root cause. You won't know if you need a better summarizer or a better retrieval algorithm. Without knowing why the agent is failing, engineers are left guessing which part of the stack to optimize.
How It Works
The authors introduce AgingBench. This is a longitudinal benchmark designed to measure the "lifespan" of an agent. Instead of a single snapshot, it tracks reliability over many sessions. The methodology relies on three architectural pillars:
- Temporal Dependency DAGs: The authors use a Directed Acyclic Graph (DAG) to encode cross-session structure. This helps track "version chains" (where one fact supersedes another). It also tracks "dependency edges" (where a probe requires facts from multiple prior sessions). This distinguishes between a model lacking capacity and one failing to handle evolving state.
- Four-Mechanism Taxonomy: The researchers categorize aging into four failure modes. These are compression (loss of detail during write), interference (confusion during retrieval), revision (failure to update derived state), and maintenance (regressions triggered by system events like memory compaction). This taxonomy is visualized in .
- Counterfactual Diagnostic Probes: This framework uses an ablation ladder to move from detection to diagnosis. It replaces parts of the memory pipeline with "oracles" (perfect implementations of retrieval or writing). This isolates where the error originates. As shown in, the pipeline includes Write (W), Retrieval (R), Utilization (U), and Store (S).
If an oracle retriever (P2) restores performance, the failure was in retrieval. If injecting gold context (P3) restores it, the failure was in the model's utilization.
Numbers
The authors conducted over 400 runs across 14 models and 7 scenarios. Agent aging is highly multi-dimensional. No single model dominates across all categories.
The paper reports that behavioral compliance and factual precision can decouple. In scenario S2, the authors observe a decoupling. The "Constraint Violation Rate" (CVR) remains near zero. This means the agent still follows conversational patterns. However, its actual factual precision collapses [Figure 7b]. This "silent aging" is dangerous for production monitoring. Standard safety filters may not catch it.
Regarding specific interventions, the authors show that addressing the right mechanism yields gains. For "revision aging" (tracking numeric updates like budgets), they tested a "Typed-state Overlay." This is a JSON sidecar that handles math outside the LLM's lossy text summarization. The paper reports this overlay reduced accumulator error by up to 47% compared to standard careful compaction [Table 6]. This represents a significant reduction in mathematical drift over time.
What's Missing
While AgingBench is a significant step, it has gaps. First, the scenarios are programmatically generated. This allows for "pressure dials" seen in .
However, it may not capture the messy entropy of real human interaction. Real users provide noisy and contradictory inputs.
Second, the benchmark focuses heavily on compaction-based memory. Many production agents use vector-indexed retrieval (RAG). The authors acknowledge that vector database aging might differ fundamentally. This could include issues like index staleness or embedding drift.
Finally, the counterfactual attribution is a "diagnostic profile." It is not a unique causal decomposition. Components like Write, Read, and Utilize are tightly coupled in reasoning loops. An intervention in one stage might mask issues in another. This makes attribution a useful guide rather than a definitive debugging tool.
Should You Prototype This
Yes, but specifically as a testing harness for your memory architecture. Do not use it as a replacement for your unit tests.
If you build an agent meant to live in a user's environment, do not rely on day-one benchmarks. The findings in prove that even "strong" models exhibit predictable decay curves.
You should use the principles of AgingBench. Specifically, use mechanism-level probing. Determine if failures are due to poor summarization (Write), retrieval noise (Read), or reasoning lapses (Utilize).
The code is reportedly available at https://AgingBench.github.io/. Start by implementing the "counterfactual probe" logic in your integration tests. This is a high-leverage way to ensure your agent does not silently lose its mind after ten sessions.