LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysis

LongDS-Bench: Exposing the State-Management Bottleneck in Long-Horizon Data Analysis Agents

Current AI agents struggle with long data analysis tasks. They lose track of previous steps, definitions, and data filters. This new benchmark, LongDS, tests if AI can handle complex, multi-turn workflows. It examines cases where one mistake early on ruins everything later. Even the best models fail more often as the analysis gets longer and more complex.

The Problem

Most existing benchmarks for LLM-based data science focus on isolated tasks. They evaluate an agent's ability to solve a single, self-contained request in a resettable environment. This essentially asks the model to write a code snippet and then wipes the slate clean. This ignores the reality of professional data analysis. Real work is inherently iterative and stateful (where the outcome depends on the history of previous actions).

In a real production workflow, an analyst builds a foundation and refines metrics. They might apply a filter or realize they need to roll back to a previous version of the data. Current benchmarks fail to test "evolving analytical state." This is the accumulation of scopes, definitions, and intermediate results that shift across turns. As seen in, a single session might require an agent to track how a definition evolves over dozens of turns.

How It Works

The authors move away from static task evaluation. Instead, they use a framework centered on state-evolution patterns. LongDS constructs tasks around six specific ways an analytical state changes [Table 1]:

1. Initial: Establishing foundational analytical objects like cleaned tables.
2. Inheritance: Reusing the most recent valid state without redefining it.
3. Update: Revising a previous definition to become the new default.
4. Counterfactual: Applying a temporary, local change without altering the permanent state.
5. Rollback: Reverting to an earlier version of the analysis.
6. Composition: Combining multiple prior states to form a new perspective.

To build this, researchers used a pipeline to curate real-world Kaggle notebooks. They used a coding agent (Codex) to decompose these notebooks into "analysis clusters" .

These clusters are groups of code cells with shared objectives. The benchmark then injects long-range dependencies. A request in turn 24 might depend on a state from turn 1. The average dependency span is 11.3 turns. illustrates this through a 36-turn Netflix market analysis.

Numbers

The results are sobering for anyone building autonomous data agents. The authors evaluated five state-of-the-art models. These included GPT-5.4, Gemini-3.1-Pro, and Claude-4.6-Sonnet. Even the best performer, Gemini-3.1-Pro, reached an average accuracy of only 48.45% [Table 2]. This means the leading model fails more than half the time.

The performance degradation is severe. There is a nearly 47-percentage-point drop in accuracy between the first 10% and the last 10% of task progress [Figure 5a]. This shows that agents struggle as analytical states accumulate. Furthermore, long-horizon errors dominate failures. Specifically, cascading errors (where an early mistake propagates) and state-management errors account for 52% to 69% of all failures [Figure 6c].

Crucially, adding more "interaction budget" does not help. Increasing the number of agent steps does not consistently improve accuracy [Figure 6a]. More steps can actually increase state drift (the tendency for an agent's internal state to deviate from the correct path). The bottleneck is not coding ability. It is the ability to maintain a coherent mental model of the evolving data environment.

What's Missing

LongDS is a rigorous stress test, but it has gaps. First, the benchmark relies on public Kaggle notebooks. These offer realistic workflows but may not capture proprietary data structures found in enterprises.

Second, the evaluation is strictly quantitative. The authors focus on "quantitatively verifiable questions" to ensure deterministic grading. This means the benchmark lacks coverage for open-ended insight generation or visualization-heavy analysis. If your use case involves an agent explaining why a trend occurs, LongDS won't test that.

Finally, the domain distribution is imbalanced. Certain sectors like Sports have significantly fewer tasks .

Conclusions about "general" performance might be skewed by the characteristics of more represented domains like Geoscience or Business.

Should You Prototype This

If you are building an agent to replace a junior analyst, you are not ready to ship. The core issue is not a lack of reasoning power. It is a fundamental failure in state maintenance.

Instead of making your agent "smarter" with more reasoning steps, focus on robust state-management architectures. This might mean implementing a layer that tracks variable definitions and data lineage. This layer should operate separately from the LLM's context window. If you want to begin testing, the authors released the code and data at https://github.com/zjunlp/DataMind. Start by running your agent through "Rollback" or "Counterfactual" patterns. If your agent cannot handle those, it will not survive a long production session.

Figures from the paper