TUA-Bench: A Benchmark for General-Purpose Terminal-Use Agents

The strongest AI agents today fail nearly 35% of the time when tasked with using a computer terminal. Researchers report that the top-performing configuration—Claude Code using Claude Opus 4.8 with maximum reasoning effort—achieves only a 65.8% success rate on the new TUA-Bench. This gap reveals that even frontier models struggle with the complex planning and error recovery required for professional work.

As large language models evolve into autonomous agents, the goal has shifted toward functional utility. We want agents to operate computers on our behalf. Currently, most research focuses on Graphical User Interfaces (GUIs)—the windows and buttons humans use. However, much professional work happens in the terminal. This is a text-based interface where users issue commands to a shell (a command interpreter) to manipulate files or run simulations.

Existing benchmarks have struggled to bridge this gap. Most evaluations focus on the visual complexities of GUIs. This forces models to spend "cognitive" effort on pixel-level coordination rather than pure reasoning. Other benchmarks focus on narrow programming tasks. These ignore the broader digital life of a general user. There is no unified way to test if an agent can move from editing a spreadsheet to conducting medical imaging using only text commands.

The mismatch in current agent evaluations

Current benchmarks are often misaligned with the strengths of language models. GUI-based benchmarks, like OSWorld, require intense visual perception. Agents must interpret screenshots and ground actions to precise screen coordinates. This creates a "perception tax." An agent might fail a task because it misread a button's position rather than lacking logic.

Conversely, terminal-based benchmarks have historically been too narrow. They emphasize shell-native, technical workflows. These essentially ask the agent to be a DevOps engineer or a coder. This leaves a void in evaluating "general-purpose" terminal use. We do not know if an agent can use a command-line interface (CLI) for routine tasks. These include managing emails, shopping online, or handling multimedia files. As shown in the TUA-Bench taxonomy, much of everyday digital work remains untested .

Bridging the gap with TUA-Bench

The authors introduce TUA-Bench to evaluate agents across routine and professional workflows. The methodology uses a dual-track curation process. This ensures the benchmark is neither too easy nor too specialized.

First, the researchers use GUI-to-CLI conversion. They took tasks from GUI benchmarks like OSWorld and reformulated them. Instead of "clicking a button," the agent receives a text-based intent. This preserves the task's complexity while shifting the interaction to text.

Second, they created professional scientific tracks. The authors worked with PhD-level experts in biology, medical physics, and engineering. These tasks are not simple coding puzzles. They require agents to operate specialized software. Examples include OpenFOAM for fluid dynamics or 3D Slicer for medical imaging.

Finally, the system uses deterministic execution. Every task runs in an isolated Linux container. This is managed by the Harbor orchestration framework. This ensures every run starts from a known state. Results are verified by an automated scoring protocol. This protocol checks final files against a ground truth (the known correct answer).

Measuring the limits of frontier reasoning

TUA-Bench reveals that advanced models are far from mastering the terminal. The results show that performance is highly sensitive to the "agent scaffold." This is the software framework that wraps the model and manages terminal interactions.

The paper finds that increasing "thinking effort" helps. This refers to the computational resources allocated to reasoning before an action. Higher effort leads to monotonic improvements in success rates .

Moving from a "none" setting to a "medium" setting can recover significant performance. However, these gains follow a law of diminishing returns. The jump from "high" to "xhigh" effort adds little value while nearly doubling the token cost .

There is also a significant cost-performance trade-off .

Top-tier Claude and GPT configurations reach the highest success. Yet, they do so at a steep price. Some runs exceed $170. In contrast, using open-weight models (models with publicly available weights) within the Terminus-2 scaffold is more efficient. These offer a better Pareto frontier (the set of optimal trade-offs) for developers .

Persistent failures in long-horizon planning

Even frontier models show critical weaknesses. One major issue is the "timeout" problem. The authors report that many failures happen because the agent runs out of time .

This suggests agents struggle with "long-horizon planning." This is the ability to maintain a strategy over many sequential commands.

The benchmark also shows extreme variation in task difficulty. The task-level heatmap reveals "red bands" of failure . Certain categories, like Multimedia and Office productivity, contain tasks that almost all models fail. These represent intrinsic difficulties. Examples include aligning textboxes in a presentation or manipulating image layers. These are not just intelligence failures. They are failures of execution monitoring and error recovery.

The path toward general-purpose autonomy

TUA-Bench proves the terminal is a viable medium for general computing. However, it also shows we are not there yet. The benchmark has some current limitations. It does not cover applications that lack "headless" support (running without a GUI). The professional tracks only cover a few scientific domains. Also, all instructions are currently in English.

Is the industry ready for terminal-based agents? Not quite. The gap between 65.8% and 100% reliability is massive. For practitioners, the takeaway is clear. The choice of agent scaffold is as vital as the model itself. Maximately increasing reasoning effort may simply be an expensive way to mask poor planning and tool use.

The research is available at https://github.com/facebookresearch/TUA-Bench. It provides a foundation for agents to move into professional computing.

Figures from the paper