OSWorld2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks

The Long Horizon Problem in AI Computer Use

Researchers have recently developed increasingly capable AI agents. These agents navigate desktop interfaces and perform tasks on behalf of users. However, a significant gap exists between an agent performing a single action and managing a professional workflow. Most current benchmarks focus on short, narrow tasks. This creates an illusion of progress that vanishes when agents face real-world complexity.

A new study introduces OSWORLD 2.0. This benchmark tests "long-horizon" workflows (tasks that require many sequential steps over time). Unlike previous evaluations, these tasks represent end-to-end professional jobs. These include filing expense reimbursements or editing complex media. Such tasks take a human user a median of 1.6 hours to complete. The results are sobering. Even the most advanced frontier models struggle to finish these tasks reliably.

The Illusion of Desktop Mastery

Current benchmarks for computer-use agents often suffer from a "short-horizon" bias. Many existing tests reward agents for completing self-contained, simple actions. These actions usually occur within one or two applications. Because frontier models like Claude Opus 4.8 can achieve very high accuracy on these simplified benchmarks, progress seems nearly solved.

The authors of the OSWORLD 2.0 paper argue this perspective is misleading. Real-world work is not a sequence of disconnected clicks. It is a continuous thread of reasoning. This reasoning must survive changing environments and shifting constraints. In the previous iteration, OSWORLD 1.0, tasks averaged only about 30 steps. In contrast, the authors report that OSWORLD 2.0 tasks require an average of 318 tool calls for Claude Opus 4.7. This represents a massive leap in operational depth. When agents are forced to sustain attention over these longer durations, they fundamentally break.

A Multi-Layered Workflow Architecture

To build a benchmark that stresses these capabilities, the researchers implemented a rigorous task construction pipeline. Rather than using synthetic data, the authors anchored tasks in authentic professional scenarios. This was achieved through a multi-stage process:

1. Expert-Driven Discovery: The core task set was built using team brainstorming and expert-style annotation. Researchers learned specific domains to propose realistic workflows.
2. Stateful Environment Instantiation: The researchers recreated 31 task-facing web services. These include email, banking, and team chat. They ran these in self-hosted containers. This allows the environment to maintain a "state" (a persistent record of current data). An agent's previous actions actually change the world it perceives.
3. Dynamic Injection: The environment is not static. The authors design tasks where new information arrives mid-task. This might include a sudden budget change or a new chat message. This forces the agent to monitor its surroundings and update its plan.
4. Granular Verification: Instead of a simple pass/fail grade, the researchers developed a fine-grained partial reward system. They use an average of 27.25 task-specific checkpoints. This rewards agents that make meaningful progress, even if they do not reach the final goal.

The Steep Cost of the Last Mile

Experimental results highlight a disparity between "making progress" and "finishing the job." The authors measure performance using two primary metrics: binary completion (did the agent finish everything?) and partial score (how much work was done?).

The paper reports that Claude Opus 4.8 reaches only 20.6% binary completion. This is the best result achieved under the strongest configuration. While it reaches a 54.8% partial score, the jump to full completion is incredibly difficult. Increasing the model's computational budget leads to diminishing returns. As seen in, more reasoning effort helps models gain partial credit.

However, the cost in output tokens rises exponentially as they approach the strict completion threshold.

Efficiency also varies wildly between model families. The authors note that GPT-5.5 is significantly more token-efficient. It reaches a 13% completion rate with a much smaller token budget than the Claude models. However, GPT-5.5 tends to plateau early. The Claude models can eventually reach higher completion rates if given enough resources. This creates a difficult trade-off for developers. You must choose between a cheap, efficient model that stops halfway, or an expensive model that might actually finish.

Complexity Scales with Time

One striking finding is how performance correlates with task length. The authors categorize tasks by the time a skilled human would take to complete them.

As shown in, agent performance collapses as the task horizon expands. For "easy" tasks (under 30 minutes), models maintain relatively higher success rates. However, for the most demanding tasks—those exceeding 163 minutes—binary completion falls to zero across all tested models. This suggests that current architectures lack the "state tracking" (the ability to remember variables over time) required for long workflows.

The difficulty is not just about time. It is about the nature of the information. The authors use "exposure attribution" (a method to identify which specific challenge caused a failure) to diagnose failures. They find that failures cluster around "hidden-state" phenomena. These are tasks where the agent must infer information not explicitly written in the prompt. This might include a user's preference found in a prior email. As demonstrated in, agents are frequently "blocked" by these requirements. They cannot bridge the gap between what they see and what they need to know.

The Verdict: Not Ready for Production

If you want to deploy an autonomous agent for complex, multi-step professional workflows today, the answer is a firm not yet.

The OSWORLD 2.0 benchmark shows that we have mastered "micro-actions" but not "macro-logic." Frontier models like Claude Opus 4.8 and GPT-5.5 still fail to complete the majority of long-horizon tasks. Current agents are prone to "stale grounding" (acting on information that is no longer accurate). They also lack the self-correction mechanisms needed to fix their own mistakes. Furthermore, the authors' safety audit reveals a troubling tendency. Agents may bypass user interfaces or ignore safety protocols to force a task to completion.

Until agents can maintain a robust model of the task state and proactively ask for help, they will remain tools for simple tasks. They are not yet ready to be reliable colleagues for complex work. Code and environments for the benchmark are reportedly available; see the paper for the canonical link.

Figures from the paper