ProCUA-SFT Technical Report

ProCUA-SFT: Scaling Computer-Use Agents via 3.1M Synthetic Step-Level Samples

Computer-use agents (CUAs) are models designed to interact with graphical desktops. They perceive screenshots and execute keyboard or mouse commands. Progress in this field is currently bottlenecked by the scarcity of high-quality training data. Creating CUA trajectories requires a fully instantiated desktop environment. This includes booted VMs (virtual machines), installed applications, and realistic file content. This makes large-scale data collection much more expensive than traditional web scraping.

Until now, the industry has relied on limited human-annotated resources like AgentNet. AgentNet provides roughly 22.5K trajectories. However, researchers found that simply using this human data can actively damage performance. This paper introduces ProCUA-SFT. It is a massive synthetic dataset designed to solve the diversity and feasibility problems in existing CUA training methods.

The failure of human-centric SFT

Common wisdom suggests that more human data is always better. This paper finds the opposite. Using the largest available human resource, AgentNet, for supervised fine-tuning (SFT) actually causes the UI-TARS 7B model to regress. Specifically, the authors observe that fine-tuning on AgentNet causes the OSWorld success rate to plummet from 26.3% to just 8–10% .

This means the model becomes significantly worse at performing realistic desktop tasks after seeing the human data.

The authors attribute this "negative transfer" to three specific failures in the AgentNet data. First, there is a lack of task diversity. Most trajectories are shallow, single-application workflows with a median of only 17 steps. Second, the data lacks complex, cross-application reasoning. Finally, crowd-sourced annotations introduce noise. Naive synthetic generation fails similarly. Generated tasks often reference non-existent files or unavailable apps. This leads to wasted compute and "hallucinated" states. In these states, the model learns to invent files that do not exist.

A closed-loop synthetic pipeline

To solve the feasibility problem, the authors implement an in-loop verification mechanism. The ProCUA-SFT pipeline uses a single Vision-Language Model (VLM), Kimi-K2.5. This model plays three roles: goal generator, precondition judge, and trajectory executor. This unified approach closes the "planner-actor capability gap." This gap occurs when a strong planner proposes goals that a weaker actor cannot execute.

The mechanism follows a strict four-stage logic:

1. Grounded Task Synthesis: The generator emits a goal paired with binary preconditions (e.g., "Does the file Q3.xlsx exist on the Desktop?").
2. Precondition Verification: A separate judge prompt evaluates these requirements against the current screenshot and OS configuration. Only if all preconditions pass does the rollout proceed. If they fail, the verdict is fed back to the generator for an iterative retry.
3. Real-World Content Seeding: The authors seed desktops with heavy, real-world content. This includes 912 spreadsheets from SpreadsheetBench and ~10K presentations from Zenodo10K. This forces the agent to perform genuine data extraction and cross-app coordination.
4. Step-Prefix Expansion: The authors expand a single trajectory into $T$ individual training samples. Each sample includes the full history of screenshots and actions up to that specific step. This matches the exact context layout the model sees during inference.

This architecture scales across heterogeneous compute. A stateless vLLM (a high-throughput serving engine) endpoint handles inference. Independent processes manage the local KVM (Kernel-based Virtual Machine) or serverless NVCF (NVIDIA Cloud Functions) environments.

Massive scale and performance gains

The results of this synthetic approach are significant. The authors report that fine-tuning UI-TARS 7B on ProCUA-SFT for one epoch yields a 45.0% success rate on the OSWorld benchmark. This is an 18.7 percentage-point improvement over the base model. It is also a 35.0 percentage-point jump over the AgentNet-trained version . For a developer, this means the agent is nearly twice as capable as the original model.

The superiority of the dataset comes from its structural characteristics. ProCUA-SFT trajectories are much longer. The mean length is approximately 29.7 steps, compared to AgentNet’s 18.6 steps .

The action distribution is also more robust. AgentNet is heavily click-dominated (~63%). ProCUA shifts more actions toward keyboard primitives and text entry. These are inherently less brittle than pixel-perfect mouse movements .

The authors also tested different ways to pick data. They fixed the training budget and compared strategies. They found that "round-robin" sampling based on application combinations achieved the highest success rate (30.9%) [Figure 2c]. Trying to balance specific action sets actually lowered performance. This suggests that covering a wide range of app combinations is the most important factor for CUA training.

Structural complexity and the long tail

The paper analyzes the "topology" of these trajectories. They model them as screen-transition graphs. This analysis shows that task difficulty depends on coordination patterns. A task might involve four applications but remain mostly linear .

Conversely, a highly non-linear task might require constant "cycles." This means the agent must backtrack between a browser and a spreadsheet to extract data .

To handle the "long tail" of rare application combinations, the authors used a complexity-aware curation strategy. They identified rare trajectories (those appearing $\leq 3$ times). They used these as few-shot demonstrations to synthesize new, complex data. The "Rare-App Synthesis" subset shows higher complexity. The mean number of applications per trajectory increases from 2.0 to 3.3. The mean number of app switches also more than doubles .

Verdict: A blueprint for synthetic scaling

The ProCUA-SFT report shows that synthetic data is essential for specialized agentic tasks. The leap from 10% to 45% success rate is a direct result of the grounded verification loop. It also comes from injecting real-world document complexity.

If you are building a CUA, the takeaway is clear. Stop chasing human demonstrations for desktop automation. Human data is often noisy and shallow. It is insufficient for the reasoning demands of modern OS environments. Instead, invest in a grounded synthesis pipeline. The core value is the verification loop. This loop prevents the model from learning to hallucinate state.

The dataset is available on Hugging Face at ProCUA-SFT. If you have the compute to run a fleet of VMs, this offers a reproducible path to better agent performance.

Figures from the paper