Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents

RoTS: Enhancing GUI Agent Robustness via Error-Driven Trajectory Synthesis

Most AI agents are impressive until they make a single mistake. In the context of Graphical User Interface (GUI) agents—models designed to automate digital devices by interacting with screens—a single wrong click can trap the agent in a loop of failure. Most current systems lack the ability to recognize they have gone off-track and initiate a recovery sequence.

The field has moved quickly toward improving grounding (the ability to map text to specific screen coordinates) and planning. However, a massive gap remains: how does an agent handle "policy-induced errors"? These are mistakes generated by the agent's own previous actions. Until now, we have lacked a way to measure this specific failure mode. We also lacked a scalable way to train agents to overcome it.

The Question

Researchers at Alibaba Cloud Computing aimed to solve a structural problem. They wanted to bridge the gap between current GUI agent capabilities and the robustness required for real-world deployment. Specifically, they investigated why existing agents fail to recover from their own mistakes. They sought to build a training pipeline that proactively discovers diverse error modes. Their goal was to generate long-horizon recovery trajectories (sequences of corrective actions) needed to fix them.

Why The Old Answer Was Incomplete

The prevailing approach to agent robustness has been flawed. First, evaluation has relied on benchmarks focusing on external noise or adversarial attacks. These involve injecting random perturbations (small, unintended changes) into the environment. They do not capture the internal logic failures an agent creates itself. Second, training data has primarily consisted of successful human demonstrations. It also relies on short-horizon errors, such as an invalid click, which are easy to identify.

The authors argue this creates a "coverage mismatch" and a "horizon mismatch." As shown in [Figure 3(a)], existing training data concentrates on low-level execution errors. Real-world failures are often compositional (made of multiple parts) and high-level. Furthermore, [Figure 3(b)] shows that while training data focuses on immediately identifiable errors, real policy-induced errors often emerge only after several steps. These require deep, long-horizon backtracking. Current agents are essentially being trained to pass a test that does not reflect the delayed nature of their own mistakes.

What They Did

To address this, the authors introduced two interconnected components: GUI-RobustEval and RoTS (Robustness-driven Trajectory Synthesis).

First, they built GUI-RobustEval. This benchmark contains 1,216 executable test cases. It systematically measures error awareness (detecting a mistake) and post-error success (fixing it) across 11 error types and four controllable error depths .

This allows engineers to diagnose exactly where an agent's reasoning breaks down.

Second, they developed RoTS, a tree-based online data synthesis framework. Instead of just collecting successes, RoTS grows a "trajectory tree" in a live GUI environment. It employs a dual-branch strategy: 1. Fragility-Driven Exploration (FDE): On successful branches, the system uses a "progress critic" (a model that judges if an action moves toward the goal) to find "fragile" states. These are nodes where the agent is technically succeeding but is close to failure. It then pushes the agent to explore these boundaries to discover new failure modes. 2. Experience-Informed Recovery (EIR): On failed branches, the system uses an "experience-informed reflector" to localize the error. It looks at "neighboring branches" (successful sibling paths in the tree) to derive natural-language recovery guidance. This guidance is then used to train a "recovery actor" to perform the fix.

This co-expansion process, visualized in, turns a single failure into a rich training pair. This pair includes the error, the reflection on that error, and the successful recovery.

What They Found

The results suggest that the bottleneck for GUI agents is specialized robustness training. The authors report that their RoTS-32B model achieves state-of-the-art performance on the OSWorld benchmark. It hits a 47.4% success rate and a 33.8% All-Pass@4 score [Table 3]. The All-Pass@4 metric measures the fraction of tasks solved in all four independent runs. This serves as a proxy for true reliability and consistency.

On the GUI-RobustEval benchmark, RoTS-32B demonstrated gains across all error categories [Table 12]. For example, it achieved a 33.2% success rate in the most challenging setting (error depth 5). This proves that long-horizon recovery training actually works.

They also identified a critical hyperparameter for practitioners: the ratio of reflection-related data ($\lambda_{ref}$). According to, there is a "sweet spot" around $\lambda_{ref} = 0.1$.

Too little reflection data leaves the agent fragile. Too much leads to "ineffective reflections" that actually degrade performance.

What This Changes

If this methodology scales, it marks a shift from "demonstration-based learning" to "failure-based learning" for autonomous agents.

First, it provides a blueprint for building self-improving data flywheels. Instead of relying on expensive human labeling, engineers can use tree-based exploration. This lets the model "discover" its own weaknesses and synthesize its own curriculum.

Second, it changes how we define "performance" in agentic systems. High average success rates on simple tasks are becoming a vanity metric. The real frontier is the "All-Pass" rate. This is the ability to maintain consistency in the face of self-induced drift.

Finally, the paper highlights a practical warning regarding "over-reflection." As noted in the analysis of failure cases, models trained heavily on recovery can sometimes misinterpret a normal state transition as an error. This leads to redundant, wasteful "corrections" that consume inference budget without adding value.

Figures from the paper