GUICrafter: Weakly-Supervised GUI Agent Leveraging Massive Unannotated Screenshots

Scaling GUI Agents via Weakly-Supervised Pretraining

Training AI to use computers usually requires expensive human labeling. Most current approaches rely on massive datasets of human-annotated screenshots. People must manually mark exactly where to click or what to type. This process is slow and costly. It is also difficult to scale across the infinite variety of websites and mobile apps we use daily.

A new study from Tsinghua University and Tencent Hunyuan introduces GUICrafter. This is a framework that sidesteps the data bottleneck. Instead of waiting for humans to label everything, the researchers propose a different way. The agent learns from massive amounts of unannotated screenshots and webpages first. It then refines its skills with a tiny amount of high-quality data.

The results are striking. The authors report that GUICrafter achieves competitive or even superior performance to advanced systems like UI-TARS. Remarkably, it does this while using only 0.1% of its training data.

The bottleneck of manual annotation

Current GUI (Graphical User Interface) agents navigate software by "seeing" the screen. These models struggle with two core issues: visual grounding and generalization. Visual grounding is the ability to accurately locate a specific, fine-grained element. Examples include a tiny checkbox or a specific icon on a cluttered screen. Generalization is the ability to handle a new app or website. This site might look nothing like the ones seen during training.

As shown in, many existing models fail due to these gaps.

They lack a comprehensive understanding of diverse interface layouts. To fix this, developers typically use supervised fine-tuning. This involves feeding the model millions of trajectories (sequences of actions) labeled by humans. However, the authors note that automatic annotation methods often produce unreliable labels. This creates a "data dilemma." The high cost of quality data prevents the scale needed to solve the problem.

Transforming pixels into meta-tasks

The GUICrafter approach utilizes a curriculum learning framework. This is a teaching strategy where a model starts with simple concepts before moving to complex ones. The framework consists of two progressive stages.

In Stage 1, the researchers implement "weakly-supervised GUI pretraining." They do not use human labels here. Instead, they scrape massive amounts of real-world webpages and mobile screenshots. They use web scraping tools to extract "interactive signals." These are the underlying code elements that tell a browser a button is clickable or a field is typable. They then transform these signals into "meta-tasks." For instance, they do not tell the model "Click the Login button." Instead, they give a generic command: "Click any clickable area on the page, such as a button, but not a blank space."

As illustrated in, these meta-tasks act as an inductive summary of all possible GUI interactions.

The model learns through Reinforcement Learning with Verifiable Rewards (RLVR). This is a method where an agent learns through trial and error guided by deterministic feedback. The model receives rewards based on whether its predicted coordinates land within the correct interactive region.

To make the feedback more precise, the authors use a Gaussian position reward. A simple binary reward only cares if you hit the box or not. In contrast, the Gaussian reward provides a smoother signal. It is based on the distance between the predicted point and the center of the target. This helps the model converge more effectively.

In Stage 2, the model moves to high-quality, manually annotated data. This stage "calibrates" the model. It teaches the agent to transition from finding any button to finding the correct button for a specific user intent.

Efficiency through curriculum learning

The primary strength of GUICrafter lies in its extreme data efficiency. The authors report that their 3B parameter model reaches performance levels comparable to or better than UI-TARS. It does this despite using only a fraction of the training data.

On the Mind2Web benchmark, Stage 1 alone provides a significant boost. It improves grounding accuracy by over 10% across various subcategories compared to the base model. When the full two-stage process is completed, GUICrafter-3B achieves an average grounding accuracy of 60.2%. This surpasses the performance of UI-TARS-2B (59.5%). Note that UI-TARS-2B was trained on 18.4 million samples.

The scalability of this method is also evident in .

The authors demonstrate that performance improves as the volume of Stage 1 data increases. There are no signs of saturation up to 50,000 samples. This suggests the model is robust to "noise." This noise refers to imperfect data like overlapping or missing interactive elements.

Limits of the two-stage approach

While the results are impressive, the framework is not entirely autonomous. The authors acknowledge a continued reliance on a small amount of human-annotated data in Stage 2. This data is necessary to reach peak performance. Without this calibration, the model might excel at finding a button. However, it may struggle to distinguish between the "Submit" button and the "Cancel" button in a complex workflow.

The effectiveness of the meta-task approach also depends on the availability of interactive signals. Extracting these from HTML-based webpages is straightforward. However, extracting reliable signals from compiled mobile application binaries is a harder engineering challenge. Finally, the paper does not explore how this framework handles extremely high-resolution displays. It also does not cover unconventional input methods, such as gesture-only interfaces.

The verdict

If you are building a GUI agent and face high data costs, GUICrafter is a compelling blueprint. It proves that you do not need 18 million human-labeled trajectories to build a powerhouse. You can instead build a foundation of "cheap" knowledge from the unlabelled web. You can then refine it with precision.

The method is ready for researchers looking to scale. This assumes they have the infrastructure to scrape and parse environments. Code, data, and models are reportedly available at https://github.com/fansunqi/GUICrafter.

Figures from the paper