Guava: An Effective and Universal Harness for Embodied Manipulation

Most attempts to build autonomous robot arms rely on massive, end-to-end vision-language-action (VLA) models. These models try to map pixels directly to motor torques (the forces driving movement). This approach is notoriously data-hungry and expensive to run. It is also brittle. If a grasp fails, the model often lacks the explicit reasoning loop required to realize it made a mistake and try again.

Researchers have recently explored "harnessing" models. This involves giving a large language model a set of specialized tools to use. Instead of controlling raw motors, the model calls commands like "grasp." This paper introduces Guava, a framework that optimizes how these tools are presented and used. The striking result is that a well-designed harness allows even a tiny 4B-parameter model to match the performance of massive, proprietary frontier models. The authors suggest that the "intelligence" of an embodied agent might reside more in the interface than in the size of the neural network.

The brittleness of end-to-end control

The current state of the art in embodied AI generally splits into two camps. The first involves VLA models that attempt to internalize all low-level perception, planning, and control. As the authors note, these models require enormous amounts of robot demonstration data. This data is difficult to scale and highly dependent on the specific robot (embodiment) used. The second camp uses vision-language models (VLMs) as high-level planners. These models generate code or instructions for a robot to follow.

However, many existing harness-based systems suffer from a "one-shot" problem. They generate a single plan and then execute it blindly. If the robot slips or an object shifts, the system has no inherent mechanism to detect the error. This lack of closed-loop interaction makes them unreliable for long-horizon tasks. These are sequences of actions that require multiple steps to reach a goal. Furthermore, asking a model to reason about raw geometric coordinates (e.g., "move to x=0.54") places an immense cognitive load on the model. This is a known weakness in current VLMs regarding spatial reasoning.

Three pillars of the Guava harness

The authors propose that an effective harness must move away from open-loop execution. Instead, it should favor structured, iterative interaction. They identify three critical design principles, which they validate through ablations (tests that remove components to measure their individual impact) in .

1. Iterative Perception-Reasoning-Action Loops: Instead of a single turn of thought, Guava employs a ReAct-style (Reason + Act) workflow. The agent observes the scene, thinks about its progress, calls a tool, and then observes the result. This creates a closed-loop system. It is capable of detecting execution failures and triggering recovery behaviors.
2. Semantic Action Abstractions: To reduce the geometric reasoning burden, Guava provides a toolkit of high-level, semantic commands. Rather than commanding specific joint angles, the model uses tools like grasp(object). This allows the VLM to focus on the logic of the task. Meanwhile, the physics and geometry are delegated to lower-level controllers.
3. Multimodal Observations: The harness provides both visual images and textual state representations. Textual representations include the current gripper position and orientation. This redundancy helps ground the model's reasoning. It ensures that "left" in the model's mind matches the actual spatial orientation in the environment.

Once these principles were established, the authors moved to the distillation phase. They used a frontier model (GPT-5.4) to interact with a simulator via the Guava harness. They collected 1,934 trajectories. Crucially, they did not just collect successful runs. They intentionally injected errors to generate "recovery trajectories." This teaches the model how to respond when things go wrong .

This data was then used to fine-tune a 4B-parameter Qwen3.5 model. They used a two-stage pipeline: Supervised Fine-Tuning (SFT) followed by Group Relative Policy Optimization (GRPO), a reinforcement learning technique focused on long-horizon reasoning.

Efficiency and real-world transfer

The primary takeaway for engineers is the sheer sample efficiency of this approach. The authors report that distilling these capabilities into a 4B model requires fewer than 2,000 simulation trajectories.

In simulation, the resulting Guava-Agent-4B achieved an overall success rate of 75.6%. This outperformed both the base 4B model and the much larger GPT-5.4, which had a 70.2% success rate [Table 2]. Perhaps more importantly, the model demonstrates significant zero-shot Sim2Real (simulation-to-reality) transfer. This refers to the ability to move from a virtual environment to a physical one without extra training. When deployed on a physical Franka Research 3 robot arm, the 4B model maintained high performance. It achieved an 86% success rate on in-distribution tasks and 92% on out-of-distribution tasks .

These percentages show the model can handle both familiar and entirely new scenarios in the real world.

The authors also highlight that the model is more token-efficient than using a frontier API. It consumes fewer tokens per episode . This reduces the latency (delay) and operational cost of deploying embodied agents in the real world.

Limitations of the semantic interface

While the results are impressive, there are clear boundaries to what this harness can achieve. First, the reliance on semantic abstractions means the system is limited by its toolset. Because the current primitives (basic building blocks) are relatively coarse, the model cannot perform dexterous manipulation. It cannot, for example, rotate a pen within its fingers. It is essentially a "pick-and-place" specialist.

Second, the system is vulnerable to "tool-level" failures. The authors admit that while the model can detect if a grasp failed, it cannot correct errors within the tools themselves. For instance, it cannot fix an error if the segmentation model (SAM3) incorrectly identifies an object's boundaries. The agent can only react to the outcome of the tool, not the process.

Finally, the current setup relies on a single-view, fixed-camera perspective. In complex real-world environments, occlusions (where an object is hidden) are common. Without a multi-view setup or a dynamic camera, the agent's "eyes" are easily deceived. This could lead to failures in unconstrained settings.

The verdict: A blueprint for compact agency

If you want to deploy sophisticated robotic behavior on edge hardware (local devices), the Guava approach is a strong candidate. This avoids the overhead of a trillion-parameter model. The paper demonstrates that the "intelligence" required for manipulation comes from interface design and structured reasoning. It does not come solely from parameter count.

By shifting complexity from model weights to the harness structure, the authors provide a path for scaling. This brings embodied reasoning to smaller, faster, and cheaper open-source models. The ability to teach a 4B model to recover from errors using only 2,000 simulated trials is a powerful proof of concept. Code and project details are reportedly available at https://guava-harness.github.io.

Figures from the paper