Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots

Bridging the Gap: Transferring Human Manipulation Skills to Robots via Wrist Translation

Researchers have found a better way to teach robots by watching humans. Instead of trying to copy every complex finger movement, they focus on how the human's wrist moves. This "bridging action" allows robots to learn diverse tasks like opening microwaves or drawers much more effectively than traditional methods.

Scaling robot learning relies heavily on human data. This data is cheap and abundant compared to expensive robot trajectories. However, moving knowledge from a human hand to a robotic gripper is difficult. Most existing approaches treat the human as just another robotic limb. They attempt to map 6-degree-of-freedom (6DoF) hand poses—the full range of spatial orientation and position—directly to the robot.

This strategy often fails. Human hand-pose estimation is inherently noisy. Furthermore, the way human fingers interact with objects differs from how a parallel gripper (a simple two-pronged robotic claw) operates. The authors argue that forcing a robot to mimic human finger rotations is sub-optimal. Instead, they propose focusing on a shared physical reality: how the wrist moves through space.

The failure of 6DoF imitation

The current standard involves extracting full 6DoF wrist actions from human video data. This includes both the XYZ position and the roll, pitch, and yaw rotations. The authors report this approach is problematic for two reasons. First, hand-pose estimators are prone to errors. This makes the extracted rotational data jittery or incorrect. Second, the "contact patterns"—how digits touch and manipulate an object—are mismatched. A human uses complex finger dexterity to rotate an object. Conversely, a robot gripper primarily relies on the movement of the entire arm.

As shown in, training with these noisy 6DoF human actions leads to problems. The resulting robot behavior is often distorted or off-target. Instead of performing a smooth task, the robot may exhibit twisted, unnatural poses. These poses do not align with the intended manipulation goal. This mismatch creates a semantic gap. It prevents the robot from translating human "intent" into executable commands.

Translating motion through the wrist

To bridge this gap, the authors propose a "bridging action" based solely on relative wrist translation. They move away from complex rotations. Instead, they focus on the 3D displacement of the wrist within the head-camera frame (the coordinate system established by the camera on the robot's head). Because both humans and robots act based on what they perceive visually, this translation vector serves as a universal language of motion.

The researchers implement this via a Vision-Language-Action (VLA) model. This model uses flow matching—a generative technique that transforms noise into structured data—to predict action chunks. To handle different levels of data detail, they use "interleaved action tokens." As illustrated in, the model organizes information in a specific sequence.

It starts with 3D wrist translation, followed by the robot's 6DoF end-effector action, and ends with the gripper signal.

The model uses attention masking (a technique to ignore certain parts of a data sequence) to deal with missing components. For example, human data lacks robot-specific gripper signals. In these cases, the model simply masks those parts of the sequence. This allows the model to learn the "what" and "where" of a task from humans. It reserves the "how"—precise 6DoF and gripping—for robot-specific data.

Scalability and transfer efficiency

The effectiveness of this method is demonstrated through complex manipulation tasks .

These include opening microwave doors, unstacking cups, and operating drawers. The authors report that the bridging action enables the robot to perform tasks beyond simple pick-and-place operations.

In their comparative analysis, the authors find that training a robot only on generalized pick-and-place data results in very poor performance .

However, co-training with human actions using the bridging representation increases the success rate significantly. The paper highlights that this representation is highly scalable. Increasing the amount of human-only pre-training on wrist translations leads to measurable improvements in task performance .

Furthermore, the authors demonstrate improved data efficiency. Even when the robot receives only a "few-shot" amount of real robot data (just 10 trajectories per task), performance improves with human-only pre-training . This suggests that the "knowledge" of how to navigate a workspace is successfully transferred. The robot learns the environment's geometry even without seeing a human use a gripper.

Limitations in fine dexterity

Despite these gains, the paper identifies clear boundaries. Because the model intentionally discards rotational supervision from human data to avoid noise, it faces challenges. It struggles with tasks requiring extremely precise angular adjustments.

The authors note several failure cases in their analysis . Tasks like "inserting a straw into a cup" or "opening a drawer" can be problematic. In these scenarios, the robot might understand the general direction of motion. However, it may fail to execute the specific wrist rotation needed for proper contact. Additionally, the model occasionally struggles to grasp thin or delicate objects. The authors attribute this to the "embodiment gap" (the physical difference between a human hand and a metal gripper) and noise in human motion data.

The verdict on human-to-robot scaling

Is human video a viable resource for teaching robots? The findings suggest yes. Human video is a high-leverage resource for teaching the fundamental geometry of manipulation. By shifting focus from "copying fingers" to "translating wrist paths," the authors align two different physical forms.

However, this is not a complete replacement for robot-specific training. The "bridging action" provides the roadmap. The robot still needs its own data to learn its mechanical constraints and contact physics. For those building foundation models for embodied AI, the takeaway is clear. Prioritize translation-based signals for pre-training. Use interleaved architectures to let the model distinguish between universal motions and machine-specific ones. Code for the project is reportedly available at the official project page: https://translation-as-a-bridging-action.github.io/.

Figures from the paper