μ_0: A Scalable 3D Interaction-Trace World Model

Can We Learn Physics Without Labels?

Robot learning faces a fundamental data paradox. We have an abundance of video showing humans and robots interacting with the world. However, we lack the massive, labeled datasets of specific motor commands needed to train high-performance policies. These commands include joint torques or end-effector velocities (the movement of the robot's hand in space). Most attempts to bridge this gap involve predicting dense pixels. This wastes capacity on irrelevant background textures. Other methods train Vision-Language-Action (VLA) models. These require expensive, hardware-specific labels that do not scale across different robots.

The question facing researchers is: can we find a middle ground? Is there a way to extract the essence of physical motion from raw video? We need a representation that is compact, metric (measurable in real-world units), and usable by any robot, regardless of its morphology (physical shape).

The search for a universal motion interface

The authors of $\mu_0$ investigate whether 3D interaction traces can serve as this universal interface. Instead of asking a model to reconstruct a high-fidelity video frame, they predict the 3D trajectories of salient interaction points. These points include objects, tools, hands, and contact regions. They capture the "what" and "where" of a task. This strips away the "how" of a specific robot's kinematics (the mathematical description of its motion).

This approach aims to decouple the physics of the world from the mechanics of the agent. By focusing on these sparse, semantic 3D traces, the model can learn from any heterogeneous video source. This includes human, robot, or handheld camera footage. The model then provides a reusable motion prior (a learned starting point for understanding motion) for any downstream controller.

Cracks in the pixel and action paradigms

Current approaches generally fall into two camps. Both hit walls in production-scale learning. Pixel-space video models attempt to learn world dynamics by generating future frames. While visually impressive, they spend too much parameter budget on reconstructing stable backgrounds. They struggle to capture the metric geometry and contact structures required for precise manipulation.

Direct action models, like the $\pi_0$ series, sidestep the pixel problem. They predict control commands directly. However, they are tethered to the data they were trained on. If you train on a specific UR5 arm, the model is often useless for a different gripper. Earlier attempts at trace-based modeling, such as TraceGen, had limitations.

They relied on fixed-density grids. These grids wasted queries on static surfaces. They also struggled with varying camera motions. The field needed a way to select meaningful points. It also needed to maintain their 3D identity across long sequences.

Building the TraceExtract engine

To solve this, the researchers developed TraceExtract. This is a data engine designed to turn uncurated video into structured 3D supervision. The process starts by using DINOv2 (a self-supervised vision transformer) to cluster features. This helps identify distinct entities. Instead of a blind grid, the system allocates a keypoint budget to these entities. This ensures that small but critical parts like tool tips are not ignored .

Once keypoints are selected, the system performs a hybrid global-local 3D reconstruction. It establishes a shared global coordinate frame using sparse anchor frames. It then aligns dense local video chunks back to that frame. This prevents the "drift" typically seen in long-sequence tracking. To make the data actionable, the engine uses a Savitzky–Golay filter (a method for smoothing data) to detect acceleration peaks. This segments the video into "events," such as grasping or pouring. It then pairs these with hierarchical language captions .

The $\mu_0$ model itself is a query-conditioned world model. It uses a pretrained VLM (Vision-Language Model) backbone for semantic context. It also uses a specialized "Trace Expert" to predict future motion. Crucially, the expert predicts B-spline control points (mathematical points that define a smooth curve). This ensures the predicted trajectories are smooth and physically plausible. It avoids the jittery waypoints found in other models .

Surprising parity with action-labeled models

The results suggest that this "action-free" pretraining is remarkably potent. In 3D trace forecasting benchmarks, $\mu_0$ outperformed established baselines like TraceGen and various tokenized VLM methods across all tested horizons [Table 1]. Qualitatively, the predicted traces were more coherent and goal-directed. They avoided the misalignment seen in prior methods .

The most striking finding comes from downstream deployment. When the authors froze $\mu_0$ and trained a task-specific "Action Expert" on the RoboCasa365 simulation, $\mu_0$ achieved an average success rate of 30.25%. This exceeded the performance of $\pi_0$. Note that $\pi_0$ is a VLA model pretrained with actual action supervision [Table 2]. In real-world UR3 manipulation tasks, the advantage was even clearer. $\mu_0$ reached a 91.7% success rate. This significantly outperformed both VLM-only baselines and action-labeled models like $\pi_0$ and $\pi_0.5$ .

Implications for cross-embodiment scaling

If these results hold, the implication is powerful. We can build capable robot controllers using nothing but the vast, unlabeled ocean of internet video. We no longer need to wait for expensive teleoperation datasets. We can simply observe how humans interact with objects and extract the 3D traces.

There are two immediate consequences for practitioners. First, the "frozen $\mu_0$" architecture suggests a modular path to deployment. You can ship a heavy, high-capacity world model as a frozen feature extractor. You then only train a lightweight, embodiment-specific head for the actual robot. Second, the research shows that intermediate trace-denoising features are a superior signal. They provide better conditioning for action heads than raw video or end-to-end latent features.

However, we should be cautious. The authors admit that $\mu_0$ is still beholden to its perception stack. If the initial 3D reconstruction or semantic clustering fails, the world model inherits those errors. It also lacks an explicit model for contact forces or tactile feedback. These are vital for high-precision tasks. The next step is to test how this approach handles tasks where force-sensing is the primary modality.

Figures from the paper