AnyMo: Scaling Any-Modality Conditional Motion Generation with Masked Modeling

Can We Scale Human Motion Generation Like Language?

We have seen massive scaling successes in Large Language Models (LLMs). However, applying those same principles to the physical, temporal complexities of human motion is much harder. This paper investigates whether human motion generation can achieve robust, multi-modal controllability by scaling both data and architecture.

The Question

The authors ask if a unified framework can handle arbitrary combinations of control signals. Can a model generate a dance driven by music while following a specific spatial trajectory? They want to move beyond "multi-task but single-input" models. They seek a truly omni-modal system that understands cross-modal dependencies in human movement.

Why The Old Answer Was Incomplete

The field has hit a wall caused by two distinct bottlenecks. First, there is a chronic scarcity of large-scale, multi-modally aligned motion data. Most existing datasets rely on expensive, labor-intensive optical motion capture. These systems provide high fidelity but lack the volume needed for scaling laws. While some researchers extract motion from "in-the-wild" web videos, these datasets often lack precise alignment with text, speech, or music.

Second, existing generative architectures are often rigid. Most models are designed for a single modality. They might do text-to-motion or audio-to-dance, but they cannot easily generalize to combined inputs. As shown in [Table 1], previous efforts like HumanML3D or MotionHub offer scale or modality. They rarely offer the combination of both at the required level of alignment.

What They Did

The authors first built OmniHuMo. This is a massive dataset containing over 5,000 hours of motion. It includes 3.2 million sequences extracted from web videos via an automated five-stage pipeline .

This pipeline uses YOLOv11 for detection and GVHMR for 3D reconstruction. It turns raw pixels into structured SMPL (a standard human body model) parameters.

With this data, they proposed AnyMo .

Instead of using standard Vector Quantization (VQ)—which often suffers from "codebook collapse"—they implemented a Residual Finite Scalar Quantizer (R-FSQ). This approach uses hierarchical quantization. A base stream captures coarse motion. Subsequent streams progressively encode finer details.

The backbone is a LLaMA-based bidirectional Transformer. Autoregressive models predict tokens one-by-one. This can struggle with long-range temporal coherence. Instead, AnyMo uses a "Parallel Mask Modeling" strategy. The model masks tokens across all residual levels simultaneously. It reconstructs them in a single pass. This allows the model to leverage global context from both past and future frames.

What They Found

AnyMo achieves significant performance gains as the model scales. For text-driven generation, performance improves as the model grows from 111M to 3B parameters [Table 5]. The authors report stable gains in R-Precision (retrieval accuracy) and FID (a metric measuring how closely the generated distribution matches reality).

The R-FSQ tokenizer shows superior reconstruction capabilities. On the OmniHuMo test set, it achieves a Mean Per Joint Position Error (MPJPE, a measure of geometric accuracy in millimeters) of 13.2 mm. This significantly outperforms previous methods like ScaMo or GoToZero [Table 4].

However, scaling is not perfectly linear across all tasks. In audio-driven generation, scaling is inconsistent. While the Beat Alignment Score (BAS, a measure of temporal synchronization) improves with model size, the FID does not always decrease. This suggests potential overfitting. The model may be struggling with the relatively small subset of audio-aligned data (~500 hours) compared to the massive text-only corpus.

What This Changes

This approach suggests that the "missing link" in robotics and digital human synthesis is better data pipelines. We need systems that can automate the alignment of heterogeneous signals.

There are three immediate implications. First, for practitioners in animation or gaming, combining modalities is highly effective. Adding trajectory cues to text prompts improves motion realism (FID) and reduces trajectory error [Table 8]. Second, the success of R-FSQ suggests that hierarchical, residual-based quantization is more stable for physical signals than traditional VQ. Third, the non-monotonic scaling in audio tasks serves as a warning. Scaling model capacity without proportional increases in specifically aligned data may lead to diminishing returns.

One logical follow-up would be testing the "Parallel Mask Modeling" strategy at even higher resolutions. We should see if it maintains its efficiency advantage during longer temporal windows.

Figures from the paper