OmniDirector: General Multi-Shot Camera Cloning without Cross-Paired Data

Precise Cinematography Without the Math

Researchers have developed a way to copy camera movements from a reference video and apply them to a still image. Instead of forcing users to input complex mathematical coordinates, this method turns the movement into a simple visual "grid" that looks like a 3D room. This allows an AI to understand and replicate even complex scene changes and specialized camera effects.

In the field of video generation, controlling the "lens" is as critical as controlling the subject. Currently, creators must choose between two imperfect paths. They can type descriptive text like "slow pan right," which lacks cinematic precision. Alternatively, they can input explicit camera parameters (mathematical matrices defining rotation and translation). This latter method creates a steep technical barrier for non-experts. A third option—cloning motion from a reference video—is intuitive. However, it is historically difficult because it requires massive amounts of "cross-paired" data. This refers to datasets where the AI sees the exact same motion performed in two different scenes. Such data is incredibly rare in the real world.

The OmniDirector framework, reported by researchers at Kuaishou Technology and Tsinghua University, breaks this bottleneck. By representing camera motion as a visual signal rather than a set of numbers, the authors allow the model to learn from any arbitrary video. This enables high-precision, multi-shot camera cloning without needing specialized paired datasets.

The limitations of parameters and pixels

Current approaches to camera control generally fall into two camps: explicit or implicit. Explicit methods inject camera motion into models using geometric primitives like Plücker coordinates (a way to represent lines in 3D space) or rotation matrices. While mathematically precise, these methods struggle with "scale ambiguity." This is a problem where a translation distance extracted from a close-up shot in a reference video is applied incorrectly to a wide-angle shot in the generated video. This causes unnatural geometric distortions.

Implicit methods attempt to avoid this math by training models on videos that share identical camera movements but differ in content. However, the authors note that such strictly controlled data is exceedingly scarce. To compensate, some researchers use synthetic data from game engines. The paper argues these often lack the narrative complexity found in real cinema. This includes things like abrupt shot cuts or intricate scene transitions. Most importantly, many existing models suffer from "information leakage." This happens when the model accidentally copies the actual appearance or characters from the reference video instead of just its motion .

Visualizing motion through the camera grid

To solve the data scarcity and scale problems, the authors introduce the "camera grid." Rather than feeding the model raw numbers, they render the camera's trajectory within an empty 3D environment. Think of this like a motion-capture suit for a camera. Instead of recording muscle voltages, it records the visible path of a dancer through a room.

The mechanism works in several stages:

1. Spatial Modeling: The system abstracts the real world into an empty room. It consists of a floor, a ceiling, and a "tubular boundary" of vertical lines that trace the camera's path .

2. Rendering: The camera's mathematical parameters are used to render this 3D grid from the perspective of the moving camera. This transforms abstract motion into a "grid motion video." This video looks like a series of moving lines in a void.
3. Effect Encoding: The grid can be modified to represent specialized cinematography. For example, the authors show how a "dolly zoom" can be encoded. This is a technique where the subject stays the same size while the background stretches. It is achieved by combining a 3D cube at the subject's location with a secondary tracking view .

4. Integration: This grid is treated as a standard visual signal. It is encoded using a 3D Variational Autoencoder (VAE) (a tool used to compress visual data into a simpler format). It is then concatenated with the reference image and the noisy video latents within a Multi-Modal Diffusion Transformer (MMDiT) .

This visual representation decouples the motion from the content. Because the grid is an "empty" scene, the model learns the geometry of the movement. It is not distracted by the textures or characters of the original video.

Measuring director-level control

The authors report that OmniDirector significantly outperforms existing state-of-the-art methods across nearly every metric. In a quantitative evaluation, the paper finds that OmniDirector achieves a 39.3% relative improvement in translation precision (T-Pre) over the previous best method, CamCloneMaster. This jump is largely attributed to the camera grid's ability to generalize across different spatial scales.

Regarding the difficulty of managing multiple shots, the authors report impressive results for transition accuracy. The model achieved a Temporal Transition Precision (Tem-Pre) of 96.52%. This means the timing of shot changes is almost perfectly aligned with the reference. It also achieved a Semantic Transition Precision (Sem-Pre) of 83.79%. This indicates the model correctly understands the type of transition occurring.

Crucially, the authors measure "leakage." This quantifies how much the model accidentally steals content from the reference. OmniDirector maintains very low rates. It reports 0.51% for frames and 3.38% for shots. In contrast, other models like LTX-LoRA exhibit much higher leakage.

Complexity and temporal limits

While the results are strong, the paper identifies specific areas where the technology is not yet "production-ready" for long-form content. The authors admit that their method of injecting control signals via direct token concatenation struggles with long-term memory. It may also struggle with temporal consistency as video sequences grow significantly longer. For a filmmaker, this means a single complex shot might look perfect. However, a ten-minute film might eventually drift in its visual logic.

Additionally, the performance relies heavily on the accuracy of the initial pose estimation from the reference video. While the authors include a "hierarchical prompt expansion agent" to help clean up these descriptions using a large language model, inaccuracies could still occur. Errors in the underlying camera tracking could lead to suboptimal results.

The verdict: A new standard for motion cloning

OmniDirector is a significant step toward making AI video generation a tool for actual directors. By turning math into vision, the authors have bypassed the need for impossible-to-find paired datasets. They have also solved the problem of scale mismatch. The discovery of "emergent" control is also noteworthy. The model can follow camera motions even when given raw edges or standard video instead of a formal grid . This suggests the representation is even more robust than intended.

If you are looking to prototype high-fidelity, controllable video workflows, this is a highly promising direction. The project page is available at https://ymlinfeng.github.io/OmniDirector.github.io/. However, until the authors solve the long-term temporal consistency issue, expect this to remain a tool for shot-by-shot creation rather than full-scene automation.

Figures from the paper