DomainShuttle: Freeform Open Domain Subject-driven Text-to-video Generation

Can AI Subjects Survive a Change of Scenery?

Researchers have developed AI systems that can take a single photo of a person or object and animate it into a video. This field, known as subject-driven text-to-video (S2V) generation, allows users to provide a reference image and a text prompt to create custom content. However, a persistent tension exists: if you ask the AI to move a real-world person into a fantasy setting—like a watercolor painting or a 3D animation—the system often struggles. It either refuses to change the art style to keep the person looking "real," or it changes the style so much that the person becomes unrecognizable.

The central question facing researchers is whether a single model can "shuttle" between these worlds. Can an AI preserve the intrinsic identity of a subject—their specific facial features or unique textures—while remaining flexible enough to adopt entirely new domain attributes like lighting, brushstrokes, or cinematic styles?

The rigidity of in-domain fidelity

Until recently, the prevailing strategy in S2V research was to maximize "in-domain" fidelity. This means the model focuses almost exclusively on replicating the reference image as accurately as possible. As seen in, this works well when the goal is to keep a subject in their original environment. However, this creates a massive bottleneck for creativity.

Because existing models treat the reference image and the target video as a single, intertwined stream of information, they often suffer from "entanglement." In this context, entanglement refers to the inability of the model to distinguish between the subject's permanent features (like a person's eye color) and the environment's temporary features (like the sunlight hitting their face). If the reference image is a bright, sunny photo, the model often forces that sunny lighting onto every video it generates. This lack of editability makes current state-of-the-art methods rigid. This limits their use in professional creative pipelines like filmmaking or advertising.

Disentangling identity from environment

To solve this, the authors of the DomainShuttle study propose a framework that treats the video and the reference image as two distinct entities. They trained their model on the Wan2.1-14B and Wan2.2-14B architectures. The total training cost reached approximately 30,000 GPU-hours. Their investigation centers on three specific architectural moves designed to break the bond between a subject and their original domain.

First, they introduce Domain-MoT (a Mixture-of-Transformers). Think of this as a specialized sorting office. Instead of mixing all incoming data into one pile, Domain-MoT uses independent processing paths for the video and the reference images. This allows the model to apply "domain-aware" adjustments to the reference features without accidentally overwriting the structural integrity of the video [Figure 2(a)].

Second, the researchers address spatial confusion through Video-Reference DualRoPE. Most models use Rotary Positional Encoding (RoPE)—a method that uses rotation matrices to encode the relative positions of tokens—by treating reference images as just extra frames in a video sequence. The authors argue this is flawed because a reference image is a blueprint, not a moment in time. By placing reference tokens in a separate RoPE space, they allow the model to understand that the subject in the image is a distinct entity from the moving actors in the video [Figure 2(b)].

Finally, they implement a Cross-Pair Consistent Loss (CCL). During training, the model is shown two different sets of reference images of the same subject. These sets might vary in lighting or angle. The model is tasked with ensuring the generated video remains consistent regardless of which set it sees [Figure 2(c)].

Breaking the cross-domain barrier

The results of this approach suggest that decoupling these features unlocks the "shuttling" capability the authors sought. The study finds that DomainShuttle achieves a significant 18.7% improvement in the Cross-Domain (CD) Score over existing state-of-the-art methods. This score measures how well a model maintains a subject's identity when forced into a new visual style.

The qualitative evidence is perhaps more striking than the metrics. In tests involving real-to-fantasy transformations, such as turning a real person into a paper-cut art style, DomainShuttle successfully transitioned the entire scene while keeping the subject recognizable .

In more complex scenarios, such as mapping a fantasy character onto a real-world object (like a character printed on a bus), the model demonstrated far superior text controllability compared to its predecessors . Even in challenging "real-meets-fantasy" interactions, such as a woman interacting with a living painting, the model maintained the logical boundaries of the two different domains .

From rigid replication to fluid creation

The implications of this work reach beyond simple video generation. If the ability to decouple identity from style can be generalized, it marks a shift from "copy-paste" AI to truly "generative" AI. Instead of a model that merely mimics a provided image, we move toward models that understand the concept of a subject.

For practitioners, this suggests a future where digital assets can be instantly ported into any cinematic universe. A character could move from a realistic film to a vibrant animation without losing their core identity. For researchers, the success of the Cross-Pair Consistent Loss highlights a vital lesson. Teaching a model to ignore "noise" (like lighting or occlusion) is just as important as teaching it to recognize "signal" (like shape and texture).

While the paper demonstrates remarkable success in open-domain scenarios, it does not explore how these decoupled paths might handle extremely high-resolution requirements or long-form temporal consistency. A logical next step would be to investigate whether this dual-path architecture can maintain subject identity during rapid, complex physical transformations, such as a character changing clothes mid-motion.

Figures from the paper