TryOnCrafter: Unleashing Camera Trajectories for Realistic Video Virtual Try-on via a Renderable 4D Try-on Proxy

Can We Break Free from the Source Camera?

Current virtual try-on videos are stuck with the camera angle of the original video. If you record a person walking in a shirt, the AI can swap the shirt. However, you can only watch that movement from the same perspective. TryOnCrafter attempts to solve this. It allows users to rotate the camera and view the person and clothing from any angle. This essentially enables 360-degree orbital viewing.

The goal is to transition from passive video replay to active, camera-controllable video virtual try-on (CaM-VVT). This requires the model to hallucinate textures for unseen viewpoints. It must also maintain strict structural synchronization between a moving human and a static background under arbitrary camera paths.

Defining the CaM-VVT Frontier

The central question the authors investigate is whether a unified framework can decouple garment synthesis from the input video's fixed trajectory. In standard Video Virtual Try-on (VVT), the AI is a passenger. It follows the camera's lead. But for a real "try-before-you-buy" experience, a user wants to lean in or orbit the subject. They might want to inspect the side profile of a jacket.

The authors identify this as a new research frontier: Camera-controllable Video Virtual Try-on. To succeed, the system must solve two conflicting problems. It must handle viewpoint-agnostic texture hallucination (predicting how fabric looks from new angles). Simultaneously, it must ensure non-rigid human dynamics (movement of soft bodies) stay aligned with the background context as the camera moves.

The Cracks in Sequential Pipelines

Before this work, many treated this as a two-stage problem. First, you perform the virtual try-on. Second, you apply a video-to-video (V2V) camera control model to redirect the perspective. The authors argue this approach is fundamentally flawed for high-fidelity fashion for three reasons.

First, there is cascaded error accumulation. If the initial try-on stage produces slight texture inconsistencies, the subsequent camera control model amplifies those errors. It often treats "glitched" textures as actual scene geometry. Second, existing V2V methods lack explicit modeling of human geometry and garment deformation. This leads to a loss of spatio-temporal coherence (consistency across time and space). Finally, the computational cost of running two heavy models sequentially is high. This makes near-real-time inference impractical. As seen in, the authors suggest a unified architecture using geometry as a primary anchor.

Anchoring Synthesis in 4D Geometry

The investigation replaces implicit pixel-space manipulation with an explicit geometric anchor. The authors propose a two-stage pipeline. They first construct a 4D Try-on Proxy. Then, they feed that proxy into a Proxy-Anchored Video Diffusion Transformer (DiT).

The "secret sauce" is the 4D Try-on Proxy. Instead of representing the whole scene as a fragmented point cloud, the authors decouple the human from the environment. They distill 2D try-on priors into a clothed 3DGS-based avatar. 3DGS (3D Gaussian Splatting) uses volumetric primitives to represent high-fidelity textures. This avatar is then animated via SMPL-X sequences (a parametric model of the human body). It is finally aligned into a reconstructed background point cloud .

The DiT does not just guess the next frame. It uses the rendered video from this 4D proxy as a pixel-aligned structural prior. To handle fine-grained identity, they introduce a Cross-view Reference Adapter (CRA) [Figure 3(c)]. This module uses a weight-sharing attention mechanism. It pulls identity cues from the source image into the generative process. This ensures the garment's pattern stays consistent even as the camera orbits.

High Fidelity via Structural Grounding

The results suggest that explicit geometric anchoring is more effective than purely generative approaches. On the ViVid benchmark, the authors report state-of-the-art (SOTA) performance. In the paired setting, they report a $VFID^p_I$ of 9.6085 and a $VFID^p_R$ of 0.1817 (Table 1). These lower values indicate better video-level fidelity and realism compared to older models.

In the challenging unpaired scenario, the model maintains much higher fidelity. This is where the target garment and source video are completely unrelated. Qualitatively, shows that competitors like Magic-Tryon struggle with structural collapses.

For example, Magic-Tryon sometimes turns long trousers into shorts. TryOnCrafter preserves complex silhouettes and intricate textures instead. On the custom CaM-VVTBench, the model achieved an overall score of 75.47. This outperformed two-stage baselines that suffered from limb distortions and background warping .

Implications for Interactive Commerce

If this approach generalizes, it changes the math for digital fashion. We move from watching a video to interacting with a 3D asset. This happens without the massive overhead of manual 3D modeling.

There are three immediate implications. First, decoupling the human from the background allows for "Human Relocalization." A subject can be moved within a scene while maintaining metric-scale harmony [Figure 6(a)]. Second, the ability to perform 360-degree orbital viewing [Figure 6(b)] solves the "side-profile" problem in e-commerce. Third, the 4D proxy is highly reusable. Once built, changing the camera trajectory only requires a fast re-rendering step. This takes roughly 14.3s on an A100 [Table S3]. This is much faster than generating a new video from scratch.

However, the paper notes current limitations. Extreme viewpoint transitions still trigger parallax issues (apparent motion of objects due to camera shift). They also cause inaccuracies in SMPL-X estimation. This can lead to occasional hand-pose misalignments . Additionally, the 14B-parameter DiT backbone is computationally heavy. This makes real-time trajectory editing a difficult goal.

Figures from the paper