Holo-World: Unified Camera, Object and Weather Control for Video World Model

Researchers have developed a new AI model that can take a single image and turn it into a video. You can control how the camera moves, how objects behave, and even change the weather—like adding rain or snow. Unlike previous tools that require a full source video to begin an edit, this model works from just one starting picture. It does this while keeping the scene looking consistent.

The field of video world models is moving toward creating dynamic, controllable environments. Ideally, a user should be able to steer a camera through a scene, move objects, and modify the environmental state simultaneously. However, current approaches treat these controls as isolated axes. Camera control methods focus on viewpoint shifts. Object control methods steer specific entities. Weather generation typically relies on existing video-to-video editing. In those cases, a complete source video already provides the necessary future structure and motion.

This fragmentation creates a significant gap. How do you generate a video from a single image that follows specific camera paths and object motions while also transforming the scene into a new weather state? A new study introduces Holo-World to solve this "first-frame source-to-state" problem. It aims to unify these disparate controls into a single interface.

The struggle for unified control

Current video generation architectures suffer from a fundamental modeling conflict. To generate a video, a model must balance two opposing goals: world preservation and state transfer. World preservation ensures the background, geometry, and object layouts remain consistent as the camera moves. State transfer requires the model to change the appearance of those same elements. For example, it might turn a dry road into a wet, reflective surface during a rainstorm.

As shown in, controlling these elements jointly is difficult.

Most weather-oriented models are formulated as video-to-video editors. They are effective if you already have a video of the scene. However, they fail in the "source-to-state" setting. In this setting, the model must synthesize the future layout, motion, and temporal continuity from a single image. If a model applies heavy weather effects using standard techniques, it often disturbs the underlying scene. This causes the background to drift or objects to warp.

Factorizing the scene and the weather

To resolve this conflict, the authors propose the Unified Scene Adapter (UniSA). This operates within a frozen video backbone (a pre-trained neural network that remains unchanged during training). Instead of forcing one set of parameters to learn everything, UniSA factorizes the task into two distinct, parameter-disjoint residual subspaces (separate mathematical pathways for different tasks). Think of this like a dual-track audio mixer. One track handles the "background music" (the stable world structure) while the other handles the "vocal effects" (the weather transformations).

The architecture consists of two primary components: 1. The World Adapter: This learns preservation residuals. It uses rendered background controls—specifically RGB, depth, and normal maps—to act as geometric anchors. These anchors ensure the scene structure stays rigid. This prevents the "floating texture" artifacts common in many generative models. 2. The State Adapter: This learns weather-transfer residuals. It takes the source appearance and the target weather text to model how particles and atmospheric effects should appear.

As illustrated in, these two adapters feed their respective "hints" into the frozen DiT (Diffusion Transformer) blocks of the backbone.

Furthermore, the authors introduce Scene-Weather Decomposed CFG (SW-CFG) to manage guidance during sampling (the process of generating the video from noise). Rather than using a single global scaling factor, SW-CFG calculates a separate "scene residual" and a "weather residual" .

This allows the model to strengthen the weather effects without losing the integrity of the original scene.

Evidence of consistent transformation

The authors evaluated Holo-World using a custom dataset called HoloStateData. They built this to provide aligned supervision for camera, object, and weather controls .

To prove the model's effectiveness, they tested it on two distinct fronts: world preservation and weather transfer.

On the "Real" subset, the goal is to follow camera and object instructions without changing the weather. The authors report that Holo-World achieves a VBench-I2V overall score of 89.05. This outperformed several specialized baselines like Uni3C and VerseCrafter. The model also showed high precision in motion. It reported a camera rotation error (RotErr) of 0.719 and a translation error (TransErr) of 1.123 [Table 2]. These low error numbers mean the generated camera paths stay very close to the intended targets.

On the "Weather" subset, the results were even more striking. Despite having much weaker "source evidence" than video-to-video baselines, Holo-World achieved an 86.00% Weather Alignment rate. It also earned a VLM (Vision-Language Model) evaluation score of 68.51 [Table 3]. In blind human preference tests, users preferred Holo-World over established weather-editing baselines like Cosmos-Transfer2.5 in 83.00% of cases. Qualitative results in show that the model successfully transitions scenes into snow, fog, or rain.

Throughout these changes, the underlying road geometry and camera path remain strictly consistent with the input.

Limitations in simulation and scope

While the results are impressive, the paper does not claim to have built a perfect physical simulator. The authors explicitly state that the scope of Holo-World remains "controllable video generation" rather than a full-scale physics engine. This distinction is important for practitioners. While the model can simulate the appearance of rain or snow, it is not necessarily calculating the fluid dynamics or the physical impact of snow according to Newtonian laws.

Additionally, the reliance on rendered geometry buffers (depth and normal maps) suggests a dependency on initial estimations. If the initial monocular depth estimation is inaccurate, the anchors provided to the World Adapter may be flawed. This could lead to structural inconsistencies in the generated video.

The verdict: A new standard for world modeling

Holo-World represents a significant step toward truly interactive video world models. By separating the "what" (the scene structure) from the "how" (the environmental state), the authors have addressed the trade-off between creative editing and structural stability.

For engineers working on autonomous driving simulations or high-fidelity digital twins, this approach offers a path toward generating diverse environmental scenarios from simple snapshots. While it is not a replacement for a dedicated physics engine, it provides a tool for rapid, high-quality visual prototyping. Code and project details are reportedly available at https://xiangchenyin.github.io/Holo-World/.

Figures from the paper