ActWorld: From Explorable to Interactive World Model via Action-Aware Memory

Beyond Walking Through Virtual Rooms

Current AI world models are excellent at simulating the sensation of moving through a space. However, they are remarkably poor at actually doing anything within it. We have seen rapid progress in models that simulate long-horizon navigation (the ability to move through a scene over a long period). These systems allow a user to "walk" through a generated environment. Yet, they are largely confined to locomotion and viewpoint control. You can turn corners and traverse landscapes. But the moment you try to pick up a plate or open a door, the illusion shatters.

The field has reached a plateau where worlds are visually explorable but not truly actionable. This gap exists because existing models focus almost exclusively on motion. They handle actions like walk, turn, or look around. They leave object-level interaction to specialized models or slow, offline video generation. A new paper, ActWorld, argues that this "navigation–interaction gap" is caused by two specific failures. These are a lack of dense interaction data and a memory architecture that forgets the most important moments.

The Action-Forgetting Pathology

The status quo in world modeling relies on recency-biased memory. To keep generation computationally tractable (efficient enough to run), models typically organize history into temporal buckets. These consist of short-term, mid-term, and long-term storage. Older frames are compressed to save space. This works well for roaming. If you are just walking, the most relevant information for your next step is what you saw a few frames ago.

However, interactive generation breaks this assumption. The causal drivers of an object's state are sparse, transient events. These include the precise moment a hand makes contact with a bottle. In a standard recency-based system, these critical "event-transition" frames are often aggressively compressed. They may even be evicted from the buffer as the model continues to navigate. The authors identify this as "action-forgetting." The model loses the visual evidence required to predict subsequent object states. This happens because the interaction occurred too many chunks ago. As shown in the qualitative comparisons in, existing baselines like Yume 1.5 or HY-World 1.5 frequently lose track of manipulated objects.

They fail to follow complex sequences like "pour then stir" because their memory focuses on scenery rather than interaction physics.

Hierarchical Action-Aware Memory

To solve this, the authors propose a hierarchical memory design. It prioritizes interaction importance over mere temporal recency. The architecture, detailed in, moves away from simple time-based bucketing through three primary mechanisms:

1. Event-Aware Frame Re-assignment (EAFR): Instead of grouping frames by how long ago they occurred, EAFR ranks them by an importance score. This score combines a phase prior (which peaks during "contact" and "manipulating" phases) with a decay function for recency. This allows a high-impact contact frame from ten seconds ago to stay in the high-fidelity Sfine bucket. Meanwhile, a recent but boring stretch of hallway drops to Scoarse.
2. Action-Conditioned History Amplification (ACHA): Within the Transformer blocks, the authors implement a mechanism to sharpen self-attention (the process by which a model weighs the relevance of different parts of its input). Using a learned embedding of the current action class, ACHA applies a multiplier to the history keys. This tells the model to pay extra attention to past frames related to the current action.
3. Persistent Action-Aware Memory Bank: To combat the eviction horizon (the limit at which old data is deleted), the authors add a small, structured channel of up to 16 tokens. This bank survives indefinitely. It stores two types of information: Event tokens, which summarize state transitions like "the grip was released," and Object tokens. These use DINOv3 (a vision transformer feature extractor) to anchor the visual identity of touched objects. Because these tokens are pinned during interaction phases, the model can "remember" an object's appearance even after the camera turns away.

Breaking the Interaction Barrier

The authors validate ActWorld on a new long-horizon benchmark called I-Bench. This benchmark forces models to interleave navigation with object manipulation. The results suggest that the memory redesign is highly effective.

The paper reports that ActWorld achieves a 57.8% success rate for Level-3 instructions on I-Bench. This represents the rate at which the model fully completes a complex task. This success rate is more than double the performance of several baselines. Specifically, in terms of semantic instruction following (measured by a VLM-based judge), ActWorld hits a mean score of 2.557. This significantly outperforms models like Matrix-Game 3, which scores only 0.295.

Crucially, the authors demonstrate that this doesn't come at the cost of movement. In the Key-Mouse-Following (KMF) metric, which measures how accurately the model follows geometric camera commands, ActWorld achieves a joint accuracy (Accfull) of 20.62%. This indicates the model maintains high geometric controllability. The qualitative results in and illustrate this duality.

They show continuous trajectories that seamlessly transition from WASD-driven movement to complex tasks like "pickup" and "wipe."

Implementation Costs and Blind Spots

There are practical trade-offs to consider. The persistent memory bank requires an online inference step. The model must VAE-decode (convert compressed data back into pixels) the generated chunk. It then runs a forward pass through a frozen DINOv3 encoder to extract object anchors. The authors report that this adds an average overhead of less than 15% to the chunk-generation budget. While manageable, this increases latency compared to pure navigation models.

The model's capability is heavily tied to its 100K-video interaction dataset. The reliance on Chain-of-Thought (CoT) prompting via GPT-5.4 for per-chunk annotation is a clever way to generate dense labels. However, this suggests the model's "intelligence" regarding physics is partially a reflection of the high-quality semantic supervision it received. The paper does not explore how the model behaves in "out-of-distribution" physics scenarios. For example, it does not test interactions with objects that have significantly different mass or friction properties than those in the training set.

The Verdict

ActWorld is a successful proof of concept. It proves that the "action-forgetting" problem is a structural memory issue rather than a lack of model capacity. By decoupling low-level camera control from high-level semantic actions, the authors have bridged a significant gap in interactive simulation.

If you are building real-time AI-generated environments, this architecture is a clear signal. Move away from simple temporal buffers. The ability to persist object identities and interaction events across long rollouts is a prerequisite for meaningful world models. Code and project details are reportedly available at https://interactwm.github.io/ActWorld.

Figures from the paper