Where to Look: Can Foundation Models Reach a Target Viewpoint Through Active Exploration?

Where to Look: Can Foundation Models Reach a Target Viewpoint Through Active Exploration?

Researchers have developed a new way to test if AI can move around a 3D room to match a specific target photo. Most current AI models excel at describing a picture. However, they struggle when they must move their bodies to find that exact perspective. This paper introduces TVRBench, a benchmark designed to bridge the gap between passive seeing and active, embodied spatial intelligence.

The core tension is simple: we have moved from models that answer "what is in this image?" to models that should answer "how do I move to see this image?". Current foundation models are surprisingly bad at the latter.

The Problem

Most spatial intelligence research treats vision as a passive task. We feed a Multimodal Large Language Model (MLLM) a static image. We then ask it to reason about relative positions or 3D layouts. The model does not handle the cost of movement. It does not face the consequences of a bad turn. It simply processes the pixels it is given.

True intelligence requires a closed perception-action loop. This means the ability to observe, act, update a spatial belief, and repeat. In the Target Viewpoint Reproduction (TVR) task, an agent must navigate a 3D environment until its egocentric view matches a target image .

Off-the-shelf models fail catastrophically here. The authors find that even the strongest closed-source models reach only 12.0% success. Humans achieve 93.0% on a similar subset. The failure is structural. As shown in, untrained models fall into two stable failure modes.

They either "walk in circles" by oscillating between adjacent cells. Or they "look in loops" by rotating heads without translating their bodies. The paper identifies a massive bottleneck in mapping spatial discrepancies to body translation (moving the center of mass) rather than just head rotation.

How It Works

To solve this, the authors propose a unified post-training framework. The architecture focuses on two critical components: how the model remembers the past and how it optimizes its trajectory.

1. Visual-Action (VA) Memory: Unlike "Action-Only" (AO) memory, which provides only a text summary of previous moves, VA memory provides the full multi-turn sequence of past RGB observations and actions. This allows the model to maintain a rich visual history. This history is essential for avoiding redundant movements.
2. Visual-Action SFT: The first stage is Supervised Fine-Tuning (SFT) using expert trajectories. These are generated by a rule-based oracle. The authors found that adding Chain-of-Thought (CoT) rationales actually degraded performance in closed-loop control. Instead, they focused on raw visual-action mapping.
3. Multi-turn GRPO: The second stage uses Group Relative Policy Optimization (GRPO), a reinforcement learning (RL) algorithm. GRPO optimizes policies by comparing a group of sampled responses against their mean reward. Unlike "Single-turn" RL, which matches individual actions in isolation, Multi-turn GRPO operates on live simulator rollouts. It uses a trajectory-level reward. This includes a "progress term" that rewards the agent only when it achieves a new minimum distance to the target pose .

This trajectory-level approach is designed to combat the compounding errors seen in per-step optimization.

Numbers

The performance delta is significant. Applying Visual-Action SFT to a 9B Qwen3.5 model raises its success rate from a 2.8% baseline to 50.8%. Adding Multi-turn GRPO provides a further refinement. This pushes the overall success to 51.4%.

The breakdown of these gains is telling. SFT is highly effective at mastering single-room tasks. It reaches 82.4% success in "Single-easy" scenarios. However, it struggles with multi-room navigation. Multi-turn GRPO specifically addresses this weakness. It lifts success in "Multi-easy" tasks from 27.2% to 34.4%.

The authors also highlight a massive "compounding error" problem. In a controlled ablation, they show a major gap between per-step and closed-loop performance. A model might achieve 72.1% accuracy when predicting a single action in isolation. Yet, that same model achieves only 9.4% success in a closed-loop episode. This gap exists because per-step training does not teach the model how to recover from mistakes. Once the model drifts off the expert path, it lacks the training to return.

What's Missing

Several gaps remain. First, the entire benchmark is built in simulation using AI2-THOR and ProcTHOR-10k. It uses a discrete pose grid. In real-world robotics, you deal with continuous control and sensor noise. The "exact pose match" success criterion is a clean mathematical target. It likely won't survive contact with real-world hardware.

Second, the post-training results are tied exclusively to a 9B parameter backbone. We do not know if these gains scale linearly with model size. Larger models might possess inherent spatial priors that make the GRPO phase more efficient. Finally, the paper notes that CoT supervision currently degrades performance. This suggests that current methods for generating "reasoning" for embodied tasks are flawed. We are essentially teaching the model to justify actions rather than reason about spatial geometry.

Should You Prototype This

Yes, if you are working on embodied AI or navigation. The transition from passive perception to active exploration is the next frontier for MLLMs. The findings regarding memory formats are highly actionable. Prioritize Visual-Action memory over textual summaries. The difference in the model's ability to avoid "walking in circles" is too large to ignore.

However, do not rely on Single-turn RL or CoT-heavy SFT for closed-loop tasks. The authors' evidence shows these methods degrade performance. This is a clear warning against treating embodied navigation like a standard text-based reasoning task. If you want to implement this, start with the VA-SFT approach. Move to Multi-turn GRPO only when you have the compute budget for live simulator rollouts.

Code, data, and models are available at https://github.com/aim-uofa/TVRBench.

Figures from the paper