Seeing Isn't Knowing: Do VLMs Know When Not to Answer Spatial Questions (and Why)?

Seeing Isn't Knowing: VLMs Struggle to Recognize Unreliable Spatial Information

Current AI models are often too confident when they look at pictures. Even if an object is hidden or the camera angle makes it look distorted, the AI tends to guess an answer. It fails to say "I don't know" or ask to see it from a better angle.

In the field of Vision-Language Models (VLMs)—systems that process both visual pixels and text—spatial reasoning is a core requirement. This is vital for robotics or embodied agents (AI that interacts with the physical world). Historically, benchmarks have focused on whether a model can correctly identify object relationships. They assume the provided image contains all necessary information. The question remains whether these models possess "observational awareness." This is the ability to realize when visual evidence is insufficient or misleading.

This paper argues that current frontier models lack this awareness. They do not just fail to answer spatial questions. They fail to realize they should not be answering them.

The Problem

The status quo in VLM evaluation assumes visual observations are reliable proxies for the 3D world. However, as noted in, 2D projections are fundamentally limited.

Real-world observations suffer from two main issues: occlusion (where an object is hidden behind another) and perspective ambiguity (where the camera angle distorts geometric properties like size).

Existing benchmarks typically reward a model for getting the right answer. They rarely penalize a model for guessing when evidence is missing. This creates a blind spot for engineers. If an autonomous agent relies on a VLM to navigate, an overconfident guess caused by an occluded doorway could cause a collision. Current models treat "Cannot determine" as just another multiple-choice option. They do not treat it as a critical safety fallback.

How It Works

To expose this, the authors developed SPATIALUNCERTAIN. This is a controlled evaluation framework built in 3D simulated environments. It uses AI2-THOR and Holodeck to ensure precision. Unlike messy real-world datasets, this allows for surgical manipulation of the environment.

The framework operates through three main stages:

1. Scene Generation: Using an LLM-based layout generator (Holodeck), the authors create diverse indoor scenes. Because these are 3D simulations, they have perfect ground-truth metadata for every object's position and size.
2. Controlled Perturbation: The authors inject two specific failure modes. For occlusion, they place an "occluder" object in the line of sight of a target [, top].

For perspective ambiguity, they shift the camera to create a viewpoint that biases perception, such as making a distant object appear larger [, bottom]. 3. Multi-Stage Tasking: The evaluation goes beyond simple QA. They introduce ViewSel (picking the best viewpoint from a set) and AbstainViewSel (a two-stage task). In the latter, the model must first correctly identify an unanswerable question and then select an informative alternative view.

By controlling the exact degree of occlusion and camera angles, the authors distinguish between poor spatial reasoning and a lack of awareness.

Numbers

The results across eight frontier models (including GPT-4o, Gemini-3.0-Flash, and Qwen2.5-VL) are stark. Models perform reasonably well when evidence is sufficient. However, they collapse when observations become unreliable.

Under occlusion, models show an average accuracy of around 30% for unanswerable questions. This means they frequently guess instead of choosing "Cannot determine." Under perspective ambiguity, the situation is worse. Accuracy for unanswerable questions drops below 10% [Table 1]. This confirms the "overconfidence" hypothesis. Models effectively hallucinate spatial relationships when the pixels are misleading.

The most telling delta appears in the AbstainViewSel task. Some models show decent standalone viewpoint selection (ViewSel). However, performance craters when they must combine abstention with selection. For example, GPT-5.4 drops from 70.9% in ViewSel to just 22.6% in AbstainViewSel. This suggests that the combined task of recognizing uncertainty and then seeking information is extremely difficult.

Finally, the paper highlights a dangerous asymmetry. Adding visual input helps models detect missing info under occlusion. But it actively misleads them under perspective ambiguity [Table 2]. In the latter case, the "distorted" visual signal acts as a trap. It suppresses the model's ability to abstain.

What's Missing

While the framework is rigorous, there are gaps for production engineers:

• Synthetic Bias: The framework relies on simulated 3D environments. There is a leap between "simulated occlusion" and the noisy, motion-blurred feeds found in real cameras.
• Static vs. Dynamic Uncertainty: The paper treats uncertainty as a static property of a single frame. In reality, uncertainty is often temporal. You might not know an object's location now, but you expect to know it once the robot moves.
• Single-Step Reasoning: The tasks are discrete. The paper does not show how these failures propagate in a long-horizon planning task (a sequence of many actions).

Should You Prototype This

Not yet.

If you are building a consumer chatbot, this is an academic curiosity. But if you are building an embodied agent, this is a mandatory warning.

The paper demonstrates that fine-tuning is a viable path forward. They show that prompting is a limited fix. Prompting often trades off accuracy for caution. Instead, fine-tuning on diverse ambiguity conditions (using LoRA on Qwen2.5-VL-7B) successfully teaches models to abstain [Table 3].

Do not try to "prompt" your way out of spatial uncertainty. If your product requires spatial reliability, you must incorporate observational uncertainty into your training loop. The capability is learnable. You must provide enough examples of being wrong to teach the model when to stay silent.

Figures from the paper