Thinking with Visual Grounding

Can Models Prove Their Reasoning?

New AI models are increasingly capable of "thinking" through complex problems by generating long chains of natural-language reasoning. However, for vision-language models (VLMs), this thinking often happens in a vacuum. While a model might correctly state that "the blue mug is near the chair," it doesn't explicitly show which pixels constitute the mug or the chair. This leaves the reasoning trace implicit. This makes it difficult for engineers to verify if the model is actually looking at the right objects or just hallucinating a coherent-sounding explanation.

The researchers behind "Thinking with Visual Grounding" argue that visual thinking shouldn't just sound right; it should show its evidence. They propose a method where models interleave their linguistic thoughts with explicit coordinate-based grounding. This uses either points or bounding boxes (rectangular areas enclosing an object) directly in the reasoning stream.

The problem of implicit evidence

The core question the authors investigate is whether tying intermediate reasoning steps to specific image regions improves both the accuracy and the verifiability of VLM reasoning. In current state-of-the-art reasoning traces, the connection between a verbal claim and its visual support is invisible. As noted in, a standard natural-language trace might describe a black laptop on a table.

However, it lacks the spatial metadata to prove the model has localized that laptop.

This lack of grounding creates a massive supervision gap. If a model's reasoning is purely textual, we can only reward it for the final answer being correct. We cannot easily penalize it for "right answer, wrong reason." This occurs when a model arrives at the correct conclusion based on a completely incorrect visual premise. This makes the internal logic of the model a black box. It is notoriously difficult to debug or fine-tune for precision.

Cracks in the pure-text approach

Before this work, the prevailing wisdom was that increasing the length and complexity of textual reasoning traces was the primary lever for improving problem-solving. For VLMs, this meant encouraging models to "think" more in text before answering.

The authors identify a significant crack in this logic. Visual reasoning is fundamentally constrained by the image. Text alone cannot capture the necessary spatial nuances. Furthermore, they observe that attempting to train models via reinforcement learning (RL, a method where models learn through trial and error guided by rewards) using only text-based reasoning often leads to "length collapse." In their experiments, the non-grounded thinking baseline saw its response length decrease linearly during training. This effectively killed the model's ability to explore complex reasoning paths. Without the structural anchor of grounding tags, the model loses the "local structure" required to maintain a coherent, multi-step thought process during the RL rollout.

Building a grounded reasoning engine

To move beyond pure text, the authors developed a multi-stage investigation centered on a scalable data synthesis pipeline. The goal was to create a dataset where every reasoning step is paired with high-fidelity visual evidence.

The process starts by distilling correct reasoning traces from existing VLMs like Qwen3-VL-Plus. Once a correct trace is identified, an LLM extracts the essential visual objects mentioned. The heavy lifting happens in the "agentic visual grounding" stage. Instead of relying on noisy VLM predictions, the authors use a SAM3-based (Segment Anything Model, a tool for identifying object shapes) agent. This agent iteratively probes the image. It proposes noun phrases, requests masks (pixel-level outlines of objects) from SAM3, and verifies them against the raw image .

Once they have high-quality masks, they derive two distinct supervision modes: box mode (using $[x_1, y_1, x_2, y_2]$ coordinates) and point mode (using a single $[x, y]$ coordinate). Crucially, they don't just use this for Supervised Fine-Tuning (SFT, training on labeled examples). They implement a grounding-aware RL framework using GRPO (a reinforcement learning algorithm). Because the model might name an object differently than the ground truth, they use a lightweight VLM "router" to match model-generated tags to the ground-truth objects .

This allows them to provide dense rewards based on Intersection-over-Union (IoU, a measure of how much two shapes overlap) for boxes or F1 scores (a measure of accuracy for discrete classifications) for points.

Small models, big reasoning gains

The findings are surprisingly potent. The authors report that adding visually grounded thinking to a relatively small Gemma3-4B-IT model yields massive improvements in counting and spatial reasoning.

Most notably, the 4B visually grounded models achieve performance comparable to, and in some cases superior to, the much larger Gemma3-27B-IT model on spatial reasoning benchmarks [Table 2]. For example, on the SpatialMQA benchmark, the 4B model outperformed the 27B model. On counting tasks, the results are similarly robust. The point-mode grounded model reached 39.31% accuracy on TallyBench [Table 1]. This represents a significant jump from the 21.73% accuracy seen in the non-grounded thinking baseline.

The ablation studies reveal a nuance in how these modalities function. Point grounding is exceptionally well-suited for counting tasks. This is likely because it provides a compact way to identify individual instances without the overhead of defining precise boundaries. Conversely, box grounding provides richer geometric signals. These include object extent and overlap. Such signals are vital for complex spatial tasks. The authors note that the grounding reward provides a consistent boost for box-mode RL. This is particularly true when the task requires resolving fine-grained spatial relationships.

Implications for the prod path

If these results generalize, the implication is that we can achieve "frontier-class" reasoning capabilities in much smaller parameter footprints. This is achieved by shifting the burden from raw model capacity to structured, grounded reasoning. For practitioners building edge-deployed VLMs, this suggests a new direction. Investing in high-quality, grounded synthetic data might be more cost-effective than chasing larger model weights.

There are two immediate takeaways for system design: 1. Mode Selection Matters: If you are optimizing for counting (e.g., inventory management), prioritize point-based grounding. This minimizes the difficulty of the optimization landscape. If you are optimizing for spatial navigation, use bounding boxes. This provides the necessary geometric context. 2. Grounding Stabilizes RL: The presence of grounding tags acts as a structural regularizer. It prevents the "length collapse" seen in pure-text reasoning. This makes RL training more stable and predictable.

The obvious follow-up is to test this on dynamic or video-based reasoning. If a model can ground a moving object in a temporal sequence, it could transform how we supervise autonomous agents in real-world environments.

Figures from the paper