S-Agent: Spatial Tool-Use Elicits Reasoning for Spatial Intelligence

Beyond the Static Snapshot: Building Spatial Intelligence Through Evidence Accumulation

Current AI models struggle to understand 3D space from videos. They typically only look at single, isolated frames. While a human can watch a person walk around a table, they maintain a mental map. Most Vision-Language Models (VLMs)—AI that processes both text and images—treat each frame as a disconnected snapshot. This makes it nearly impossible to reason about moving objects or precise geometric measurements.

Existing approaches try to solve this by training models on massive 3D datasets. Others connect VLMs to external Python scripts to calculate geometry. However, these methods often remain "stateless." This means they lack a persistent memory of past observations. They essentially try to guess the 3D layout from a single 2D projection. This task is mathematically underspecified and prone to errors.

A new study introduces S-AGENT. This paradigm shifts the focus from single-shot prediction to "spatio-temporal evidence accumulation." Instead of asking a model to guess a location, the authors propose an active process. A VLM acts as a semantic planner (a high-level controller that decides which actions to take). It decides which specialized tools to use to gather evidence over time.

The limitations of frame-centric reasoning

Most modern VLMs suffer from a "semantic-to-geometric gap." They excel at identifying that a "chair" exists (semantic inference). However, they struggle to determine its exact distance from a "table" (geometric grounding). Because they are primarily trained on 2D internet images, they lack an inherent sense of depth and scale.

As shown in, current spatial VLM models often rely on single-shot predictions. They process a single image or a compressed video summary. Then, they attempt to output a spatial fact immediately. This approach fails in complex scenarios. For example, an object might be partially occluded (hidden by another object). Or, it might only be visible from one specific angle. If the model cannot see the whole object in one frame, it cannot "look around" to reconstruct a coherent 3D scene.

Orchestrating a hierarchy of spatial tools

The S-AGENT framework treats spatial reasoning as an iterative investigation. The architecture separates high-level "thinking" from low-level "measuring." It uses three distinct layers of operation, as detailed in .

1. Level 1: 2D Visual Evidence. The VLM acts as a planner. It requests tools to ground objects in 2D. This involves identifying specific regions of interest. It also involves selecting the most informative frames from a video stream.
2. Level 2: 2D-to-3D Geometric Lifting. Once objects are identified in 2D, the agent "lifts" that information into 3D. This uses metric depth estimation (calculating actual distance) to recover 3D coordinates and camera poses. This allows the agent to reason beyond the flat image plane.
3. Level 3: Spatial Knowledge Aggregation. Raw 3D data is often too dense for a language model to interpret. To solve this, the authors employ specialized "experts." Examples include a Metric Measurement Expert or a Relative Position Expert. These experts convert complex geometric signals into structured facts. They turn raw data into statements like "the distance is 1.2 meters."

Crucially, this process is supported by a dual-memory system. Scene Memory acts like a persistent mental map. It stores grounded entities and their geometric attributes. This prevents the agent from re-identifying the same object every time the camera moves. Agent Memory functions like a reasoning log. It records which tools were called and what the results were. This prevents the model from repeating redundant tool requests.

Scaling intelligence through trajectory distillation

The authors demonstrate that this agentic framework can "teach" smaller models. They do not just use S-AGENT as an inference-time boost for large models like GPT-5.4. Instead, they use a high-performing "teacher" S-AGENT to generate successful reasoning trajectories.

They fine-tune a compact 8B parameter model on this synthetic data. This dataset is called S-300K. The results are significant. The authors report that S-AGENT-8B achieves a 10.5% improvement on the MMSI-Bench benchmark. Accuracy jumped from 31.1% to 41.6% compared to the base model. Remarkably, this small 8B model performs comparably to much larger, closed-source models like Gemini 3 Pro and GPT-5.4 [Table 4].

In the zero-shot setting (where no extra training is performed), the framework improves existing models. For instance, it improves Gemini 3 Pro by 1.2% and GPT-5.4 by 4.5% on MMSI-Bench [Table 1]. These improvements represent a measurable gain in spatial accuracy for established models.

Complexity costs and interpretive bottlenecks

The paper reveals a clear trade-off in how information is processed. The authors note in their ablation studies (tests that remove parts of a system to see what they do) that Level 2 provides limited benefit. This happens when raw 3D lifting is fed directly to the VLM. Raw 3D evidence contains dense numerical data like camera poses and point clouds. This data can actually distract the planner.

This highlights a critical dependency. The system's success relies heavily on the Level 3 experts to act as a translator. Without these experts, the "intelligence" of the agent is bottlenecked. The model cannot easily parse raw mathematical coordinates into useful logic. Furthermore, the computational overhead is higher than a single-pass prediction. Running multiple rounds of tool calls and depth estimations takes more resources than a standard model.

The verdict: A blueprint for embodied AI

S-AGENT is an effective paradigm for building agents that operate in the physical world. This includes robotics or AR/VR systems. By moving toward a "collect and verify" workflow, the authors provide a way to imbue VLMs with spatial awareness.

The ability to distill these reasoning patterns into a compact 8B model is very practical. It suggests that we do not always need infinitely larger models. We might just need better ways to teach them to use tools. Code and model weights are reportedly available via the Ropedia/S-Agent project page.

Figures from the paper