On Locality and Length Generalization in Visual Reasoning

Beyond the Single Snapshot: Why Vision Models Struggle with Complexity

Most modern AI models look at an entire image all at once. They attempt to digest every pixel in a single, global computation. This approach works for identifying a cat in a photo. However, it often fails when tasks require counting or tracking information across a complex scene. A new study from Qualcomm AI Research suggests this "global" habit causes models to struggle with harder problems.

The researchers propose moving away from single-shot processing. They suggest a system of local, sequential glimpses—similar to the human eye. By combining high-resolution local views with a low-resolution "peripheral" sense and a recurrent memory, the authors report models can solve visual reasoning tasks. These tasks are significantly more complex than anything the models saw during training.

The Shortcut Problem in Visual Reasoning

The core issue addressed by the paper is "length generalization." This is the ability of a model to perform a task on a sequence of items longer than those seen during training. Imagine teaching a child to count to ten. If they truly understand "plus one," they should count to twenty. If they have merely memorized the sounds of numbers one through ten, they will fail at eleven.

The authors argue that current vision models memorize patterns through global shortcuts. Because these models see the whole image at once, they learn to associate specific global layouts with an answer. They do not learn the underlying logic of the task. When the task gets longer or the image gets more crowded, these shortcuts break. Consequently, the model's performance collapses.

The Mechanics of Sequential Sight

To understand the study's approach, we must define two concepts: recurrence and foveation. Recurrence refers to a model's ability to use a hidden state. This is a sort of internal mental notepad used to carry information from one step to the next. This is common in Recurrent Neural Networks (RNNs), which process data like a ticker tape. They update their "memory" with every new piece of information.

Foveation is a biologically inspired way of perceiving the world. Instead of seeing everything at once in high detail, the human eye uses a "fovea" (the center of the retina) to see small areas in sharp focus. The rest of the field is perceived as a blurry, low-resolution periphery. This allows us to navigate a room using the periphery while reading a word using the fovea.

The authors introduce a model called FOVEAGENT-LSTM. As shown in, the model does not take one big picture.

Instead, it receives a high-resolution "local" glimpse of its current focus. It also receives a low-resolution "peripheral" glimpse to help it decide where to move next. It then uses an LSTM (a type of recurrent network) to update its internal state based on these glimpses.

Testing the Limits of Logic

The researchers built a suite of visual reasoning tests. These included "Visual Parity," where a model determines if the number of "on" switches in an image is even or odd. They also used "State Machine," a task where the order of visiting switches changes the outcome .

The results were striking. The authors report that standard, global-view vision-language models (VLMs) achieve high accuracy when the task is simple. However, their performance drops sharply as the number of switches increases .

In contrast, the FOVEAGENT-LSTM maintained high accuracy even when the task length exceeded the training range .

The study also investigated a trade-off regarding glimpse size. Larger glimpses allow a model to cover more ground quickly. This helps with learning. However, they also provide too much information at once. This allows the model to fall back on "global shortcuts." The authors find an optimal balance exists. The peripheral view must be large enough for navigation. It must also be low-resolution enough to prevent the model from "cheating" .

Furthermore, the paper demonstrates that this is not just about having more pixels. In a "Finding Roots" task involving mathematical plots, the authors compared their foveated model to a global model. They matched the total number of visual tokens (the computational cost). The authors report the foveated approach nearly doubled the accuracy of the baseline for the same compute budget . This means for complex reasoning, it is more efficient to spend compute on high-resolution local details than on a uniform, blurry global view.

Redefining Visual Intelligence

These findings change how we think about the requirements for robust AI. The paper suggests that "locality of perception" is an essential requirement for compositional generalization. This is the act of gathering information through small, focused windows. Without it, models are prone to superficial pattern matching.

The research also distinguishes between two different types of visual intelligence. In a "Recall" task, a model simply finds an object in a cluttered scene. Global models actually perform better than foveated models here . This implies that while global sight is excellent for retrieval, local, recurrent sight is the key to true logical reasoning and state tracking.

Where the Edges Are

Despite the promising results, the authors note several limitations. Currently, the FOVEAGENT models are trained using imitation learning. This means they essentially watch an "expert" navigate the image. Learning to develop these navigation policies from scratch remains a significant challenge.

Additionally, while the model excels at extending task length, the "Finding Roots" results represent a moderate form of extrapolation. This is somewhat different from the extreme, systematic length generalization seen in the synthetic parity tasks. The paper does not explore how these models might behave in dynamic, video-based environments.

Figures from the paper