ProMSA:Progressive Multimodal Search Agents for Knowledge-Based Visual Question Answering

Instead of just looking up information once, this new AI agent acts like a researcher. It can look at images or search text multiple times. It corrects itself if it finds the wrong information. It continues until it has enough evidence to answer a question.

This approach addresses a fundamental bottleneck in Knowledge-based Visual Question Answering (KB-VQA). This task requires combining visual scene understanding with external factual knowledge. In many real-world scenarios, the entities in an image belong to "long-tail" distributions. This means they are rare or obscure. Such entities are unlikely to be contained within a model's internal weights (parametric knowledge). Traditionally, this has been solved using a fixed Retrieval-Augmented Generation (RAG) pipeline. In this setup, the model retrieves a set of documents once. It then tries to answer the question. However, the authors of ProMSA argue this "retrieve-then-generate" paradigm is brittle. If the initial retrieval is wrong, the model is trapped in a cycle of reasoning over incorrect evidence.

The brittleness of static retrieval pipelines

Most current KB-VQA methods treat retrieval as a preprocessing step. They do not treat it as an active part of reasoning. They typically employ a fixed top-$k$ setting. This means the model always pulls a predetermined number of snippets. They also use a static modality. They often choose between image or text retrieval before reasoning begins. This creates several points of failure.

First, there is the problem of "retrieval drift." As illustrated in, a standard RAG-based method might retrieve a Wikipedia page that looks superficially relevant.

However, it may be factually incorrect for the specific entity in the image. Once this error is injected into the context, a static model cannot realize it has been misled. Second, these models struggle with multi-hop reasoning. This refers to questions requiring a chain of evidence. For example, a model might need to identify a landmark and then find its location. A single-shot retrieval step rarely captures the breadth of information needed. Finally, the lack of adaptability means models cannot distinguish between simple and complex questions. They may waste compute on easy tasks or fail prematurely on hard ones.

Closing the loop with progressive search

ProMSA reframes KB-VQA as a "budgeted progressive search-and-reasoning" problem. The authors propose an agentic loop. Here, the model functions as a researcher with finite resources. At each step $t$, the model observes the current state $s_t$. This state includes the image, the question, and the accumulated evidence. The model then selects an action $u_t$ from a discrete set: img_search, text_search, or stop.

The architecture relies on two primary tools. The img_search tool uses an EVA-CLIP encoder to find relevant Wikipedia pages. The text_search tool uses a BGE retriever to fetch specific text sections. Crucially, the agent maintains a "deduplication index." This prevents the model from repeatedly visiting the same pages. This allows it to "move past" incorrect candidates to find the truth.

As seen in, the agent does not simply dump raw Wikipedia text into its context.

Doing so would cause a catastrophic context window explosion (where the input becomes too large for the model to process). Instead, it employs a summarization service. This compresses retrieved content into short, question-conditioned snippets. This keeps the reasoning trace manageable. The intelligence emerges from this iterative refinement. The model can trigger an image search to identify an unknown object. Once identified, it can pivot to a text search to find a specific attribute.

Scaling through tool-horizon normalization

The leap from a functional agent to a high-performing one comes from its training. The authors utilize a two-stage process. First, they use a "cold-start" phase. This uses rejection-sampling Supervised Fine-Tuning (SFT) to teach the model basic tool-call syntax. Second, they use a Reinforcement Learning (RL) phase.

The core technical contribution is TN-GSPO (Tool-Horizon Normalized Group Sequence Policy Optimization). In standard sequence-level RL, updates are often normalized by the number of generated tokens ($L$). However, the authors argue that complexity is driven by "interaction depth." This is the number of times the model calls a tool. This is often more important than the sheer word count of its reasoning.

If a model is penalized purely based on token length, it may develop a "length bias." It might favor shorter reasoning paths to stabilize the optimization signal. TN-GSPO solves this. It incorporates the tool-interaction depth $H(\tau)$ into the normalization denominator $D(\tau)$, as shown in . This ensures the policy update scale aligns with the actual decision structure. The effectiveness of this approach is supported by ablation studies in [Table 3]. TN-GSPO consistently outperforms standard GRPO and GSPO methods on both E-VQA and InfoSeek benchmarks.

Evidence of efficacy and remaining blind spots

The empirical results reported by the authors are significant. On the E-VQA benchmark, using a Qwen3-VL-8B backbone, ProMSA achieves an "All" accuracy of 53.4%. This outperforms strong baselines like CC-VQA and REAL. The generalization capability is also noteworthy. The paper reports an accuracy of 85.6% on the OK-VQA dataset. This suggests the learned search policy has acquired a functional ability to navigate external knowledge.

However, there are nuances in the data. While the agent is better at recovering from errors than static RAG, it is not perfect. Looking at the retrieval behavior analysis in [Table 9], the authors note a non-trivial "Others" ratio in the second round of interaction. Specifically, some instances involve the model stopping even when the retrieval is still incorrect. The authors attribute this to the tool-cost penalty ($\lambda$) used during training. This penalty pushes the model to be efficient. There is a tension here. If the penalty for searching is too high, the model may exhibit "premature stop" behavior. It may decide that the risk of an incorrect answer is lower than the cost of searching.

Furthermore, while adding both text and image tools improves performance ([Table 4]), the marginal gains of increasing the top-$k$ retrieval count diminish quickly ([Table 8]). This suggests the bottleneck is not necessarily the quantity of information. Instead, it may be the model's ability to reason over the noise introduced by larger retrieval sets.

Verdict: A blueprint for agentic RAG

ProMSA represents a shift from passive retrieval to active, budgeted agency. By treating search as a sequence of decisions, the authors address the issue of retrieval drift. The introduction of TN-GSPO is an elegant solution to the optimization instabilities of multi-turn agent training.

Is this ready for production? If your use case involves high-stakes, long-tail knowledge, the answer may be yes. The ability to self-correct is worth the increased inference latency. The code is reportedly available at https://github.com/DingWu1021/Promsa. For researchers, the takeaway is clear. When training agents, normalize for the actions taken, not just the tokens produced.

Figures from the paper