SAAS: Self-Aware Reinforcement Learning for Over-Search Mitigation in Agentic Search

When AI agents search the internet to answer questions, they often search too much. They might keep searching even after they have the answer. This inefficiency slows down response times. It also incurs significant computational costs and increases inference latency (the delay before a model starts responding).

The field of agentic search moves beyond static RAG (Retrieval-Augmented Generation). Static RAG simply fetches documents and generates an answer. Agentic search uses autonomous agents that iteratively reason and search. Ideally, an agent should only trigger a search when its internal parametric knowledge (the information stored in the model weights) is insufficient. It should stop as soon as it collects enough evidence. However, current models struggle with this "self-awareness." They fall into patterns of unnecessary or redundant searching.

The Problem

Most agentic search systems rely on outcome-based reinforcement learning. In this setup, the model is rewarded primarily for the correctness of its final answer. This creates a massive incentive for "over-search." As shown in, outcome-based optimization encourages the model to treat search as a default, beneficial action.

Because external search often improves accuracy, the model learns to rely on it almost exclusively. This causes the ratio of "no-search" trajectories to drop toward zero.

This manifests in two ways. First is question-level over-search. The model triggers a search even when it already knows the answer. Second is step-level redundancy .

The model continues to issue queries even after the necessary evidence is retrieved. Current attempts to fix this using static penalties often fail. These penalties simply dock points for every search call. As the authors show in, a model's capabilities evolve during training.

A fixed penalty can become misaligned with the model's growing knowledge. This leads to "reward hacking." In this scenario, the model avoids necessary searches just to avoid the penalty. This causes performance to collapse.

How It Works

The SAAS framework treats the search boundary as a dynamic property. It must be tracked alongside the model's evolving policy (the strategy the model uses to pick actions). The architecture rests on three pillars :

1. Search Boundary Modeling: The system identifies the boundary for each question $q$. It compares two groups of rollouts (sequences of actions taken by the model). One group is "search-disabled" ($G_d$), using only parametric knowledge. The other is "search-enabled" ($G_e$), using external tools. By comparing success rates, the system categorizes questions into NOSEARCH, NEEDSEARCH, or UNDETERMINED.

2. Boundary-Aware Reward: The reward function is tailored to the category assigned. For NOSEARCH questions, the system applies a zero-tolerance penalty for any search actions. For NEEDSEARCH questions, the system calculates $N_{min}$. This is the minimum number of searches required among successful trajectories in the enabled group. It then only penalizes searches that exceed this $N_{min}$ threshold. This targets redundant steps without suppressing necessary ones.

3. Stage-wise Optimization: To prevent the model from avoiding search entirely, the authors use a sequential curriculum. In Stage I (Capability Acquisition), the model is trained with only outcome-based rewards. This allows it to master reasoning and tool use first. Only after performance stabilizes does the system move to Stage II (Efficiency Refinement). In this stage, boundary-aware penalties are activated to prune search behavior.

Numbers

The goal of SAAS is a better accuracy-efficiency trade-off. The authors evaluate the framework on seven benchmarks using Qwen2.5 models.

On the Qwen2.5-3B-Instruct backbone, SAAS achieves an average accuracy of 45.8%. This surpasses the HiPRAG baseline (43.6%). Simultaneously, it reduces the average search count to 1.13. This is significantly lower than HiPRAG's 1.69. The efficiency gains are even more striking on the larger Qwen2.5-7B-Instruct model. The authors report that SAAS reduces the average search count from 2.94 (in a standard GRPO baseline) to just 0.97. This represents a roughly 67% reduction in search calls. Crucially, it maintains a competitive accuracy of 48.7%.

The impact on over-search metrics is also significant. SAAS achieves a much lower average Question-level Over-search Ratio (QOR) of 45.9% on the 7B model. This is a major improvement over the 100% QOR seen in baselines like StepSearch or HiPRAG [Table 2]. Furthermore, the Step-level Over-search Ratio (SOR) is reduced to 6.3% on the 7B model. This demonstrates that the $N_{min}$-based penalty successfully teaches the agent when to stop.

What's Missing

There are gaps that a practitioner should consider.

First, the evaluation is strictly unimodal. The paper focuses on textual retrieval. Real-world agentic workflows often require interacting with multimodal evidence like tables or images. While the authors argue the framework is theoretically agnostic to modality, it hasn't been tested against visual or structured data.

Second, the paper does not report the computational cost of the training process. Search Boundary Modeling requires generating dual groups of rollouts. This means the system generates both search-disabled and search-enabled trajectories for every question. This likely increases the number of rollouts required per training step.

Finally, the sensitivity to the $\delta$ hyperparameter is noted. This parameter determines the threshold for identifying the search boundary. The paper does not explore how this behaves in extremely long-horizon reasoning tasks. In such tasks, the "minimum sufficient search" might be highly variable.

Should You Prototype This

Yes, if you are building high-throughput agentic search systems.

If your bottleneck is the cost of external search APIs, SAAS is a strong candidate. It offers a way to prune costs without using blunt, fixed penalties. The stage-wise curriculum is a vital takeaway. Do not try to optimize for efficiency until your agent has mastered basic tool use.

The code is reportedly available at https://github.com/XMUDeepLIT/SAAS. If you can manage the increased training compute for dual-rollout modeling, the inference savings are significant. The improved accuracy-efficiency profile makes this worth investigating.

Figures from the paper