VideoSearch-R1: Iterative Video Retrieval and Reasoning via Soft Query Refinement

If you have ever built a video search pipeline, you know the frustration of the "one-shot" failure. Most systems attempt to retrieve a video from a massive corpus in a single pass. They then hand that video to a reasoning model to find a specific moment. But if the initial retrieval misses the mark, the entire downstream reasoning process is dead on arrival. This happens if the query was too coarse or lacked a specific visual cue. There is no feedback loop to say, "That wasn't quite right; let me look for something else."

Current video understanding models generally split the problem into two decoupled stages. First is inter-video reasoning (finding the right video in a huge corpus). Second is intra-video reasoning (pinpointing the exact timestamp within that video). As shown in, existing architectures typically treat retrieval as a static preprocessing step.

When the retriever returns a video that is superficially similar but semantically incorrect, the error propagates. The reasoning module assumes the video is correct. It then attempts to find a non-existent event, leading to total system failure.

The decoupling bottleneck

The status quo relies on a rigid pipeline. Here, the retriever and the reasoner operate in isolation. Standard approaches use dual-encoder models (systems that encode videos and text separately) to rank videos. They use cosine similarity (a measure of how similar two vectors are) in a shared embedding space. While efficient, these models often capture only coarse-grained alignments. Some researchers use Vision-Language Models (VLMs, models that understand both images and text) to rerank the top-$K$ candidates. However, the fundamental flaw remains: the system is reactive, not iterative.

If the top-ranked video doesn't match, the system cannot adjust its search parameters. This is particularly painful in Video Corpus Moment Retrieval (VCMR) tasks. In these tasks, a query might require highly specific attributes. For example, a query might specify a "man in a dark gray t-shirt" instead of a "man in a light gray t-shirt." As illustrated in, a single mismatched attribute renders subsequent temporal grounding (locating a specific time range) impossible. Current agentic frameworks often bypass this. They assume the video is already provided, making them useless for large-scale corpus searches.

Continuous latent query refinement

VideoSearch-R1 breaks this cycle. It introduces an agentic loop that treats retrieval as a multi-turn conversation. Instead of traditional "hard" query refinement—where a model rewrites a query into new discrete text—the authors propose Soft Query Refinement (SQR).

In SQR, the model generates $N$ soft query tokens directly in the continuous latent space (a mathematical space of hidden numerical representations). These tokens are learned embeddings. They are appended to the original query to steer the search engine. The process follows a structured loop: 1. Retrieval: The system calls a video search engine using the current query $q_t$. 2. Verification: The model examines the retrieved video $v_t$. It generates a reasoning trace to determine if it matches the query. 3. Decision: If the model outputs <answer>not matched</answer>, it triggers the <REFINE> token. 4. Refinement: The model generates $N$ soft tokens ($q^{soft}t$). These are concatenated with the original query to form $q_t]$.} = [q_1 \parallel q^{soft

This approach avoids the semantic noise of text-based rewriting. As seen in, hard query refinement (HQR) requires a verbose string of text to express nuance.

Conversely, SQR uses a compact set of latent vectors for fine-grained adjustments. To optimize this, the authors use Group Relative Policy Optimization (GRPO). This is a reinforcement learning (RL) algorithm. It uses task-level rewards—including retrieval accuracy and temporal IoU (Intersection over Union, a metric for how well two time ranges overlap)—to train the model.

Efficiency gains in the latent space

The empirical results suggest that operating in the latent space is more efficient. The authors report that SQR achieves state-of-the-art performance across three VCMR benchmarks: ActivityNet-FIG, Charades-FIG, and DiDeMo-FIG.

On the DiDeMo-FIG dataset, the impact on retrieval is significant. While a standard fine-tuned baseline struggles, VideoSearch-R1 achieves a 0.7/R@1 of 19.7. This means the model correctly identifies the ground-truth video as the top choice 19.7% of the time for high-overlap requirements. Perhaps the most compelling metric is the token economy. The paper finds that SQR reaches superior retrieval performance using only 8 soft tokens. In contrast, hard query refinement (HQR) requires an average of 26.8 tokens to achieve lesser results [Table 5]. This reduction in tokens can lead to lower inference costs.

Furthermore, the effectiveness of the iterative loop is quantified in .

Performance on VCMR tasks improves steadily from the first to the second turn. It eventually saturates at $T=3$. This means you do not need an infinite loop to succeed. A shallow multi-turn depth is sufficient to correct most initial retrieval errors.

Unverified scaling and saturation

Despite the strong results, certain questions remain. First, the paper notes that performance saturates at $T=3$ turns . This implies diminishing returns on computation for extremely complex queries.

Second, the relationship between the capacities of the search engine and the SQR module is not fully explored. The authors show that scaling the search engine from 2B to 8B parameters helps [Table 4]. However, they also note that scaling the search engine alone does not solve the inherent weaknesses of hard query refinement. This supports their argument for SQR. Yet, it remains unclear if a much larger search engine would eventually make soft refinement unnecessary.

Finally, the training complexity is high. The two-stage pipeline starts with Supervised Fine-Tuning (SFT, training a model to mimic specific patterns) using a heavy teacher model. This is followed by GRPO. The authors do not explicitly detail the total GPU-hours required for the RL stage. This is important because RL is often the primary bottleneck in deploying agentic systems.

The verdict

VideoSearch-R1 is a step forward for large-scale video indexing. It moves from "retrieve-then-reason" to "retrieve-while-reasoning." This addresses the structural fragility of current video pipelines. The move to Soft Query Refinement is a highlight. It is a more elegant way to handle semantic drift than brute-force text rewriting.

If you are building a system for massive video archives with high specificity, this is worth a prototype. The token savings alone make the latent-space approach attractive for production costs. However, if your video corpus is small, the overhead of a multi-turn RL-trained agent may be unnecessary.

Code and model checkpoints are reportedly available at mlvlab.github.io/VideoSearch-R1.

Figures from the paper