Most AI models try to guess what you want when your instructions are vague. This leads to expensive mistakes. In the field of referring segmentation—where an AI must identify and mask a specific object in an image or video based on a text prompt—current models operate under a dangerous assumption. They assume the user's query is perfectly precise.
In reality, users are rarely that careful. They might ask to "segment the red car" when there are three red cars in the frame. Current models respond to this ambiguity by arbitrarily guessing which object the user intended. This often results in incorrect masks. This paper proposes a shift from passive recognition to active interaction. Instead of guessing, the IC-Seg framework uses a multi-turn dialogue. It proactively asks the user for the missing details needed to resolve the ambiguity before any segmentation occurs.
The Problem
The status quo in referring segmentation is a single-pass inference pipeline. The model receives a prompt and immediately attempts to localize and mask the target. This works well if the prompt is "the blue bird on the branch." However, it breaks down in complex scenes.
As shown in, when a query lacks sufficient detail to distinguish between multiple candidates, existing models tend to pick a target based on internal biases.
This "guessing" behavior is a fundamental failure mode for any system requiring high precision. Even when researchers have applied reinforcement learning (RL)—a method where models learn through trial and error—the models often fall into traps. They generate repetitive, uninformative conversation that fails to actually reduce the search space of possible targets .
How It Works
The authors propose IC-Seg, an agentic framework that treats segmentation as a collaborative process. The architecture moves away from the single-pass paradigm toward a dynamic, multi-turn interaction loop .
The process follows three main stages: 1. Clarification Loop: The policy MLLM ($\pi_\theta$) reasons over the visual and textual input. If it detects ambiguity, it generates a clarifying question. To train this, the authors use a User Simulator ($\pi_B$). This is an MLLM that acts as a human by providing feedback based on the ground-truth target. 2. Intent Confirmation: The dialogue iterates until the ambiguity is resolved. The model uses these turns to prune the list of potential candidate objects. 3. Segmentation: Once the intent is confirmed, the model triggers the segmentation engine. For video, it selects a keyframe (a representative frame from a video) and outputs a bounding box and a point prompt. These are processed by a frozen SAM2 (Segment Anything Model 2) to generate the final masks.
To solve the "sparse reward" problem in RL, the authors introduce Hi-GRPO. Sparse rewards occur when a model only gets feedback at the very end of a long sequence. Hi-GRPO is a hierarchical optimization strategy. It injects supervision at three distinct granularities: * Trajectory Level: Rewards the model for achieving high final segmentation accuracy. Accuracy is measured via IoU (Intersection over Union, a metric for overlap) and bounding box precision. * Turn Level: Incentivizes "good" dialogue. It rewards entropy reduction (shrinking the candidate set) and interaction efficiency (avoiding redundant questions). * Step Level: Uses "expert diagnostic feedback" from the simulator. This provides token-level guidance (feedback on individual words or symbols). It tells the model exactly which parts of its reasoning contributed to success or failure.
Numbers
The authors evaluate IC-Seg on a new benchmark, Ambi-RVOS. This benchmark contains ambiguous queries. The results are significant. The paper reports that IC-Seg-8B achieves an overall J & F of 55.1 on Ambi-RVOS. J & F is a combined metric of region similarity and contour accuracy. This represents a massive leap over existing MLLM-based methods like UniPixel-7B, which only scores 29.2.
Comparing the 4B and 8B versions, the authors find that IC-Seg-4B provides a +21.6% J & F improvement over the non-interactive Qwen3-VL-4B* baseline. Crucially, the model doesn't just win on the "hard" ambiguous tasks. It also maintains state-of-the-art performance on standard reasoning benchmarks like ReasonSeg and ReasonVOS [Table 2, Table 3].
From a production standpoint, the efficiency trade-offs are worth noting. The authors report that IC-Seg averages 2.5 clarification turns per sample. While this adds latency, the training overhead is manageable. Adding the hierarchical step-level optimization increased training time per step by approximately 13% (from 295s to 333.5s) compared to the version without it [Table 12].
What's Missing
While the results are strong, there are gaps that a practitioner should consider: * Modality Constraints: The framework is currently limited to text-based interaction. Real-world users might prefer pointing or using voice. The paper acknowledges this limitation. The current architecture is heavily optimized for text-token generation. * Simulator Dependency: The training and evaluation loop relies on an MLLM-based User Simulator. The authors include a user study [Table 7] to argue that the simulator behaves similarly to humans. However, any systematic bias in the simulator will be baked into the policy. * Scalability of Interaction: The paper sets a maximum of 5 turns. This prevents infinite loops. However, it does not explore how the system handles extremely complex scenes. Some scenes might require many more turns to specify a tiny, moving object.
Should You Prototype This
Yes, if you are building interactive visual tools.
If your product involves users commanding a system to manipulate visual content, the current "guess-and-fail" paradigm is a major issue. IC-Seg demonstrates that adding a conversational layer is a mathematically sound way to increase segmentation accuracy. It does this by reducing entropy (uncertainty).
The implementation details are transparent. They use the VERL framework and base their policy on Qwen3-VL. The code and data are available at https://github.com/iSEE-Laboratory/IC-Seg. You can verify the training dynamics and the Hi-GRPO implementation yourself. If you have the compute to run RL on MLLMs, this hierarchical approach is a stable way to train agents.