TACO: Tool-Augmented Credit Optimization for Agentic Tool Use

AI agents that use tools like cropping or zooming often struggle to know if a tool actually helped or just made things worse. This creates a massive credit assignment problem (the difficulty of determining which specific actions in a sequence led to a final outcome). If an agent gets a question right, was it because of the clever code it wrote, or was it right despite the code causing a distraction?

Current agentic multimodal models—systems that "think with images" by writing Python code to manipulate visual inputs—typically rely on outcome-only rewards. In this paradigm, the model is rewarded based solely on whether the final answer is correct. While effective for basic tasks, this approach fails to distinguish between useful, redundant, or even misleading tool calls. As shown in, a single crop operation can turn a wrong answer right (useful), leave it unchanged (redundant), or flip a correct answer to wrong (misleading).

Because standard reinforcement learning (RL) broadcasts the final success signal across the entire trajectory (the full sequence of actions), it risks reinforcing useless habits. It may also penalize correct reasoning that was subsequently sabotaged by a bad tool call.

The breakdown of outcome-only rewards

The central tension in training code-tool agents is the confounding of tool-induced gains with the model's intrinsic capabilities. Recent analyses suggest that much of the perceived progress in "crop-and-zoom" RL is actually driven by the model's underlying improvement rather than the utility of the tool itself.

Existing attempts to solve this via process rewards—which provide feedback on intermediate steps—face a difficult trade-off. One path involves using an external judge model (like GPT-4o) to score each tool's output. This is computationally expensive and inherits the biases of the judge. Another path involves step-wise rewards for textual reasoning. However, these cannot effectively isolate the impact of an external visual observation. Without a way to measure the marginal delta (the specific change caused by an action) provided by a tool, the agent remains stuck in a cycle of over-calling tools or failing to realize when a tool has led it astray.

Dual-channel credit assignment via TACO

To resolve this, the authors propose Tool-Augmented Credit Optimization (TACO). This is a variant of Group Relative Policy Optimization (GRPO, a reinforcement learning algorithm that compares a group of rollouts to a group mean). TACO utilizes two distinct, coupled advantage channels .

Instead of a single scalar reward, TACO splits the learning signal into a process channel and an accuracy channel.

1. Differential Answer-Probe Reward (DAPR): This is the "process" channel. To estimate the value of a tool call without an external judge, the authors insert two "probes" into the reasoning chain. The first (tool-off) asks the model what it would answer before executing the code. The second (tool-on) asks what it answers after seeing the tool's output. The reward is the difference ($\Delta$) between these two outcomes. This effectively calculates a counterfactual: "What was the marginal benefit of this specific tool call?" Because it uses the difference, it is robust to "probe-hacking." This is a failure mode where a model tries to cheat by writing the answer early in its reasoning to inflate its probe scores.

2. Outcome-Gated Advantage Routing (OGAR): Even with a good tool-value signal, the reward must reach the correct tokens. OGAR is a parameter-free rule that gates the final-answer advantage ($A_1$). It ensures that if a tool call is "right-but-redundant" (meaning the model already had the right answer), the advantage is withheld from the tool tokens. This prevents the reinforcement of unnecessary computations. Conversely, if a call is "misleading," the penalty is directed at the tool branch. This protects the integrity of the initial reasoning.

By combining these, TACO optimizes for both the correctness of the final answer and the efficiency of the path taken to get there.

Efficiency gains and accuracy surges

The empirical results are significant, particularly regarding the relationship between accuracy and inference cost. In Table 1, the authors report that their 7B model reaches an average accuracy of 68.1% across twelve benchmarks. This substantially outperforms other code-tool agents like PyVision-RL-7B (63.7%). Most notably, it outperforms the closed-source GPT-4o (58.5%) in this specific agentic context.

Crucially, the authors demonstrate that TACO does not achieve these gains through brute-force tool use. As shown in [Table 2], TACO is simultaneously the most accurate and the fastest agent. For example, on the $V^*$ benchmark, it achieves 89.6% accuracy with a latency of only 2.3 seconds. This is faster than competitors like PyVision-RL-7B, which takes 3.6 seconds for 88.7% accuracy. This speedup is a direct consequence of the learned economy. The model learns to skip the tool when it isn't needed. This shortens the completion length [Figure 3b] and reduces the number of sandbox execution rounds.

Constraints of the verifiable reward paradigm

While the results are compelling, there are notable boundaries to the paper's claims. First, the TACO framework relies heavily on a rule-based, verifiable outcome checker (rout). This means the methodology is currently specialized for tasks with discrete, checkable answers. It is unclear how easily this would translate to open-ended, generative tasks. Such tasks might require a high-capacity LLM judge, which would reintroduce the costs the authors sought to avoid.

Second, the "before/after" logic assumes that the tool's effect is captured by the model's predicted answer. There is a latent assumption of "faithful probing." This assumes that the greedy decoding (selecting the most likely next token) of a probe accurately reflects the model's internal state. If the model's internal representation is poorly captured by a short decode, the $\Delta$ signal might become noisy. Finally, the current implementation is optimized for a single-call regime. While the authors mention multi-call support, attributing credit within a complex, multi-turn trajectory remains an open challenge.

The verdict: A blueprint for efficient agency

If you are building multimodal agents that interact with environments via code, TACO is a highly relevant advancement. I would characterize this as a "yes" for practitioners focused on high-precision, low-latency deployment.

The brilliance of the paper lies in its refusal to accept "more reward" as a solution to "better behavior." By using the model's own predicted outcomes as a counterfactual baseline, the authors have created a self-supervised way to implement a minimalist policy. This policy uses tools only when they provide a measurable delta in certainty. The fact that this mechanism generalizes across different base models (from Qwen2.5-VL to Qwen3-VL) suggests that the credit-assignment logic is a fundamental architectural improvement. Expect to see differential prompting or similar counterfactual reward structures becoming standard in the next generation of agentic RL.

Figures from the paper