Why Your Multimodal Agent Can't Find a Hammer
In the field of Large Multimodal Models (LMMs), we have made massive strides in perception. Models label objects and describe scenes with high fidelity. However, a significant gap remains. There is a disconnect between recognizing an object and understanding its physical utility. Most current models excel at pattern recognition. They fail at "creative tool repurposing"—the ability to use random objects in non-obvious ways. This might mean using a rubberized handle for grip or a serrated edge to cut tape.
Evaluating this has been difficult. Previous methods relied on text-based reasoning or static images. These do not capture the iterative nature of real-world problem solving. Researchers have essentially tested if an AI can "imagine" a solution without forcing it to "look" at the evidence. This paper introduces MM-CreativityBench to bridge that gap. It moves from passive observation to an interactive protocol. Models must actively inspect scenes, zoom in on parts, and justify decisions based on visual physics.
The Problem
The status quo in multimodal reasoning favors "System 1" inference (fast, intuitive, and often superficial). Current LMMs can describe a scene or retrieve common patterns. However, they struggle when a task requires connecting a goal to a fine-grained physical attribute. They might know a "key" is a tool. But they fail to recognize that the serrated edge is the functional module required for the task.
The authors find that even frontier models suffer from this disconnect. They struggle with the gap between coarse semantic plausibility and actual physical grounding. As shown in, performance drops when environments contain distractors with similar affordances (the potential for an object to perform a specific action).
Models do not necessarily fail because they cannot find a "tool." They fail because they cannot disambiguate between candidates. These candidates look functionally similar but differ in critical physical properties like material hardness or edge geometry. Furthermore, shows that even when the target affordance is "typical," models do not find it easier.
Instead, they enter long, aimless exploration loops without improving accuracy.
How It Works
The researchers treat creative tool use as an interactive, evidence-driven search process. Their method, "Affordance-Grounded Alignment," follows a structured pipeline:
- Benchmark Construction via Reverse Engineering: The authors use an existing affordance knowledge base to work backward. They pick a target entity-part pair. They identify the physical attributes that make it work. Then, they generate a task description that necessitates that specific functionality.
- Hierarchical Visual Interaction: The benchmark forces a three-level inspection protocol: Environment $\rightarrow$ Entity $\rightarrow$ Part. The model starts with a wide shot of the scene. It must choose an entity to "inspect" to receive a full-object view. It can then choose to "inspect a part" to receive a zoomed-in view. This mimics how a human scouts a workspace.
- Two-Stage Alignment: To fix failure modes, they use a dual-training strategy. First, Supervised Fine-Tuning (SFT) uses a "knowledge-guided exploration stack" (an ordered list of inspection targets). This teaches a disciplined search policy. Second, they apply Direct Preference Optimization (DPO) (a method to align models with preferred behaviors) using "hard negatives." These are trajectories that look fluent but actually hallucinate attributes. This teaches the model to prefer reasoning tied strictly to visible pixels.
Numbers
The authors report substantial gains from the combined SFT+DPO approach. For a Qwen3-4B-VL model, the "Gold Correct Rate" jumped significantly. This metric measures if the model identified both the correct entity and the correct specific part. The rate rose from 0.156 in the base model to 0.417 after SFT+DPO.
This improvement did not come from making the model "think longer." Efficiency actually improved. For the 4B model, average exploration turns dropped from 18.92 to 6.21. This means the model is more selective and decisive. It gathers sufficient evidence earlier and stops searching once a valid part is found. The error analysis in confirms this. The primary driver was a massive reduction in "Affordance Mismatch" (Category A2) errors. In these errors, models picked parts that looked right but lacked necessary mechanical properties.
What's Missing
There are a few gaps to note. First, the benchmark relies on synthetic images generated by Gemini. This provides a controlled environment for testing grounding. However, it avoids the messy reality of real-world sensor noise or extreme occlusions.
Second, the scope is limited to "constrained creativity." The paper defines creativity as finding a valid solution within set parameters. It does not address truly open-ended, generative creativity. If your goal is high-level design ideation, this framework may not apply. Finally, the paper does not report the exact computational cost of the training regime. Specific GPU hours or throughput metrics for the DPO phase are not provided.
Should You Prototype This
Yes, if you are building embodied agents. This applies if your roadmap involves robots or agents in unstructured environments. The "Affordance-Grounded Alignment" strategy is highly actionable. The takeaway is clear: do not just train for the final answer. Train for the process of inspection. Using hard-negative DPO to penalize "plausible-sounding hallucinations" is a powerful lever. This is vital for improving agent reliability. However, if you work on purely digital assistants, this specific methodology may not yield a meaningful delta.