IndustryBench-MIPU: Benchmarking Multi-Image Attribute Value Extraction for Industrial Products

The Completeness Gap in Industrial AI

Why do some automated systems accurately identify a part but fail to capture its full technical profile? In industrial supply chains, products like valves and circuit breakers are defined by dense, complex specifications. These details govern everything from procurement to operational safety. Currently, these details are often scattered across multiple images. These include specification tables, nameplates, and technical drawings.

While Multimodal Large Language Models (MLLMs) show promise in general visual reasoning, it remains unclear if they can reliably reconstruct a complete product profile. This requires looking at a collection of heterogeneous (diverse in form or character) images. Researchers from the Alibaba Group addressed this by introducing IndustryBench-MIPU. This is a new benchmark designed to test industrial understanding. Their findings reveal a striking mismatch. While models are highly accurate in what they do extract, they are fundamentally incomplete. They often miss more than half of the necessary technical data when forced to look across multiple images.

Beyond Simple OCR

Existing multimodal benchmarks typically focus on broad visual reasoning or basic Optical Character Recognition (OCR). OCR is the process of converting images of text into machine-encoded text. While these are useful for general tasks, they fall short in industrial environments. As illustrates, understanding an industrial product requires four distinct capabilities.

These are recognizing text on labels, performing visual reasoning over technical drawings, applying domain knowledge to decode specialized terminology, and integrating evidence scattered across different images.

Current approaches often treat these as isolated problems. Most benchmarks evaluate single images or general consumer goods. They fail to account for the "multi-image" reality of industrial catalogs. In a real-world workflow, a single product might have several images. One might be a photo of its exterior. Another could be a close-up of its nameplate. A third might be a separate technical drawing. The authors argue that existing models struggle at this intersection. They struggle to locate and synthesize fragments of information from a diverse visual set.

A Multi-Model Consensus Approach

To create a benchmark that measures the upper limits of model capability, the researchers developed a semi-automated construction pipeline. Instead of relying on a single model to define the "correct" answer, they used a multi-model consensus mechanism. This means they used multiple models to agree on the ground truth. As shown in, the process begins with stratified sampling (selecting samples that represent the whole population) from a large-scale Chinese industrial e-commerce platform.

The authors implemented a three-stage annotation pipeline using five different MLLMs: 1. Entity Recognition: The models first identify the core product. This prevents "entity drift," which is when a model extracts attributes belonging to a different, nearby product. 2. Image Filtering: Models determine which images actually contain relevant specifications. They filter out irrelevant content like marketing banners. The authors report that approximately 31% of candidate images are discarded here. 3. Per-Image Extraction: Valid images are processed to extract structured property-value pairs. These are formatted according to a product-specific schema (a predefined blueprint for data).

To ensure high quality, the team applied a three-tier quality assurance process. This included a "Frontier Model Audit" using an independent model to flag hallucinations (generating false information). They also used a "Gold-Standard Cross-Check" against verified attributes. Finally, they performed human verification. The authors report a 96.7% pass rate during human spot-checks. This suggests the automated tiers are highly effective.

The Cost of Complexity

The most significant finding is a persistent "completeness gap." When evaluating nine MLLMs, the authors found a precision–recall asymmetry. Precision (the accuracy of the extracted values) remained high, between 86% and 94%. However, recall (the percentage of total attributes successfully recovered) was remarkably low. Even the best-performing model, Gemini 3.1 Pro, recovered only 49.9% of product-level attributes. This means even the strongest model misses more than half of the required data.

The transition from one image to a whole product profile comes at a steep price. Moving from single-image to multi-image extraction causes a drop in recall of 15 to 34 percentage points. This suggests the bottleneck is not the model's ability to read a single label. Instead, the bottleneck is its ability to manage evidence across a set of images.

The authors provide a breakdown of where this failure happens. As seen in, performance declines as the number of input images or the density of specifications increases.

Difficulty is also tied to the type of information requested. The study finds a "recall gradient" based on cognitive demand. "Direct standardized" fields are the easiest. "Visual reasoning"—interpreting structural cues or diagrams—is the hardest, yielding only 36.6% recall.

Premature Stopping in Dense Layouts

To understand why models miss so much data, the researchers conducted a case study on a microscope objective .

They found that models are quite good at capturing simple, isolated facts. Examples include a brand name or a single color. However, they fail when faced with "enumerative fields." These are data points that exist in a list or a dense matrix.

In one instance, a model was presented with a magnification matrix. This is a table listing various combinations of optical settings. While the table contained 27 benchmark attributes, the model only recovered 5. The authors observe that models recognize the field and extract the first few values. Then, they simply stop. This "premature stopping" explains why precision stays high while recall collapses. The values they do find are correct, but they leave the rest of the table untouched. This behavior is evident in products with high specification density. The volume of data overwhelms the model's ability to maintain attention.

Verdict: Not Ready for Autonomous Procurement

If you are looking to deploy an autonomous system for inventory cataloging today, the verdict is: not yet. While these models are reliable at telling you what they do see, they are prone to missing vital information. In an industrial setting, a missing "pressure rating" or "voltage range" is a serious issue. Such omissions create safety and compatibility risks.

The research demonstrates that scaling model size helps. The Qwen 3.5 family shows sharper performance gains in multi-image settings as parameters increase. However, even the largest models cannot yet overcome the challenge of cross-image integration. For practitioners, the takeaway is clear. MLLMs can serve as decision-support tools to highlight visible data. However, they require human oversight. Humans must ensure that "silent" missing specifications are properly accounted for. Code and the IndustryBench-MIPU dataset are reportedly available for further research.

Figures from the paper