SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models

Why Your Vision Models Struggle with Geometry

Researchers have created a new way to test if AI models truly understand object parts. They want to know if a model can distinguish a car's front-left wheel from its rear-right wheel. Previously, evaluating a model meant checking if it could label a whole object. This did not reveal if the model understood internal structure. This paper introduces SOCO, a benchmark for Semantic Object Correspondence (SOC). It tests the ability to match specific parts across different objects. The finding is striking. Modern vision models excel at recognizing what a part is. However, they struggle to understand where that part is relative to the object.

The Problem

Current evaluation protocols for vision foundation models (VFMs) are misaligned with spatial tasks. Most benchmarks focus on category-level recognition, like ImageNet. These ask if a model can classify an entire image. This provides little signal regarding "structured object understanding." This is the ability to relate semantically corresponding parts across different instances or categories.

Existing semantic correspondence (SC) benchmarks have two main issues. First, they conflate local concept recognition with object-relative identity. Local concept recognition is identifying a "wheel." Object-relative identity is distinguishing a "front-left wheel" from a "rear-right wheel." This creates measurement ambiguity. If a model fails, you do not know if it missed the wheel or confused the sides. Second, current datasets rarely evaluate transfer across related categories. As shown in, moving from a car to a bus requires mapping shared concepts like "wheels" across different geometries.

Without this, we cannot assess if a model learns generalized "part-ness" or just memorizes templates.

How It Works

The authors introduce a taxonomy-driven formulation called Semantic Object Correspondence (SOC). They decompose the task into three measurable levels:

1. Concept Correspondence (CC): This measures matching the same local semantic concept (e.g., a "corner point") regardless of position.
2. Semantic Object Correspondence (SOC): This adds geometric constraints. The model must match the concept and its specific object-relative identity (e.g., "bottom-left corner"). This forces the encoding of spatial structure.
3. Cross-category SOC (Cross-SOC): This is the most difficult tier. It requires matching keypoints across related categories (e.g., a wheel on a car vs. a wheel on a tractor) using a shared taxonomy.

The SOCO dataset uses over 1 million correspondence pairs across 100 categories .

These include transportation, furniture, and animals. To support Large Vision-Language Models (LVLMs), the authors included language descriptions for every keypoint. This allows a specialized evaluation. The model must localize a part based on a text prompt (e.g., "the center point of the front left wheel of a bus") rather than just a visual marker.

Numbers

Results reveal a "geometry gap" in advanced vision backbones. While models like DINOv2 show high accuracy in Concept Correspondence (CC), performance drops during SOC. For DINOv2, the authors report an 18.5% reduction in [email protected] (Percentage of Correct Keypoints; the ratio of predictions within a specific radius of the ground truth) when moving from CC to SOC [Table 2]. This drop means strong semantic features do not guarantee geometric awareness.

SOC is a highly useful diagnostic tool for engineers. In, the paper shows SOC correlates more strongly with dense downstream tasks than ImageNet kNN classification.

These tasks include segmentation, tracking, and 3D pose estimation. For a practitioner building robots, this is vital. Optimizing for SOC is a more reliable proxy for real-world performance than chasing ImageNet scores.

Regarding LVLMs, models show a different failure mode. They are better at text-prompted localization than at visual-reference matching. Accuracy improves when moving from a purely visual query (Vis.) to a text-described query (Desc.) [Table 3]. This exposes a gap in how these models align visual markers with linguistic concepts.

What's Missing

SOCO has limitations to consider. First, the benchmark uses sparse keypoint annotations. It does not use dense, pixel-wise semantic matching. This means it diagnoses structural understanding but may not evaluate high-fidelity segmentation.

Second, the dataset has an "image-source bias." Images come from ImageNet3D and Animal3D. This biases the data toward curated, salient, and well-lit views. Such bias might overestimate performance in messy or low-light production environments.

Finally, the language descriptions for LVLMs are template-based. More nuanced natural language might improve performance. The current "gap" between language and vision might be partly due to simplified prompting.

Should You Prototype This

Yes, if you work on 3D perception, robotics, or fine-grained manipulation. If you only need to classify images, stick to ImageNet. But if you build systems interacting with physical structures, stop using classification accuracy as your primary metric.

The SOCO dataset and framework are at https://genintel.github.io/SOCO/. The evaluation uses a zero-shot nearest-neighbor matching protocol. This is computationally cheap to run as a diagnostic on existing backbones. It is a high-signal way to see if your representations learn geometry or just textures.

Figures from the paper