Prior Availability in Industrial Visual Sim-to-Real: A Review of CAD-Guided and CAD-Unavailable Regimes

Reframing Industrial Sim-to-Real: A Taxonomy Based on CAD Prior Availability

In industrial automation, computers often need to recognize objects or defects without massive, labeled real-world datasets. To solve this, engineers rely on "sim-to-real" transfer. This means using synthetic data to train models for the messy reality of a factory floor. However, the industry lacks a unified way to categorize these methods. This leaves pose estimation researchers and anomaly detection engineers speaking different languages.

This paper argues that sim-to-real effectiveness is dictated by one variable: the availability of CAD (Computer-Aided Design) models. If you have the geometry, you can treat the domain gap as a geometric verification problem. If you do not, you are solving a purely statistical appearance problem.

The Problem

Current sim-to-real discourse often collapses the field into a narrow definition. It assumes a simple transfer from synthetic rendered images to real camera images. This is a dangerous simplification for production. In a real factory, the "source" might be various proxies. These include PBR (Physically Based Rendering, a method for simulating light interaction) simulations, pretrained feature spaces, or language prompts. The "target" involves unpredictable lighting, shifting materials, and rare defect modes.

As shown in, the mismatch is not just about pixels.

It is about the information used to make a decision. Treating sim-to-real as a generic recipe ignores the most powerful tool in the engineer's kit: explicit geometry. Current approaches often fail by assuming that making synthetic images "more real" is the only solution. This ignores the fact that even realistic renders cannot account for the geometric certainties a CAD model provides at test time.

How It Works

The authors propose a taxonomy organized by "prior availability." This dictates the mechanical approach to closing the domain gap. They divide the field into three regimes:

1. CAD-Available: The system has access to explicit 3D meshes. This allows for two critical roles. First, CAD acts as a pre-deployment renderer. It generates labeled synthetic data via domain randomization (varying lighting, textures, and poses to broaden the training distribution). Second, CAD acts as a test-time geometric prior. Instead of just predicting a bounding box, the system can perform "render-and-compare" .

It generates a hypothesis, such as a 6D pose (the position and orientation of an object in 3D space), and renders the CAD model into the camera view. It then verifies if the rendered silhouette or depth map aligns with the real observation.

1. CAD-Unavailable: In settings like surface defect inspection, geometry is often absent. The system must replace explicit geometry with replacement priors. These include normal-reference memory (storing feature distributions of "good" parts), teacher-student residuals (detecting discrepancies in how a model represents data), or vision-language priors (using semantic prompts like "scratched" to guide detection).

2. Boundary-Prior: This is the middle ground. The system uses approximate models, templates, or sparse reference views. These methods preserve some geometric authority but lack the full "render-and-verify" capability of a complete CAD pipeline.

Numbers

The paper uses empirical "anchors" to prove that the choice of prior changes the fundamental math of the task.

In the CAD-available regime (using T-LESS/BOP datasets), the authors show that increasing synthetic data volume is a trap. Increasing PBR renders from 5k to 50k actually causes mAP50:95 (a strict localization metric measuring how well a box overlaps the object) to drop from 0.1521 to 0.1287 [Table 5]. The real wins come from distribution design and calibration. Applying domain randomization (B2) jumps mAP50:95 to 0.4041. Adding a tiny 5% real-labeled calibration set (B6) pushes it to 0.7424 .

The authors also show that CAD at test time provides a unique verification channel. Using MegaPose for render-and-compare, they report a mean full-mask IoU (Intersection over Union, a measure of overlap between two shapes) of 0.7322 [Table 6]. Fusing detector confidence with rendered depth consistency improves the "good-pose" AUROC (Area Under the Receiver Operating Characteristic curve, a measure of how well a model separates classes) to 0.8804 .

This proves geometry provides a signal that appearance-only detectors cannot replicate.

In the CAD-unavailable regime (MVTec AD/VisA), the focus shifts to ranking. Normal-reference methods like PatchCore remain dominant. They achieve a pixel AUROC of 0.9801 on MVTec AD [Table 7]. Meanwhile, zero-shot vision-language models like WinCLIP struggle with dense localization. They show much lower pixel-level performance compared to geometry-free feature memory methods .

What's Missing

While the taxonomy is robust, several engineering considerations remain:

• Verification Trade-offs: The paper emphasizes the power of "render-and-compare" at test time. However, this method introduces a known trade-off regarding latency. Performing geometric verification adds computational steps that a simple feed-forward detector avoids.
• Sensor and Material Sensitivity: The review notes that transparent or reflective materials break geometric assumptions. However, it does not deeply analyze how specific sensor artifacts interact with different prior regimes.
• Complexity of Integration: Moving to a "CAD-at-test-time" workflow requires more complex inputs. These include usable camera metadata, object identity, and refinement inputs.

Should You Prototype This

Yes, but choose your architecture based on your assets.

If your product team provides high-fidelity CAD models, do not settle for a simple synthetic-to-real detector. You are leaving performance on the table if you do not implement a test-time geometric verification channel. It is the only way to turn a probabilistic guess into a verifiable geometric hypothesis.

If you are working on a legacy line where CAD is unavailable, do not rely solely on Large Vision-Language Models (LVLMs) for pixel-perfect defect localization. The numbers in suggest that for dense, texture-level industrial inspection, established normal-reference memory and dense foundation features (like DINOv2) are still more reliable.

Code and metadata are available at the project repository: https://github.com/JacksonTao888/industrial-visual-sim2real-priors.

Figures from the paper