SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation

SimFoundry: Automated Real-to-Sim Scene Generation for Robust Robot Policy Learning

Training and evaluating robot policies in the real world is notoriously difficult to scale. It is a slow, expensive process. It often requires months of human teleoperation (manual remote control of a robot). Researchers have turned to simulation as a scalable alternative. However, the bottleneck remains the manual construction of environments. These environments must match the real world in geometry, visuals, and physics. SimFoundry attempts to solve this. It turns a single real-world video into an interactive, sim-ready digital twin. As shown in, the system does not stop at a static replica. It automatically generates "digital cousins"—variations of the original scene, objects, and tasks. These provide the diversity needed for robust policy training.

The manual bottleneck in real-to-sim

The status quo for bridging reality and simulation is hindered by massive labor requirements. Current real-to-sim (reality-to-simulation) approaches typically fall into two camps. They either focus on high-fidelity 3D reconstructions that lack physical interaction. Alternatively, they focus on simulation-based evaluation using manually tuned scenes. If you want to move from a video of a kitchen to a simulation, you usually hit a wall. You face tedious manual asset creation and pose alignment. Even automated systems struggle with complexity. They often fail to support articulated objects (items with moving parts, like drawers). They also struggle with multi-step or bimanual (two-handed) coordination. This leaves a gap in robotics research. Simulation is often too simple to train useful policies or too disconnected from reality to serve as a reliable benchmark.

A modular pipeline for digital twins and cousins

SimFoundry addresses this through a three-stage modular architecture. This process is detailed in .

1. Extraction: The system ingests a raw RGB video. It uses depth estimation models to lift pixels into a scene point cloud (a collection of data points in 3D space). It employs a Vision-Language Model (VLM) to detect objects. An iterative segmentation process pulls out foreground objects. The system "cleans" the video by inpainting (digitally filling in) the background as each object is removed.
2. Generation: For every extracted object, the pipeline uses 2D-to-3D generation models to create a visual mesh. It then refines the 6-DoF (six degrees of freedom) pose of these meshes. This ensures they align with the original scene geometry. Crucially, an articulation module identifies movable parts. It generates joint parameters like friction and mass to make objects physically interactive.
3. Augmentation: This is the core differentiator. Once the "digital twin" is built, SimFoundry expands it into "digital cousins" .

These are affordance-preserving variations (changes that keep the object's function intact). Object cousins change the shape and texture of items. Scene cousins rearrange the spatial layout using semantic predicates (logical rules like Inside or OnTop). Finally, Task cousins use a VLM to propose new, feasible manipulation goals within the reconstructed scene.

High correlation and significant training gains

The authors validate the system by measuring how well simulation predicts real-world performance. They report a mean Pearson correlation of 0.911. They also report a mean maximum ranking violation (MMRV) of 0.018 .

In practical terms, this means simulation results are highly reliable. If a policy performs well in the SimFoundry simulation, it is highly likely to succeed in the real world. This significantly outperforms the state-of-the-art baseline, PolaRiS. The paper notes that SimFoundry's Pearson correlation is over 0.59 higher than PolaRiS.

The utility for training is equally concrete. The authors show that policies trained with "cousin" augmentations show substantial real-world improvements. Training with object cousins yielded a 17% improvement in success rates. Scene cousins provided a 21% improvement. Task cousins provided the largest boost at 40% .

This suggests that structured diversity is more effective than simple, unconstrained domain randomization (randomly changing simulation parameters).

Implementation costs and model dependencies

The pipeline has clear operational costs. The automatic background reconstruction uses 3D Gaussian Splatting to create a photorealistic backdrop. This is computationally heavy. The authors report that the two-pass video inpainting takes approximately 90 minutes per scene on a single GPU. This can be mitigated in multi-GPU environments. You can run background reconstruction in parallel with the foreground stages. However, it remains a significant overhead for single-user setups.

SimFoundry is also a composite system. It relies heavily on off-the-shelf foundation models for segmentation, depth, and mesh generation. Consequently, the system inherits the failure modes of these underlying models. If a VLM hallucinates an incorrect object property, those errors propagate into the physics engine. The paper also notes a current limitation regarding layout. The system is presently limited to tabletop-style layouts. It is not yet designed for complex, multi-level environments.

The verdict

Is SimFoundry worth integrating into your robotics pipeline? If you want to move away from brittle, manually authored simulations, the answer is yes. The ability to turn a single video into a diverse training distribution is a major advancement. This facilitates easier sim-to-real transfer (moving learned skills from simulation to reality).

The system is most valuable for engineers building generalist policies. These are models meant to handle many different tasks. Such policies need to generalize to unseen object shapes and layouts. However, do not expect a "one-click" solution for everything. You will likely need a few minutes of per-object manual tuning to reach peak fidelity. For those interested in prototyping, the code is reportedly available; see the canonical link at https://research.nvidia.com/labs/gear/simfoundry/.

Figures from the paper