Sim-to-Real Engine why robots trained in simulation stumble in real buildings

What the gap is

Researchers usually describe a robot’s job as a set of parts: what state the world is in, what actions the robot can take, how the world responds to those actions, what counts as success, and what the robot’s sensors report back.1 A simulator approximates every one of those parts, and any of them can be approximated wrongly.

So the sim-to-real gap is a family of different errors that all show up the same way: a policy that does well in training and worse on the real robot.1 Two robots can fail for unrelated reasons and both failures get called “the gap.” That matters because the fix depends on the cause. If the error is in how the world responds, better graphics won’t help. If it is in the camera, tuning friction values is wasted effort.

Where it comes from

Contact and materials

The deepest source and the hardest to fix. Simulators treat geometry and materials coarsely, and real objects stretch, deform, keep a memory of past loads and carry manufacturing defects no CAD model contains.1 That is why tasks with sustained contact, like insertion, assembly and deburring, transfer worse than moving through open space.

Cameras and light

A policy trained on rendered images learns a visual world that doesn’t match a real camera. Sensor noise, motion blur, reflections and the way a facility’s LEDs flicker are missing from training unless someone puts them there.

Actuation and timing

Simulated motors do exactly what they are told, immediately. Real ones have backlash, flex, thermal drift, control-loop delay and communication jitter. A policy that expects its commands to take effect at once will overshoot on hardware.

The environment

The source operations teams feel most and research discusses least. In simulation the fixture sits in the right place. On a real floor it sits roughly there, next to a cable someone rerouted and a bin that gets bumped every shift.

Why train in simulation anyway

Because the alternative is worse. Training on real hardware is slow, expensive and sometimes dangerous. In simulation a robot can fail thousands of times before it touches a machine, and one GPU can run many virtual robots in parallel.3

That scale has produced real results. Large-scale simulation with randomized conditions trained a robot hand to solve a Rubik’s Cube,6 and simulation-trained policies underpin much of what works in legged robots and agile flight.12 NVIDIA reported generating 780,000 synthetic training examples, about nine months of human demonstrations, in 11 hours of compute, and found that mixing them with real data improved its model by 40% over real data alone.11

Four ways to close it

A 2026 review by researchers at the University of Zurich, NVIDIA and the University of Washington groups the solutions into four families.1 Real projects combine them.

Randomize the simulation

Vary lighting, textures, object positions, mass, friction and delays across training, so the policy learns to handle variety and the real world becomes one more sample.45 Its weakness: ranges that are too wide produce cautious policies that hedge against conditions that never happen and lose precision on the actual job.7

Measure the real system

Instead of randomizing over what you don’t know, find out. System identification estimates the real robot’s physical values by comparing simulated runs with real ones, and newer methods have the robot make the moves that reveal the most.7 The usual practice is to measure what is cheap to measure and randomize the rest.

Build the simulation from reality

Reconstruct the scene from real sensor data and train inside the reconstruction.8 One method using this approach transferred camera-based manipulation policies to real robots with 86% average success, against 97.5% for policies trained on real data directly.10 That closes most of the gap, though not all of it. Reconstructed scenes have also predicted real-world success rates with correlations above 0.9, which lets you compare candidate policies without tying up a real cell for a week.9

Change what transfers

Some inputs cross the gap better than others. Depth, point clouds and the robot’s sense of its own joints are much closer between simulation and reality than photorealistic images.10 Training on simulated and real data together lets the real data correct what the simulator gets wrong.1

Telling which gap you have

A success rate alone won’t tell you why a robot fails. These patterns, from practice rather than formal study, point to the likely source.

Fails in the same place every time

Suspect motors and timing. The policy expects a response it isn’t getting.

Fails on some shifts and not others

Suspect vision. Changing light is the usual cause and the cheapest to test: run the task under different lighting and watch the failure rate.

Works on the demo part, fails on a variant

Suspect the environment. The policy learned one case instead of the task.

Works in open space, fails on contact

Suspect physics. This is the hardest category, and the one where a more detailed simulator helps least.

Each of these depends on logging failures with enough context to diagnose them. That is a deployment question as much as a modeling one, and it is part of why architectures that predict what will happen before they act, like the world action models we have written about, matter on a real floor.

How Bellwether uses this

We don’t think a perfect simulation can be built, and it isn’t the goal. The research reaches the same conclusion: reliable transfer comes from combining better simulation, robust training and careful system design.1 Bellwether treats simulation as the starting point.

Bellwether simulates each robot task in your workflow with physics-based objects and environments, then grades the simulation against the job the task actually has to do. We call that distance the sim-to-target gap, and measuring it tells you early whether a task is ready to train or needs better information first, such as a scanned object or more detail about the site.

A graded simulation can be run thousands of times with randomized conditions. Because the grade ties the simulation to your actual task, the variation can stay close to what really changes on your floor, which avoids the overly cautious policies that wide randomization produces.

On the real robot, a small model handles routine work and hands anything unfamiliar to a larger model. Every real run, including the failures, flows back into training through automated training data collection, and each hand-off records where the gap still exists. The gap stops being a fixed penalty at go-live and becomes a number that shrinks with operating hours.

Bellwether’s first version delivers the plan and graded simulations; training data and the deployed model follow. From plan to working robot lays out that path. We do this work inside the Center for Smart Convergent Manufacturing Systems at RPI, alongside manufacturers whose real cells are where these questions get settled.

Sources

1. Aljalbout, E., Xing, J., Romero, A., et al. “The Reality Gap in Robotics: Challenges, Solutions, and Best Practices.” Annual Review of Control, Robotics, and Autonomous Systems, Vol. 9, 2026. arXiv:2510.20808
2. Zhao, W., Peña Queralta, J., and Westerlund, T. “Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: A Survey.” IEEE SSCI, 2020.
3. Labellerr. “How to Bridge the Reality Gap in Robotics Training.” July 2026. labellerr.com
4. Tobin, J., Fong, R., Ray, A., et al. “Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World.” IEEE/RSJ IROS, 2017.
5. Peng, X. B., Andrychowicz, M., Zaremba, W., and Abbeel, P. “Sim-to-Real Transfer of Robotic Control with Dynamics Randomization.” IEEE ICRA, 2018.
6. OpenAI et al. “Solving Rubik’s Cube with a Robot Hand.” arXiv:1910.07113, 2019.
7. Sobanbabu, N., He, G., He, T., Yang, Y., and Shi, G. “Sampling-Based System Identification with Active Exploration for Legged Robot Sim2Real Learning.” arXiv:2505.14266, 2025.
8. Torne, M., Simeonov, A., Li, Z., et al. “Reconciling Reality Through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation.” arXiv:2403.03949, 2024.
9. “Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactions.” arXiv:2511.04665, 2025. real2sim-eval.github.io
10. Qureshi, M. N., Garg, S., Yandun, F., et al. “SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splatting.” arXiv:2409.10161, 2024. splatsim.github.io
11. NVIDIA. “NVIDIA Announces Isaac GR00T N1.” NVIDIA Newsroom, March 2025. nvidianews.nvidia.com