Understanding Sim-to-Real: Why Simulation-Trained Robots Fail in Real Buildings
A policy that succeeds ten thousand times in simulation gets loaded onto real hardware, runs on a real floor, and misses the mark. Nothing is broken. The code is the same code. The robot is the robot it was modeled on. It just doesn't work. This is the sim-to-real gap, also called the reality gap, and it is one of the oldest unsolved problems in robotics. It is also, in 2026, one of the most active research areas in the field because simulation has become the default way to train robot policies at scale, and the gap is the tax you pay for using it [1]. Everyone in robotics has a rough sense of what the term means. Fewer people can say precisely where the gap comes from, which is a problem, because the fix depends entirely on the source. This post is an attempt at that precision: what the gap actually is, the four places it originates, the four families of methods that close it, and how to tell which one you're dealing with.
What the sim-to-real gap actually is
Robot decision-making is typically modeled as a partially observable Markov decision process: a state space, an action space, transition dynamics, a reward function, an observation space, and a sensor model [1]. A simulator is an approximation of that process. Every one of those components can be approximated wrongly, and each wrong approximation is a different kind of gap. So "the sim-to-real gap" is not one thing. It's a collection of discrepancies that happen to produce the same symptom: a policy that performs well in training and worse in deployment [1]. Two robots can fail for entirely unrelated reasons, and both failures get filed under the same phrase. That matters practically. If your gap is in the transition dynamics, better rendering will not help you. If your gap is in the sensor model, tuning friction coefficients is wasted effort.
Where the gap comes from
Contact and material physics. This is the deepest source and the hardest to fix. Simulators approximate geometry and material properties coarsely, and real objects exhibit behaviors that most engines model poorly or not at all: viscoelasticity, plasticity, hysteresis, and the effects of manufacturing defects that no CAD model contains [1]. Rigid-body contact is an approximation of something that is not rigid and not instantaneous. This is why contact-rich tasks, such as insertion, assembly, deburring, and anything involving sustained force transfer, are worse than free-space motion.
Perception and rendering. A policy trained on RGB images from a renderer is trained on a visual distribution that does not match a real camera in a real building. Lighting, sensor noise, motion blur, specular reflections, the particular way your facility's LEDs flicker — none of it is in the training distribution unless you put it there.
Actuation and latency. Simulated actuators respond to commands more faithfully than real ones. Real systems have backlash, compliance, thermal drift, control-loop delay, and communication jitter. A policy that assumes its commands take effect immediately will overshoot on hardware.
The environment itself. This is the source that operations teams feel most and researchers discuss least. The simulated cell has the fixture in the right place. The real cell has the fixture in roughly the right place, plus a cable someone rerouted, plus a bin that gets bumped every shift, plus the pallet that arrives skewed on Tuesdays.
Why use simulation?
Given all that, it's fair to ask why the field trains in simulation at all. The answer is that the alternative is worse. Training on real hardware is slow, expensive, and sometimes dangerous; in simulation a robot can fail thousands of times in software before it ever touches a machine, and modern simulators run many virtual robots in parallel on a single GPU [3]. That scale has produced genuine milestones. Domain randomization plus large-scale simulation trained a robot hand to solve a Rubik's Cube [6], and simulation-trained policies now underpin much of what works in legged locomotion and agile flight [1][2]. On the data side, the leverage is stark: NVIDIA generated 780,000 synthetic trajectories — the equivalent of roughly nine months of human demonstration — in 11 hours of compute, and combining that synthetic data with real data improved model performance by 40% over real data alone [11]. Simulation isn't the problem. Treating simulation as the finish line is the problem.
The four families of methods that close the gap
The 2026 survey from researchers at the University of Zurich, NVIDIA, and the University of Washington organizes the solution space into four broad approaches [1].
1. Domain randomization: train across the uncertainty
The original insight, from 2017, was that if you randomize simulation parameters widely enough—textures, lighting, and object positions—a policy learns to be robust to variation rather than tuned to one specific setup, and reality becomes just another sample from the distribution [4]. The approach was extended from visual parameters to physical dynamics: mass, friction, damping, and delays [5]. It works, and it has a real failure mode. Randomization ranges that are too broad produce conservative policies that hedge against conditions that never occur, underperforming on the actual system [7]. Robustness bought with breadth is paid for in precision, which is exactly the wrong trade for a task with tight tolerances.
2. System identification: measure the real thing
Rather than randomizing over your ignorance, reduce it. System identification estimates the real system's physical parameters by matching simulated rollouts against real trajectories, then trains at the identified values [7]. Recent work makes the exploration itself active — the robot performs motions selected to be maximally informative about the parameters you're uncertain about. This is a different philosophy from randomization, and in practice the two are complements. Identify what you can measure cheaply; randomize over what you can't.
3. Real-to-sim: build the simulator from reality
The third family inverts the pipeline. Instead of authoring a simulated scene and hoping it matches, reconstruct the scene from real sensor data, then train in the reconstruction [8]. The rendering side of this has moved quickly with 3D Gaussian splatting: SplatSim replaced mesh rendering with Gaussian splats and achieved 86.25% average success transferring RGB manipulation policies to the real world zero-shot, against 97.5% for policies trained on real data directly [10]. Read that comparison carefully, because it's the honest version of the story. Zero-shot transfer from a reconstructed simulator got most of the way to real-data training. Not all of the way. That residual is the gap, and it is smaller than it used to be rather than gone. Real-to-sim also enables something operationally useful that has nothing to do with training: policy evaluation. Reconstructed simulators have been shown to predict real-world policy success rates with correlations above 0.9 on deformable manipulation tasks [9], which means you can rank checkpoints without occupying a real cell for a week.
4. Abstractions and co-training: change what transfers
The last family sidesteps the gap rather than closing it. If your policy consumes depth, point clouds, or proprioception instead of RGB, the sim-to-real gap in that modality is far smaller than it is for photorealistic images [10]. Choosing a representation that simulators reproduce faithfully is often more effective than making simulators more faithful. Sim-real co-training is the other half: train on simulated and real data together so the model sees both distributions, with real data anchoring the parts simulation gets wrong [1].
How to tell which gap you have
Fails in the same place every time, deterministically. Suspect dynamics or actuation. The policy has learned a behavior that assumes a system response it isn't getting.
Fails intermittently, correlated with time of day or shift. Suspect perception. Changing light is the usual culprit, and it's the cheapest to test; run the same task under different lighting and see if the failure rate moves.
Works on the demo part and fails on the variant. Suspect environment distribution. The policy learned the specific case, not the task.
Works in free space, fails on contact. Suspect physics. This is the hardest category and the one where more simulation fidelity helps least, because the gap is in what the engine can represent at all.
The unifying point: you cannot debug this from a success-rate number. You need the failure logged with enough context to attribute it, which is a deployment infrastructure question, not a modeling question. It's one reason we've argued that architectures producing an inspectable prediction before acting, like the world action models we covered recently, matter for operations specifically.
Where 3GEN sits on this
Our position is that perfect simulation fidelity is not attainable and not the goal. The survey literature converges on the same conclusion: robust transfer comes from combining simulation improvements, robust policy learning, and careful system design rather than from any single technique [1]. So we use simulation as a starting point rather than a destination. A policy gets bootstrapped in simulation, then corrected by contact with the actual environment — and critically, the corrections are captured rather than discarded. Every real run, including the failures, feeds back into training through automated training data collection, which turns the sim-to-real gap from a fixed penalty you absorb at deployment into a quantity that shrinks with operating hours. The architecture supports this directly. Our two-tier split keeps a fast local model handling the routine case while escalating unfamiliar situations to a larger model, and the escalations themselves are a labeled record of where the gap still exists. Bellwether closes the loop, collecting field data and pushing improved policies back to the fleet. We're doing this work from inside the Center for Smart Convergent Manufacturing Systems at RPI, alongside manufacturers whose real cells are where these questions get settled.
Frequently asked questions
What is the sim-to-real gap in simple terms? It's the performance drop that occurs when a robot policy trained in simulation is deployed on real hardware. The simulator is an approximation of reality, and the policy has learned the approximation rather than the thing itself.
Can the sim-to-real gap be eliminated? No — not by improving simulators alone. Simulations are abstractions by definition, and some real-world phenomena (material hysteresis, manufacturing variation, wear) are impractical to model exactly [1]. The realistic goal is a gap small enough that real-world adaptation closes the remainder quickly.
Is domain randomization enough on its own? Rarely. It buys robustness at the cost of precision, and overly wide randomization ranges produce policies that underperform on the actual system [7]. Most production approaches combine randomization with measurement of the real system.
How does the sim-to-real gap affect deployment timelines? It's usually the hidden cost. A policy that transfers at 60% needs weeks of on-site tuning before it's production-viable, and that tuning is invisible in a pilot quote. The relevant question for a vendor isn't whether their simulation is good — it's what happens to the residual gap after go-live.
Does more simulation data close the gap? Not by itself. Synthetic data scales volume enormously [11], but every synthetic variant inherits the assumptions of the engine that produced it. Volume helps with coverage; it doesn't correct a systematically wrong physics model.
The gap is a deployment problem, not a simulation problem
The most useful reframe we can offer: stop treating sim-to-real as a modeling challenge to be solved before deployment and start treating it as a measurable quantity to be reduced during it. Simulators will keep improving. They will not converge on reality. What changes the economics is a system that notices where the gap is, captures it, and feeds it back. If you have a task on your floor and you'd like to talk about what transfer actually looks like in your environment, request a demo or reach us at human@3genrobotics.com.
Notes
Bellwether is in alpha. Public demo material illustrates the pipeline conceptually; screens, flow, and terminology are subject to change in the shipped product. Performance figures cited above are drawn from published third-party research and vendor documentation, not from 3GEN internal benchmarks. Success rates reported in sim-to-real research are task- and platform-specific and do not generalize across setups. The SplatSim figures [10] cover four manipulation tasks on a single hardware configuration; the correlation results [9] cover deformable and rigid-body tasks in a single lab. They are cited here to illustrate the shape of the problem, not as benchmarks any deployment should expect to reproduce. Diagnostic heuristics in the "how to tell which gap you have" section are practitioner guidance, not validated findings.
References
[1] Aljalbout, Elie, Jiaxu Xing, Angel Romero, Iretiayo Akinola, Caelan Reed Garrett, Eric Heiden, Abhishek Gupta, Tucker Hermans, Yashraj Narang, Dieter Fox, Davide Scaramuzza, and Fabio Ramos. "The Reality Gap in Robotics: Challenges, Solutions, and Best Practices." Annual Review of Control, Robotics, and Autonomous Systems, Vol. 9, 2026. arXiv:2510.20808. https://arxiv.org/abs/2510.20808
[2] Zhao, Wenshuai, Jorge Peña Queralta, and Tomi Westerlund. "Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: A Survey." IEEE Symposium Series on Computational Intelligence (SSCI), 2020.
[3] Labellerr. "How to Bridge the Reality Gap in Robotics Training." July 2026. https://www.labellerr.com/blog/reality-gap-in-robotics/
[4] Tobin, Josh, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. "Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World." IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2017.
[5] Peng, Xue Bin, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. "Sim-to-Real Transfer of Robotic Control with Dynamics Randomization." IEEE International Conference on Robotics and Automation (ICRA), 2018.
[6] OpenAI et al. "Solving Rubik's Cube with a Robot Hand." arXiv:1910.07113, 2019.
[7] Sobanbabu, Nikhil, Guanqi He, Tairan He, Yuxiang Yang, and Guanya Shi. "Sampling-Based System Identification with Active Exploration for Legged Robot Sim2Real Learning." arXiv:2505.14266, 2025.
[8] Torne, Marcel, Anthony Simeonov, Zechu Li, April Chan, Tao Chen, Abhishek Gupta, and Pulkit Agrawal. "Reconciling Reality Through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation." arXiv:2403.03949, 2024.
[9] "Real-to-Sim Robot Policy Evaluation with Gaussian Splatting Simulation of Soft-Body Interactions." arXiv:2511.04665, 2025. https://real2sim-eval.github.io/
[10] Qureshi, Mohammad Nomaan, Sparsh Garg, Francisco Yandun, David Held, George Kantor, and Abhisesh Silwal. "SplatSim: Zero-Shot Sim2Real Transfer of RGB Manipulation Policies Using Gaussian Splatting." arXiv:2409.10161, 2024. https://splatsim.github.io/
[11] NVIDIA. "NVIDIA Announces Isaac GR00T N1 — the World's First Open Humanoid Robot Foundation Model — and Simulation Frameworks to Speed Robot Development." NVIDIA Newsroom, March 2025. https://nvidianews.nvidia.com/news/nvidia-isaac-gr00t-n1-open-humanoid-robot-foundation-model-simulation-frameworks