More robot data does not necessarily require more robot time. The counterintuitive idea here is to keep actions grounded in simulation while generating the observations from real-scene images: this makes it possible to vary visual appearance without restaging and teleoperating every trial.
Key Findings
- Across five bimanual tasks, policies trained on 100 generated episodes achieved 93 successes out of 100 trials in the original scenes, compared with 84 for policies trained on the same number of teleoperated episodes.
- Under unseen tablecloths and coloured lighting, the generated-data policies scored 87 successes, while teleoperation-trained policies fell to 46. The result suggests that controlled visual variation can improve robustness to appearance changes.
- The system builds a simulated scene, controller, and success judge from a task request and one calibration episode. Simulation provides the actions; a video model conditioned on real images renders synchronized camera views, and independent checks screen the generated episodes.
- The reported average cost is $0.80 per admitted episode after task adaptation. The study also finds that conventional simulation with domain randomization transfers much less successfully, highlighting the importance of realistic rendered observations in this setup.
Method and Scope
The evaluation uses five tasks on one dual-arm robot, spanning rigid objects, an articulated drawer, rhythmic contact, and cloth folding. This variety makes the results more informative than a single-task demonstration, but it does not establish transfer to other robot embodiments or broad task families. The method also depends on a simulator capable of reproducing the task; uncertain physical properties may need to be randomized rather than inferred from one calibration episode.
Who Should Read It
Great fit for researchers working on robot learning, synthetic data, and reducing the cost of collecting demonstrations, especially when visual robustness matters. Look elsewhere if you need evidence across many robot platforms or tasks that depend on precisely identified physical dynamics: the paper’s real-world evaluation is promising but still limited in breadth.