NVIDIA’s new model lets surgical robots rehearse inside a real time AI simulation

NVIDIA published Cosmos-H-Dreams on Hugging Face’s blog, describing it as a distilled, causal version of its larger Cosmos-H-Surgical-Simulator built specifically so surgical robotics researchers can interact with a generated simulation in real time rather than waiting on a slower, non interactive model. Built on the Cosmos-Predict2.5-2B foundation model and fine tuned on the Open-H-Embodiment dataset, the system generates surgical video from an initial scene frame and a robot’s kinematics, allowing closed loop control by either a human operator or a learned policy. A teacher student distillation process, paired with a technique the team calls self forcing training, brought inference speed from about 10 frames per second up to roughly 160 frames per second on a single RTX PRO 6000 GPU, using an inference library called FlashDreams with optimizations including streaming key value caching and CUDA graph capturing. The model was trained across progressive horizons of 12 to 72 frames and deliberately includes both successful demonstrations and failure cases, so the simulator can represent mistakes as well as clean executions. It supports two real surgical platforms, the da Vinci Research Kit and the Versius system, and offers interfaces ranging from browser based keyboard control to a Meta Quest VR headset to a direct policy connection. As the team put it, a useful surgical simulator has to respond correctly to actions and preserve instrument and scene structure over long rollouts so that conclusions drawn in simulation actually transfer to the physical robot.