MIT's SceneSmith uses AI agents to build richly detailed virtual rooms for training household robots

9:30 AM · July 28, 2026

MIT CSAIL, working with the Toyota Research Institute, published research on July 13, 2026 describing SceneSmith, a system that uses three separate AI agents working together to construct realistic, richly detailed 3D scenes for training household robots in simulation before they attempt tasks in the real world. One agent handles the overall layout and structure of a scene, another populates it with objects, and a third scrutinizes the results, a division of labor the researchers say lets SceneSmith generate rooms with roughly six times more items per scene than prior scene generation methods, closer to the visual clutter of an actual home or garage. Built on a leading vision language model with internet scale knowledge of what real spaces look like, SceneSmith can take a plain language prompt such as ‘generate a garage with a car, a workbench, tires stacked in the corner, and a ladder against the wall’ and produce a virtual environment rich enough for a robot to practice picking up a cup, moving a soda can from a shelf to a table, or placing fruit on plates. The researchers built over 1,300 scenes this way, though the process currently takes multiple hours per scene because each object is generated and closely checked by the agents rather than placed automatically. The team expects that additional computing power could dramatically speed up scene generation, which would let robotics researchers train on far more varied virtual environments than physically building or hand designing each one allows.