Text to Robotic Assembly

Part of Adaptive Product Assembly (APA) at Autodesk. Type “a simple chair,” and get a chair a robot actually builds. The pipeline runs a text prompt through a generative 3D model, works out what the resulting shape should be made of, and hands a build sequence to the cell.

Role

Co-author; robotic assembly with the Adaptive Product Assembly (APA) team at Autodesk

Team

Alexander Htet Kyaw, Richa Gupta, Dhruv Shah, Anoop Sinha, Kory Mathewson, Stefanie Pender, Sachin Chitta, Yotto Koga, Faez Ahmed, Lawrence Sass, Randall Davis

Paper

Text to Robotic Assembly of Multi-Component Objects using 3D Generative AI and Vision Language Models. NeurIPS 2025, Creative AI Track.

Stack

Project Bernini (generative 3D), ARaaS, Universal Robots, ABB

The pipeline

Generating a shape is the easy half. The hard part is deciding what the shape should be made of. A chair needs a seat and a back that are flat panels, and legs and a frame that are structure, and nothing in a generated mesh says which is which.

So the pipeline hands that judgement to a vision-language model. It reads the mesh and reasons about geometry and function together to decide which regions want panels and which want structural blocks, zero-shot, with no training on the component library. Get one wrong and you say so in conversation, and it revises.

Across 5 objects judged by 32 people, the model's assignments were preferred 90.6% of the time, against 59.4% for a rule-based baseline and 2.5% for random.

Assembly

The parts are designed to be assembled by a machine rather than a person. Structural blocks are cubes with a lattice interior to keep the weight down, 16 magnets per face in alternating polarity so they can only mate one way, and flat tops the gripper can trust. Panels snap onto them magnetically, with openings cut so the gripper can still reach. A UR20 with Robotiq grippers does the building.

The limits are honest ones. Two component types, common objects, simple prompts. Widening any of those is the next problem.