90% success rate in novel environments with 0 additional data or training.
Robot models, particularly those trained with large amounts of data, have recently shown a plethora of real-world manipulation and navigation capabilities. Several independent efforts have shown that given sufficient training data in an environment, robot policies can generalize to demonstrated variations in that environment. However, needing to finetune robot models to every new environment stands in stark contrast to models in language or vision that can be deployed zero-shot for open-world problems. In this work, we present Robot Utility Models (RUMs), a framework for training and deploying zero-shot robot policies that can directly generalize to new environments without any finetuning. To create RUMs efficiently, we develop new tools to quickly collect data for mobile manipulation tasks, integrate such data into a policy with multi-modal imitation learning, and deploy policies on-device on Hello Robot Stretch, a cheap commodity robot, with an external mLLM verifier for retrying. We train five such utility models for opening cabinet doors, opening drawers, picking up napkins, picking up paper bags, and reorienting fallen objects. Our system, on average, achieves 90% success rate in unseen, novel environments interacting with unseen objects. Moreover, the utility models can also succeed in different robot and camera set-ups with no further data, training, or fine-tuning. Primary among our lessons are the importance of training data over training algorithm and policy class, guidance about data scaling, necessity for diverse yet high-quality demonstrations, and a recipe for robot introspection and retrying to improve performance on individual environments.
Our RUMs attempted 5 tasks, each in 5+ environments, on a Hello Robot Stretch. They also attempted a few tasks on an xArm. See sample rollouts below:
We feed in a summary of robot observations into a multimodal LLM, which determines whether or not the task at hand has succeeded. If the mLLM determines that the task has failed, the robot automatically resets to a new initial state and retries.
We've made it possible to add the Stick gripper onto your own robot arm with a 3D-printed mount and Dynamixel set, allowing for an identical POV. Thus facilitating seemless zero-shot transfer of policies to new robots.
We release the training dataset for our Robot Utility Models, containing 5 tasks, each with on average ~1000 training demonstrations across 36 environments. The dataset contains RGB videos at 30 fps, as well as full action annotations for 6D pose of the gripper and the gripper's opening angle normalized between (0, 1).