Lerrel Pinto

A computer science researcher at New York University, Lerrel Pinto, 31,  wants to see robots in the home that do a lot more than vacuum: “How do we actually create robots that can be a more integral part of our lives, doing chores, doing elder care or rehabilitation—you know, just being there when we need them?”

The problem is that training multiskilled robots requires lots of data. Pinto’s solution is to find novel ways to collect that data—in particular, getting robots to collect it as they learn, an approach called self-supervised learning (a technique also championed by Meta’s chief AI scientist and Pinto’s NYU colleague Yann LeCun, among others).  

The idea of a household robot that can make coffee or wash dishes is decades old. But such machines remain the stuff of science fiction. Recent leaps forward in other areas of AI, especially large language models, made use of enormous data sets scraped from the internet. You can’t do that with robots, says Pinto.

Pinto hit one of his first milestones back in 2016, when he created the world’s largest robotics data set at the time by getting robots to create and label their own training data and running them 24/7 without human supervision.

He and his colleagues have since developed learning algorithms that allow a robot to improve as it fails. A robot arm might fail many times to grasp an object, but the data from those attempts can be used to train a model that succeeds. The team has demonstrated this approach with both a robot arm and a drone, turning each dropped object or collision into a hard-won lesson.