Deepak Pathak

Generative AI models are good at producing text, images, and video in part because they have so many examples of human-generated content on which to train. But since navigating the physical world is more complicated, robots aren’t so lucky.

For years, roboticists have either programmed robots to perform specific actions as they encounter familiar obstacles, or trained them to tackle new tasks using hyperrealistic simulations. But most robots still struggle to adapt to new environments or changing conditions. 

Deepak Pathak, 31, is helping robots learn on the fly. His work on adaptive robot learning has made it possible for robots to solve original challenges as they operate in the real world. 

Pathak took an unconventional approach: Instead of training his robots on realistic simulations, he deliberately made his simulations unrealistic—full of cartoonish angles and bizarre terrain—and prone to random changes. Robots in his simulations learn to adapt above all else as the world constantly shifts around them.   

Pathak has also shown that robots can also learn by watching (through a camera) YouTube videos of people performing specific tasks. The robot then practices the skill on its own until it gets it right, through what’s known as self-supervised learning.

With this method, Pathak has shown that robots can learn more than 20 tasks in just a few hours, including cleaning a whiteboard and removing a plug from a socket. However, tasks that require a particular amount of force or pressure are still a challenge. It’s hard for a robot to tell from videos alone, for example, how hard to grip a jar to open it. 

Though Pathak has helped many robots learn lots of simple tasks, he has bigger plans. He wants to create a general-purpose robot that could perform helpful household tasks or even take on dangerous or tedious work, like harvesting crops or stocking warehouse shelves, that humans must perform today. 

To that end, Pathak launched a company called Skild AI in July after raising $300 million. The company aims to build the first foundation model for robotics, which could someday be used to create that general-purpose robot.