Crowd Teaches Robot With Video Games

Crowd Teaches Robot With Video Games

Cornell researchers teach a robot simple human commands by playing a video game.

Kristina Denstitt
  • 27 june 2014

While scientists have created robots that look like humans, teaching a robot to think and respond like a human is a more challenging task. Even the most basic spoken command, such as “Heat a kettle of water on the stove”, has a slew of unspoken assumptions that are understood before the action is taken: the location of the kettle, whether it already contains water, how to turn the faucet on, and so on. In robotics, these complexities are usually handled by writing actions directly into a robot’s code, but this leaves anyone without advanced programming knowledge unable to interact with the robot.

Tell Me Dave is a project by Cornell University Assistant Professor of Computer Science Ashutosh Saxena’s Robot Learning Lab. He and his team of graduate students aim to make robots more practical by teaching them to respond to natural speech commands, and they are relying on help from the public to do it.

The project’s robot is equipped with a 3D camera to scan for objects in its environment, as well as software that identifies these objects and associates them with their common usage. This allows it to locate a kettle in the kitchen, identify it as a kettle, and understand that a kettle is used for heating liquids.

What makes the Tell Me Dave robot unique is its ability to take a string of simple actions and associate them with a spoken command. This robot understands that the vague request “Make ramen noodles” entails heating a pot of water, adding the noodles along with spices, draining the noodles after they’ve cooked, and then serving them in a bowl. At the moment, the robot can make both ramen and an Italian desert called affogato.

With a varied environment and changing commands, the robot gets all the actions required to complete its learned tasks correct 64.9% of the time; however, the robot has not yet realized its full learning potential. Saxena and his team are crowdsourcing the robot’s library of learned instructions with a video game-like interface that anyone can play once they’ve signed up at the project’s website. The research team offers a quick demo showcasing the scope of the game, in which a player guides a simulated robot to complete a task in first-person view:

The team’s hope is that users’ input will quickly create a large crowdsourced library for the Cornell robots to use. “With crowdsourcing at such a scale, robots will learn at a much faster rate,” says Saxena.

At a time when technology has already become inseparable from nearly every aspect of daily life, the next logical step for many is functioning, commercially available, complex artificial intelligence. The Tell Me Dave project blurs the line between science fiction and science reality by providing the next step toward a world where humans turn to robotic helpers for everyday tasks.

Tell Me Dave


+artificial intelligence
+Market Research
+Tell Me Dave

PSFK Launches The Fashion Debrief

Design & Architecture
Automotive Yesterday
Design Yesterday
No search results found.