Crowd Teaches Robot With Video Games

Crowd Teaches Robot With Video Games

Cornell researchers teach a robot simple human commands by playing a video game.

Kristina Denstitt
  • 27 june 2014

While scientists have created robots that look like humans, teaching a robot to think and respond like a human is a more challenging task. Even the most basic spoken command, such as “Heat a kettle of water on the stove”, has a slew of unspoken assumptions that are understood before the action is taken: the location of the kettle, whether it already contains water, how to turn the faucet on, and so on. In robotics, these complexities are usually handled by writing actions directly into a robot’s code, but this leaves anyone without advanced programming knowledge unable to interact with the robot.

Tell Me Dave is a project by Cornell University Assistant Professor of Computer Science Ashutosh Saxena’s Robot Learning Lab. He and his team of graduate students aim to make robots more practical by teaching them to respond to natural speech commands, and they are relying on help from the public to do it.

The project’s robot is equipped with a 3D camera to scan for objects in its environment, as well as software that identifies these objects and associates them with their common usage. This allows it to locate a kettle in the kitchen, identify it as a kettle, and understand that a kettle is used for heating liquids.

What makes the Tell Me Dave robot unique is its ability to take a string of simple actions and associate them with a spoken command. This robot understands that the vague request “Make ramen noodles” entails heating a pot of water, adding the noodles along with spices, draining the noodles after they’ve cooked, and then serving them in a bowl. At the moment, the robot can make both ramen and an Italian desert called affogato.

With a varied environment and changing commands, the robot gets all the actions required to complete its learned tasks correct 64.9% of the time; however, the robot has not yet realized its full learning potential. Saxena and his team are crowdsourcing the robot’s library of learned instructions with a video game-like interface that anyone can play once they’ve signed up at the project’s website. The research team offers a quick demo showcasing the scope of the game, in which a player guides a simulated robot to complete a task in first-person view:

The team’s hope is that users’ input will quickly create a large crowdsourced library for the Cornell robots to use. “With crowdsourcing at such a scale, robots will learn at a much faster rate,” says Saxena.

At a time when technology has already become inseparable from nearly every aspect of daily life, the next logical step for many is functioning, commercially available, complex artificial intelligence. The Tell Me Dave project blurs the line between science fiction and science reality by providing the next step toward a world where humans turn to robotic helpers for everyday tasks.

Tell Me Dave



Japanese Face Wash Creates A Perfect Rose Every Time

Arts & Culture
Work Today

PSFK’s Workplace Vision: The Desk Is Transformed Into Dynamic System Of Satellite Workstations

Our Future of Work vision is an app that frees employees from the tyranny of a static desk

Work Today

Editorial Roundtable: What A People-First Workplace Must Prioritize First

Managed By Q, Soma, Workbar, Primary, AltSchool and thinkPARALLAX on why employee fulfillment is a journey and not a destination


Get PSFK's Latest Report: Future of Retail: Technology Primer

See All
Food Today

A Tiny Amount Of This Powder Could Block All The Bitterness In Food

A new substance made from mushrooms can bond with taste receptors on your tongue to overcome unpleasant flavors

Asia Today

Whimsical Dental Clinic Design Aims To Calm Patient Fears

Cheerful colors and a nod to the comforts of home are design elements tactfully aimed to help people relax


Rachel Youens

fashion and ecommerce

Mobile Today

Messenger Service Only Delivers You Updates Three Times A Day

Formal is a new app designed to help people ignore the constant stream of texts from their phone and focus on what's in front of them

Design Today

Movable Vacation House Constructed Using Concrete Boxes

The 4,000 square foot building is made up of modular frames so it can be relocated if needed due to coastline erosion

Design Today

Redesigned Electric Fan Collapses For Easy Storage

Conbox can be kept in pieces and packed away when not in use


Future Of Work
Cultivating The Next Generation Of Leaders

PSFK Op-Ed august 23, 2016

Modern Workplace Culture: No More Fat Cats Or Kissing Ass

Samar Birwadker, CEO & Co-Founder of Good & Co, on designing shared organizational values to optimize employee happiness and success

PSFK Labs august 25, 2016

PSFK’s Workplace Vision: How The Nurturing Of Seeds Will Come To Define The Onboarding Process

Our Future of Work vision is a service that allows companies to assemble and deliver welcome packets that are uniquely focused on the concept of growth

Ai Today

LawyerBot Is Now Helping Homeless People Qualify For Government Housing

This robot helps people get out of parking tickets and provides free legal advice to those in need

Travel Today

This Head Hammock Promises The Best Sleep You’ll Ever Get While Traveling

NodPod holds your neck and back perpendicular, hopefully putting an end to sleepless and uncomfortable journeys

Mobile august 26, 2016

Get A Better Idea Of How You Are Wasting Your Time

The TouchTime app is trying to revolutionize personal task management by providing detailed insight on how to be more efficient

Culture august 26, 2016

London Telephone Box Repurposed As A Tiny Mobile Repair Shop

Tools and supplies to replace broken screens or damage are neatly stowed away in these micro-workrooms

Design august 26, 2016

Conceptual Sportswear Created Out Of Futuristic Condom Material

A Dutch fashion designer is experimenting with new methods and fabrics to make high performance clothing


Rio Olympics
Innovation Coverage From The Rio Games

Fashion august 26, 2016

Fashionable Tassel Will Ensure You Never Lose Your Valuables Again

The device is fashion meets connected tech, that will help you keep track of your belongings at all times

Syndicated august 26, 2016

Would You Wear Wool Shoes To Save The Environment?

As demand for wool shoes grows, a number of US footwear brands are heading directly to the source: the sheep pastures of New Zealand

No search results found.