Crowd Teaches Robot With Video Games

Crowd Teaches Robot With Video Games

Cornell researchers teach a robot simple human commands by playing a video game.

Kristina Denstitt
  • 27 june 2014

While scientists have created robots that look like humans, teaching a robot to think and respond like a human is a more challenging task. Even the most basic spoken command, such as “Heat a kettle of water on the stove”, has a slew of unspoken assumptions that are understood before the action is taken: the location of the kettle, whether it already contains water, how to turn the faucet on, and so on. In robotics, these complexities are usually handled by writing actions directly into a robot’s code, but this leaves anyone without advanced programming knowledge unable to interact with the robot.

Tell Me Dave is a project by Cornell University Assistant Professor of Computer Science Ashutosh Saxena’s Robot Learning Lab. He and his team of graduate students aim to make robots more practical by teaching them to respond to natural speech commands, and they are relying on help from the public to do it.

The project’s robot is equipped with a 3D camera to scan for objects in its environment, as well as software that identifies these objects and associates them with their common usage. This allows it to locate a kettle in the kitchen, identify it as a kettle, and understand that a kettle is used for heating liquids.

What makes the Tell Me Dave robot unique is its ability to take a string of simple actions and associate them with a spoken command. This robot understands that the vague request “Make ramen noodles” entails heating a pot of water, adding the noodles along with spices, draining the noodles after they’ve cooked, and then serving them in a bowl. At the moment, the robot can make both ramen and an Italian desert called affogato.

With a varied environment and changing commands, the robot gets all the actions required to complete its learned tasks correct 64.9% of the time; however, the robot has not yet realized its full learning potential. Saxena and his team are crowdsourcing the robot’s library of learned instructions with a video game-like interface that anyone can play once they’ve signed up at the project’s website. The research team offers a quick demo showcasing the scope of the game, in which a player guides a simulated robot to complete a task in first-person view:

The team’s hope is that users’ input will quickly create a large crowdsourced library for the Cornell robots to use. “With crowdsourcing at such a scale, robots will learn at a much faster rate,” says Saxena.

At a time when technology has already become inseparable from nearly every aspect of daily life, the next logical step for many is functioning, commercially available, complex artificial intelligence. The Tell Me Dave project blurs the line between science fiction and science reality by providing the next step toward a world where humans turn to robotic helpers for everyday tasks.

Tell Me Dave



Lyft Gives Free Rides To Those Who Have Had Too Much To Drink

Syndicated Yesterday

Banned Books Week Urges People To Seek Out Controversial Works

Joining the annual celebration of the right to read, US author Jessica Herthel called for 'more information, more voices' to protect diversity

Augmented & Virtual Reality Yesterday

Outdoor Camp Presented In 360° VR By X Games Gold Medalist

The video features campers riding BMX trails, zip lining through the woods, and performing big-air jumps


Get PSFK's Related Report: Future of Automotive

See All
Travel Yesterday

Boeing Wants Passengers To Control Their In-Flight Experience Through Their Phones

The airline manufacturer is embracing automation through a new generation of mobile travel apps

Augmented / Virtual Reality Yesterday

VR Surgery Videos Offer Interactive Medical Education

Dutch startup MDLinking hopes to globalize communication between students and medical care professionals with virtual reality content

Related Expert

Amber Case

User Experience Design Leader

Culture Yesterday

Use Twitter To Learn A New Language

tDict is an app that uses the social media platform to help you search for words in local dialects

Mobile Yesterday

This Startup Wants To Digitize The Loose Coins In Your Pocket

CoinOut is a new app that lets you save your extra change from cash transactions as electronic funds

Advertising Yesterday

McDonald’s Is Accepting Trash As Currency In Exchange For Burgers

An initiative in Stockholm is trying to keep streets clean while satisfying hunger


Future Of Automotive
Scenarios Driving The Digital Transformation Of An Industry

PSFK Op-Ed Yesterday

Why Building Better Offices Is The Key To Employee Engagement

Interaction Designer and Audio-visual Technologist at ESI Design illustrates the value in creating environments filled with surprise and delight

PSFK Labs september 22, 2016

The Future Of Work: Why Innovation Is Every Employee’s Job

PSFK Labs sits down with management at Johnson & Johnson to learn how the company comes up with their next ‘big idea’

Culture Yesterday

Google Is Using Virtual Paper Airplanes To Bring People Closer Together

The tech giant released an app that lets people throw their good wishes out into the world on the International Day of Peace

Advertising Yesterday

Get Paid For Traveling In San Francisco After Rush Hour

BART Perks rewards commuters who take early or late trains by giving them extra points to trade for money

Culture Yesterday

Artist Designs Covers For Books That Don’t Exist

Published by the fictional 'Specious Books,' the subversive works facilitate a conversation regarding the artistic integrity of graphic designers

Work Yesterday

Editorial Roundtable: The Arrival Of The People-First Workplace

Managed By Q, Soma, Workbar, Primary, AltSchool and thinkPARALLAX enumerate the reasons why companies need an employee-embracing workforce in order to exist

Food Yesterday

Bringing Food Innovation To America’s Crowded Milk Market

a2 Milk's Blake Waltrip, Chief Executive of the USA region, discusses how the distributor plans on bringing the popular drink for the dairy-sensitive to the States


Future Of Work
Cultivating The Next Generation Of Leaders

Financial Services Yesterday

Device Makes Digital Currency Feel Tangible

The concept gadget wants you to experience the highs and lows of spending money

Augmented / Virtual Reality Yesterday

NBC Is Planning To Stream The Presidential Debates In Virtual Reality

Partnering with AltspaceVR, the broadcaster offers another way for Americans to engage with the election season

No search results found.