Crowd Teaches Robot With Video Games

Crowd Teaches Robot With Video Games

Cornell researchers teach a robot simple human commands by playing a video game.

Kristina Denstitt
  • 27 june 2014

While scientists have created robots that look like humans, teaching a robot to think and respond like a human is a more challenging task. Even the most basic spoken command, such as “Heat a kettle of water on the stove”, has a slew of unspoken assumptions that are understood before the action is taken: the location of the kettle, whether it already contains water, how to turn the faucet on, and so on. In robotics, these complexities are usually handled by writing actions directly into a robot’s code, but this leaves anyone without advanced programming knowledge unable to interact with the robot.

Tell Me Dave is a project by Cornell University Assistant Professor of Computer Science Ashutosh Saxena’s Robot Learning Lab. He and his team of graduate students aim to make robots more practical by teaching them to respond to natural speech commands, and they are relying on help from the public to do it.

The project’s robot is equipped with a 3D camera to scan for objects in its environment, as well as software that identifies these objects and associates them with their common usage. This allows it to locate a kettle in the kitchen, identify it as a kettle, and understand that a kettle is used for heating liquids.

What makes the Tell Me Dave robot unique is its ability to take a string of simple actions and associate them with a spoken command. This robot understands that the vague request “Make ramen noodles” entails heating a pot of water, adding the noodles along with spices, draining the noodles after they’ve cooked, and then serving them in a bowl. At the moment, the robot can make both ramen and an Italian desert called affogato.

With a varied environment and changing commands, the robot gets all the actions required to complete its learned tasks correct 64.9% of the time; however, the robot has not yet realized its full learning potential. Saxena and his team are crowdsourcing the robot’s library of learned instructions with a video game-like interface that anyone can play once they’ve signed up at the project’s website. The research team offers a quick demo showcasing the scope of the game, in which a player guides a simulated robot to complete a task in first-person view:

The team’s hope is that users’ input will quickly create a large crowdsourced library for the Cornell robots to use. “With crowdsourcing at such a scale, robots will learn at a much faster rate,” says Saxena.

At a time when technology has already become inseparable from nearly every aspect of daily life, the next logical step for many is functioning, commercially available, complex artificial intelligence. The Tell Me Dave project blurs the line between science fiction and science reality by providing the next step toward a world where humans turn to robotic helpers for everyday tasks.

Tell Me Dave



Dubai And The Future Of Humanitarian Design

Design & Architecture
Technology october 21, 2016

Concept Camera Designed To Only Take Unique Photos

Camera Restricta is tool that prompts photographers to only capture one-of-a-kind images

Design & Architecture october 21, 2016

Fragrance Will Release The Smell Of Data If Your Private Information Is Being Leaked

The device is designed to create a physical cue for the potential dangers lurking online


Get PSFK's Related Report: Future of Automotive

See All
Retail october 21, 2016

LYNK & CO Is A New Auto Brand That Promises Mobile Connectivity On Wheels

Online access and mobility sharing are driving the company to disrupt the auto industry

Related Expert

Becky LaFranchi

Designing Rooms For Productivity

Travel october 21, 2016

Become A Citizen Of The First Nation In Space

Asgardia is a new concept for a floating society above Earth

Entertainment october 21, 2016

Speaker Displays Song Lyrics As Music Is Played

The device is able to generate the graphics on a translucent screen and retrieve the words from a connected database

AI october 21, 2016

Travel Assistant Scans Your Emails To Make Planning Easier

This AI add-on will sync with your inbox and sends reminders to make sure you don't miss anything important


Future Of Automotive
Scenarios Driving The Digital Transformation Of An Industry

PSFK Op-Ed october 21, 2016

Health Expert: Nutritional Meal Replacements Are A Solution To Corporate Wellness

Ample Foods Founder Connor Young explains why supplements are the next food trend coming to the workplace

PSFK Labs october 21, 2016

PSFK Picks: Top 5 Performance-Enhancing Wearables

Our new report looks at innovations pioneering the future of performance through intelligent activewear and predictive analytics

Millennials october 21, 2016

FOMO Survival Kit Helps Millennials Cope With Social Anxieties

The satirical product is meant to be a playful diversion for people who feel like they are missing out

Food october 21, 2016

New York Restaurant Uses Tomato Sushi As Its Newest Meat Alternative

fresh&co is using sous vide Roma tomatoes to create a vegan option that has the texture and taste of tuna

Advertising october 21, 2016

Red Bull Converts Sao Paulo Payphones Into Data-Driven Bus Schedules

The booths allow city residents to check local transit times through a simple toll-free phone call

Retail october 21, 2016

Why Experiential Events Could Replace Trade Shows

Marketers are seeking creative and impactful new ways to connect with influencers

Children october 21, 2016

Modular Kit Teaches Kids How To Make Their Own Robots

MODI features magnetic modules and a platform for programming to encourage experimentation

Infants october 21, 2016

Work Table Doubles As A Baby Seat

Designer Kunsik Choi created the furniture to facilitate emotional communication between between parents and their children

Technology october 21, 2016

Album Turns Into Something New Each Time It’s Streamed

Bill Baird's new album explores the relationship between time and music through a website crafted by design team, One Pixel Wide

Technology october 21, 2016

Wearable Device And Lamp Recreate Beautiful Sunsets In Your Home

Sun Memories can record up to six hours of natural light and reproduce it via a connected light at a later date

No search results found.