Speaking Robot-nese: A New Multi-Modal, Multi-Step Robot Game


Reading time ( words)

As robots take over our world, they must not only learn how to communicate with us but also with each other. Recent scholarship has so far demonstrated that it’s possible for two robots to communicate in a shared language in the form of binary vectors. These conversations between the sender (Robot A) and receiver (Robot B) are typically mono-directional, and limited to a fixed number of yes/no answers.

But CDS Master’s student Katrina Evtimova, professor Kyunghyun Cho, Andrew Drozdov (NYU Computer Science), and Douwe Keila (Facebook) all believe that our robots can do better. In “Emergent Language in a Multi-Modal, Multi-Step Referential Game,” they have invented a new conversational game for robots that mimics human communication more closely.

First, the robots in their new game are prompted to have bi-directional conversations, meaning that they are allowed to send as many messages to each other as they would like. Second, they must use their shared language to exchange two different modes of information: text and image.

The game itself, then, unfolds something like this. The sender robot is shown photograph of a mammal, and then instructed to communicate what it saw to the receiving robot . The receiver is prompted to guess what animal the sender saw by reading the textual data, and then asking the sender additional questions about the photograph. This means that the conversation is not merely about shifting information from one robot to another, but exchanging different modes of information (e.g. textual and pictorial) between them. To help the robots, the researchers harnessed the power of neural networks and implemented techniques like visual and textual attention into the game.

The researchers found that the multi-modal, multi-step game not only improved the robots’ predictive accuracy, but that their conversations also became more human-like. Will these developments bring humans and machines closer together—or will we become too close for comfort?

Original by: Cherrie Kwok

Share


Suggested Items

Robots: Eliminating the First Contact with an Enemy Force

03/08/2016 | U.S. Army
"We should be thinking about having a robotic vanguard, particularly for maneuver formations," said Dr. Bob Sadowski. "There's no reason why the first contact with an enemy force should be with a man-platform, because it means that platform is at the greatest risk."

NASA and Star Wars: The Connections Are Strong in This One

12/21/2015 | NASA
NASA astronauts “use the force” every time they launch … from a certain point of view. We have real-world droids and ion engines. We’ve seen dual-sun planets like Tatooine and a moon that eerily resembles the Death Star. And with all the excitement around the premiere of Star Wars: The Force Awakens, the Force will soon be felt 250 miles above Earth on the International Space Station.

DIY Electronics is Changing the World

10/30/2015 | Barry Matties, I-Connect007
From young kids to seasoned professionals, these DIY electronic platforms are changing the world. It is inspiring and empowering people to create electronic products that they can imagine—and some are even reaching the commercial marketplace. I-Connect007's Barry Matties conducted the following interview with Randy Schafer of EarthLCD to get the scoop on his work with the Arduino community.



Copyright © 2017 I-Connect007. All rights reserved.