Speaking Robot-nese: A New Multi-Modal, Multi-Step Robot Game

Reading time ( words)

As robots take over our world, they must not only learn how to communicate with us but also with each other. Recent scholarship has so far demonstrated that it’s possible for two robots to communicate in a shared language in the form of binary vectors. These conversations between the sender (Robot A) and receiver (Robot B) are typically mono-directional, and limited to a fixed number of yes/no answers.

But CDS Master’s student Katrina Evtimova, professor Kyunghyun Cho, Andrew Drozdov (NYU Computer Science), and Douwe Keila (Facebook) all believe that our robots can do better. In “Emergent Language in a Multi-Modal, Multi-Step Referential Game,” they have invented a new conversational game for robots that mimics human communication more closely.

First, the robots in their new game are prompted to have bi-directional conversations, meaning that they are allowed to send as many messages to each other as they would like. Second, they must use their shared language to exchange two different modes of information: text and image.

The game itself, then, unfolds something like this. The sender robot is shown photograph of a mammal, and then instructed to communicate what it saw to the receiving robot . The receiver is prompted to guess what animal the sender saw by reading the textual data, and then asking the sender additional questions about the photograph. This means that the conversation is not merely about shifting information from one robot to another, but exchanging different modes of information (e.g. textual and pictorial) between them. To help the robots, the researchers harnessed the power of neural networks and implemented techniques like visual and textual attention into the game.

The researchers found that the multi-modal, multi-step game not only improved the robots’ predictive accuracy, but that their conversations also became more human-like. Will these developments bring humans and machines closer together—or will we become too close for comfort?

Original by: Cherrie Kwok


Suggested Items

What’s Coming in 3D Printing Technology in 2018

12/27/2017 | Cullen Hilkene, 3Diligent
First, the arrival of extrusion metal printing. Today's extrusion printers are the most prevalent and, arguably, user-friendly 3D Printers in the market. Now, after years of there being zero metal extrusion printers, there will be two in the new year from Desktop Metal and Markforged. These technologies promise new materials and a higher degree of user friendliness for metal printing.

Robots: Eliminating the First Contact with an Enemy Force

03/08/2016 | U.S. Army
"We should be thinking about having a robotic vanguard, particularly for maneuver formations," said Dr. Bob Sadowski. "There's no reason why the first contact with an enemy force should be with a man-platform, because it means that platform is at the greatest risk."

NASA and Star Wars: The Connections Are Strong in This One

12/21/2015 | NASA
NASA astronauts “use the force” every time they launch … from a certain point of view. We have real-world droids and ion engines. We’ve seen dual-sun planets like Tatooine and a moon that eerily resembles the Death Star. And with all the excitement around the premiere of Star Wars: The Force Awakens, the Force will soon be felt 250 miles above Earth on the International Space Station.

Copyright © 2018 I-Connect007. All rights reserved.