Speaking Robot-nese: A New Multi-Modal, Multi-Step Robot Game
June 13, 2017 | New York UniversityEstimated reading time: 1 minute
As robots take over our world, they must not only learn how to communicate with us but also with each other. Recent scholarship has so far demonstrated that it’s possible for two robots to communicate in a shared language in the form of binary vectors. These conversations between the sender (Robot A) and receiver (Robot B) are typically mono-directional, and limited to a fixed number of yes/no answers.
But CDS Master’s student Katrina Evtimova, professor Kyunghyun Cho, Andrew Drozdov (NYU Computer Science), and Douwe Keila (Facebook) all believe that our robots can do better. In “Emergent Language in a Multi-Modal, Multi-Step Referential Game,” they have invented a new conversational game for robots that mimics human communication more closely.
First, the robots in their new game are prompted to have bi-directional conversations, meaning that they are allowed to send as many messages to each other as they would like. Second, they must use their shared language to exchange two different modes of information: text and image.
The game itself, then, unfolds something like this. The sender robot is shown photograph of a mammal, and then instructed to communicate what it saw to the receiving robot . The receiver is prompted to guess what animal the sender saw by reading the textual data, and then asking the sender additional questions about the photograph. This means that the conversation is not merely about shifting information from one robot to another, but exchanging different modes of information (e.g. textual and pictorial) between them. To help the robots, the researchers harnessed the power of neural networks and implemented techniques like visual and textual attention into the game.
The researchers found that the multi-modal, multi-step game not only improved the robots’ predictive accuracy, but that their conversations also became more human-like. Will these developments bring humans and machines closer together—or will we become too close for comfort?
Original by: Cherrie Kwok
Suggested Items
Stan Rak: Elevating the Ideas and Insights of IPC's Thought Leaders Program
04/25/2024 | Stanton Rak, SF Rak CompanyAs a member of the IPC Thought Leaders Program (TLP), I am responsible for identifying knowledge-sharing opportunities that can generate ideas and insights that strengthen the IPC community as well as create a sustainable and lasting future for its members. I am delighted to highlight some of my recent contributions as a member of the TLP.
indie Semiconductor Launches Advanced Smart Connectivity Solutions for In-Cabin Applications
04/23/2024 | indie Semiconductorindie Semiconductor, Inc., an Autotech solutions innovator, has introduced a family of new high-performance video converters and retimers to enable in-cabin connectivity applications, including audio and video transport and device interfacing.
The Need for a Holistic Global Sustainability Standard
04/10/2024 | Michael Ford, Aegis SoftwareNo one can deny that the resources of our fragile planet are finite. The environment seems like a third party, subject to constant degradation. We’re acutely aware of the effects of pollution on our climate, and despite our “throw-away” culture, recycling and recovery of materials has remained relatively expensive, even as we use more energy just to survive.
Technica Leads the Way Into Automation and Machine Learning at IPC APEX EXPO 2024
03/28/2024 | Marcy LaRont, PCB007 MagazineAs a significant partner to PCB fabricators since almost the beginning of multilayer boards, Technica USA brings a lot of expertise to the industry. This year’s IPC APEX EXPO show will showcase their biggest booth yet, and a whole lot of information, discussion and demonstration around automation and AI machine learning. In this interview, Jason Perry discusses Technica's plans for the show, and why attendees should stop by.
Airbus to Acquire INFODAS and Strengthen its Cybersecurity Portfolio
03/25/2024 | AirbusAirbus Defence and Space has entered into an agreement to acquire INFODAS, a Cologne-based, German company that provides cybersecurity and IT solutions in the public sector including for defence and critical infrastructures.