Abstract
While end-to-end neural machine translation (NMT) has achieved notablesuccess in the past years in translating a handful of resource-rich languagepairs, it still suffers from the data scarcity problem for low-resourcelanguage pairs and domains. To tackle this problem, we propose an interactivemultimodal framework for zero-resource neural machine translation. Instead ofbeing passively exposed to large amounts of parallel corpora, our learners(implemented as encoder-decoder architecture) engage in cooperative imagedescription games, and thus develop their own image captioning or neuralmachine translation model from the need to communicate in order to succeed atthe game. Experimental results on the IAPR-TC12 and Multi30K datasets show thatthe proposed learning mechanism significantly improves over thestate-of-the-art methods.