Abstract
There is a growing interest in studying the languages emerging when neuralagents are jointly trained to solve tasks that require communication throughdiscrete messages. We investigate here the information-theoretic complexity ofsuch languages, focusing on the most basic two-agent, one-symbol, one-exchangesetup. We find that, under common training procedures, the emergent languagesare subject to an information minimization pressure: The mutual informationbetween the communicating agent's inputs and the messages is close to theminimum that still allows the task to be solved. After verifying thisinformation minimization property, we perform experiments showing that astronger discrete-channel-driven information minimization pressure leads toincreased robustness to overfitting and to adversarial attacks. We conclude bydiscussing the implications of our findings for the studies of artificial andnatural language emergence, and for representation learning.