Abstract
There is growing interest in studying the languages that emerge when neuralagents are jointly trained to solve tasks requiring communication through adiscrete channel. We investigate here the information-theoretic complexity ofsuch languages, focusing on the basic two-agent, one-exchange setup. We findthat, under common training procedures, the emergent languages are subject toan entropy minimization pressure that has also been detected in human language,whereby the mutual information between the communicating agent's inputs and themessages is minimized, within the range afforded by the need for successfulcommunication. That is, emergent languages are (nearly) as simple as the taskthey are developed for allow them to be. This pressure is amplified as weincrease communication channel discreteness. Further, we observe that strongerdiscrete-channel-driven entropy minimization leads to representations withincreased robustness to overfitting and adversarial attacks. We conclude bydiscussing the implications of our findings for the study of natural andartificial communication systems.