Many recent works have discussed the propensity, or lack thereof, foremergent languages to exhibit properties of natural languages. A favorite inthe literature is learning compositionality. We note that most of those workshave focused on communicative bandwidth as being of primary importance. Whileimportant, it is not the only contributing factor. In this paper, weinvestigate the learning biases that affect the efficacy and compositionalityof emergent languages. Our foremost contribution is to explore how capacity ofa neural network impacts its ability to learn a compositional language. Weadditionally introduce a set of evaluation metrics with which we analyze thelearned languages. Our hypothesis is that there should be a specific range ofmodel capacity and channel bandwidth that induces compositional structure inthe resulting language and consequently encourages systematic generalization.While we empirically see evidence for the bottom of this range, we curiously donot find evidence for the top part of the range and believe that this is anopen question for the community.