Language understanding research is held back by a failure to relate languageto the physical world it describes and to the social interactions itfacilitates. Despite the incredible effectiveness of language processing modelsto tackle tasks after being trained on text alone, successful linguisticcommunication relies on a shared experience of the world. It is this sharedexperience that makes utterances meaningful. Natural language processing is a diverse field, and progress throughout itsdevelopment has come from new representational theories, modeling techniques,data collection paradigms, and tasks. We posit that the present success ofrepresentation learning approaches trained on large, text-only corpora requiresthe parallel tradition of research on the broader physical and social contextof language to address the deeper questions of communication.