This survey provides an overview of the evolution of visually grounded modelsof spoken language over the last 20 years. Such models are inspired by theobservation that when children pick up a language, they rely on a wide range ofindirect and noisy clues, crucially including signals from the visual modalityco-occurring with spoken utterances. Several fields have made importantcontributions to this approach to modeling or mimicking the process of learninglanguage: Machine Learning, Natural Language and Speech Processing, ComputerVision and Cognitive Science. The current paper brings together thesecontributions in order to provide a useful introduction and overview forpractitioners in all these areas. We discuss the central research questionsaddressed, the timeline of developments, and the datasets which enabled much ofthis work. We then summarize the main modeling architectures and offer anexhaustive overview of the evaluation metrics and analysis techniques.