Navigation research is attracting renewed interest with the advent oflearning-based methods. However, this new line of work is largely disconnectedfrom well-established classic navigation approaches. In this paper, we take astep towards coordinating these two directions of research. We set up classicand learning-based navigation systems in common simulated environments andthoroughly evaluate them in indoor spaces of varying complexity, with access todifferent sensory modalities. Additionally, we measure human performance in thesame environments. We find that a classic pipeline, when properly tuned, canperform very well in complex cluttered environments. On the other hand, learnedsystems can operate more robustly with a limited sensor suite. Both approachesare still far from human-level performance.