Abstract
Collectively, machine learning (ML) researchers are engaged in the creationand dissemination of knowledge about data-driven algorithms. In a given paper,researchers might aspire to any subset of the following goals, among others: totheoretically characterize what is learnable, to obtain understanding throughempirically rigorous experiments, or to build a working system that has highpredictive accuracy. While determining which knowledge warrants inquiry may besubjective, once the topic is fixed, papers are most valuable to the communitywhen they act in service of the reader, creating foundational knowledge andcommunicating as clearly as possible. Recent progress in machine learning comes despite frequent departures fromthese ideals. In this paper, we focus on the following four patterns thatappear to us to be trending in ML scholarship: (i) failure to distinguishbetween explanation and speculation; (ii) failure to identify the sources ofempirical gains, e.g., emphasizing unnecessary modifications to neuralarchitectures when gains actually stem from hyper-parameter tuning; (iii)mathiness: the use of mathematics that obfuscates or impresses rather thanclarifies, e.g., by confusing technical and non-technical concepts; and (iv)misuse of language, e.g., by choosing terms of art with colloquial connotationsor by overloading established technical terms. While the causes behind these patterns are uncertain, possibilities includethe rapid expansion of the community, the consequent thinness of the reviewerpool, and the often-misaligned incentives between scholarship and short-termmeasures of success (e.g., bibliometrics, attention, and entrepreneurialopportunity). While each pattern offers a corresponding remedy (don't do it),we also discuss some speculative suggestions for how the community might combatthese trends.