Abstract
The Rashomon effect occurs when many different explanations exist for thesame phenomenon. In machine learning, Leo Breiman used this term to describeproblems where many accurate-but-different models exist to describe the samedata. In this work, we study how the Rashomon effect can be useful forunderstanding the relationship between training and test performance, and thepossibility that simple-yet-accurate models exist for many problems. Weintroduce the Rashomon set as the set of almost-equally-accurate models for agiven problem, and study its properties and the types of models it couldcontain. We present the Rashomon ratio as a new measure related to simplicityof model classes, which is the ratio of the volume of the set of accuratemodels to the volume of the hypothesis space; the Rashomon ratio is differentfrom standard complexity measures from statistical learning theory. For ahierarchy of hypothesis spaces, the Rashomon ratio can help modelers tonavigate the trade-off between simplicity and accuracy in a surprising way. Inparticular, we find empirically that a plot of empirical risk vs. Rashomonratio forms a characteristic $\Gamma$-shaped Rashomon curve, whose elbow seemsto be a reliable model selection criterion. When the Rashomon set is large,models that are accurate - but that also have various other useful properties -can often be obtained. These models might obey various constraints such asinterpretability, fairness, monotonicity, and computational benefits.