Abstract
In this work we explore the relevance of dropout for modern language models,particularly in the context of models on the scale of <100M parameters. Weexplore it's relevance firstly in the regime of improving the sample efficiencyof models given small, high quality datasets, and secondly in the regime ofimproving the quality of its fit on larger datasets where models may underfit.We find that concordant with conventional wisdom, dropout remains effective inthe overfitting scenario, and that furthermore it may have some relevance forimproving the fit of models even in the case of excess data, as suggested byprevious research. In the process we find that the existing explanation for themechanism behind this performance gain is not applicable in the case oflanguage modelling.