Towards a theory of out-of-distribution learning

  • 2021-11-24 18:18:39
  • Ali Geisa, Ronak Mehta, Hayden S. Helm, Jayanta Dey, Eric Eaton, Jeffery Dick, Carey E. Priebe, Joshua T. Vogelstein
  • 0

Abstract

What is learning? 20$^{st}$ century formalizations of learning theory --which precipitated revolutions in artificial intelligence -- focus primarily on$\mathit{in-distribution}$ learning, that is, learning under the assumptionthat the training data are sampled from the same distribution as the evaluationdistribution. This assumption renders these theories inadequate forcharacterizing 21$^{st}$ century real world data problems, which are typicallycharacterized by evaluation distributions that differ from the training datadistributions (referred to as out-of-distribution learning). We therefore makea small change to existing formal definitions of learnability by relaxing thatassumption. We then introduce $\mathbf{learning\ efficiency}$ (LE) to quantifythe amount a learner is able to leverage data for a given problem, regardlessof whether it is an in- or out-of-distribution problem. We then define andprove the relationship between generalized notions of learnability, and showhow this framework is sufficiently general to characterize transfer, multitask,meta, continual, and lifelong learning. We hope this unification helps bridgethe gap between empirical practice and theoretical guidance in real worldproblems. Finally, because biological learning continues to outperform machinelearning algorithms on certain OOD challenges, we discuss the limitations ofthis framework vis-\'a-vis its ability to formalize biological learning,suggesting multiple avenues for future research.

 

Quick Read (beta)

loading the full paper ...