Learning in High Dimension Always Amounts to Extrapolation

  • 2021-10-18 17:32:25
  • Randall Balestriero, Jerome Pesenti, Yann LeCun
  • 120

Abstract

The notion of interpolation and extrapolation is fundamental in variousfields from deep learning to function approximation. Interpolation occurs for asample $x$ whenever this sample falls inside or on the boundary of the givendataset's convex hull. Extrapolation occurs when $x$ falls outside of thatconvex hull. One fundamental (mis)conception is that state-of-the-artalgorithms work so well because of their ability to correctly interpolatetraining data. A second (mis)conception is that interpolation happensthroughout tasks and datasets, in fact, many intuitions and theories rely onthat assumption. We empirically and theoretically argue against those twopoints and demonstrate that on any high-dimensional ($>$100) dataset,interpolation almost surely never happens. Those results challenge the validityof our current interpolation/extrapolation definition as an indicator ofgeneralization performances.

 

Quick Read (beta)

loading the full paper ...