Improving Simple Models with Confidence Profiles

  • 2018-07-19 15:58:14
  • Amit Dhurandhar, Karthikeyan Shanmugam, Ronny Luss, Peder Olsen
  • 2

Abstract

In this paper, we propose a new method called ProfWeight for transferringinformation from a pre-trained deep neural network that has a high testaccuracy to a simpler interpretable model or a very shallow network of lowcomplexity and a priori low test accuracy. We are motivated by applications ininterpretability and model deployment in severely memory constrainedenvironments (like sensors). Our method uses linear probes to generateconfidence scores through flattened intermediate representations. Our transfermethod involves a theoretically justified weighting of samples during thetraining of the simple model using confidence scores of these intermediatelayers. The value of our method is first demonstrated on CIFAR-10, where ourweighting method significantly improves (3-4%) networks with only a fraction ofthe number of Resnet blocks of a complex Resnet model. We further demonstrateoperationally significant results on a real manufacturing problem, where wedramatically increase the test accuracy of a CART model (the domain standard)by roughly 13%.

 

Quick Read (beta)

loading the full paper ...