Abstract
Machine learning models have become more and more complex in order to betterapproximate complex functions. Although fruitful in many domains, the addedcomplexity has come at the cost of model interpretability. The once populark-nearest neighbors (kNN) approach, which finds and uses the most similar datafor reasoning, has received much less attention in recent decades due tonumerous problems when compared to other techniques. We show that many of thesehistorical problems with kNN can be overcome, and our contribution hasapplications not only in machine learning but also in online learning, datasynthesis, anomaly detection, model compression, and reinforcement learning,without sacrificing interpretability. We introduce a synthesis between kNN andinformation theory that we hope will provide a clear path towards models thatare innately interpretable and auditable. Through this work we hope to gatherinterest in combining kNN with information theory as a promising path to fullyauditable machine learning and artificial intelligence.