Abstract
Machine learning models are notoriously difficult to interpret and debug.This is particularly true of neural networks. In this work, we introduceautomated software testing techniques for neural networks that are well-suitedto discovering errors which occur only for rare inputs. Specifically, wedevelop coverage-guided fuzzing (CGF) methods for neural networks. In CGF,random mutations of inputs to a neural network are guided by a coverage metrictoward the goal of satisfying user-specified constraints. We describe how fastapproximate nearest neighbor algorithms can provide this coverage metric. Wethen discuss the application of CGF to the following goals: finding numericalerrors in trained neural networks, generating disagreements between neuralnetworks and quantized versions of those networks, and surfacing undesirablebehavior in character level language models. Finally, we release an open sourcelibrary called TensorFuzz that implements the described techniques.