Deep neural networks have proved to be a very effective way to performclassification tasks. They excel when the input data is high dimensional, therelationship between the input and the output is complicated, and the number oflabeled training examples is large. But it is hard to explain why a learnednetwork makes a particular classification decision on a particular test case.This is due to their reliance on distributed hierarchical representations. Ifwe could take the knowledge acquired by the neural net and express the sameknowledge in a model that relies on hierarchical decisions instead, explaininga particular decision would be much easier. We describe a way of using atrained neural net to create a type of soft decision tree that generalizesbetter than one learned directly from the training data.