Without any means of interpretation, neural networks that predict molecularproperties and bioactivities are merely black boxes. We will unravel theseblack boxes and will demonstrate approaches to understand the learnedrepresentations which are hidden inside these models. We show how singleneurons can be interpreted as classifiers which determine the presence orabsence of pharmacophore- or toxicophore-like structures, thereby generatingnew insights and relevant knowledge for chemistry, pharmacology andbiochemistry. We further discuss how these novel pharmacophores/toxicophorescan be determined from the network by identifying the most relevant componentsof a compound for the prediction of the network. Additionally, we propose amethod which can be used to extract new pharmacophores from a model and willshow that these extracted structures are consistent with literature findings.We envision that having access to such interpretable knowledge is a crucial aidin the development and design of new pharmaceutically active molecules, andhelps to investigate and understand failures and successes of current methods.