Reinforcement learning (RL) has demonstrated its ability to solve highdimensional tasks by leveraging non-linear function approximators. However,these successes are mostly achieved by 'black-box' policies in simulateddomains. When deploying RL to the real world, several concerns regarding theuse of a 'black-box' policy might be raised. In order to make the learnedpolicies more transparent, we propose in this paper a policy iteration schemethat retains a complex function approximator for its internal value predictionsbut constrains the policy to have a concise, hierarchical, and human-readablestructure, based on a mixture of interpretable experts. Each expert selects aprimitive action according to a distance to a prototypical state. A key designdecision to keep such experts interpretable is to select the prototypicalstates from trajectory data. The main technical contribution of the paper is toaddress the challenges introduced by this non-differentiable prototypical stateselection procedure. Experimentally, we show that our proposed algorithm canlearn compelling policies on continuous action deep RL benchmarks, matching theperformance of neural network based policies, but returning policies that aremore amenable to human inspection than neural network or linear-in-featurepolicies.