Recent advances in Reinforcement Learning (RL) combined with Deep Learning(DL) have demonstrated impressive performance in complex tasks, includingautonomous driving. The use of RL agents in autonomous driving leads to asmooth human-like driving experience, but the limited interpretability of DeepReinforcement Learning (DRL) creates a verification and certificationbottleneck. Instead of relying on RL agents to learn complex tasks, we proposeHPRL - Hierarchical Program-triggered Reinforcement Learning, which uses ahierarchy consisting of a structured program along with multiple RL agents,each trained to perform a relatively simple task. The focus of verificationshifts to the master program under simple guarantees from the RL agents,leading to a significantly more interpretable and verifiable implementation ascompared to a complex RL agent. The evaluation of the framework is demonstratedon different driving tasks, and NHTSA precrash scenarios using CARLA, anopen-source dynamic urban simulation environment.