Background and motivation: Deep Reinforcement Learning (Deep RL) is a rapidlydeveloping field. Historically most application has been made to games (such aschess, Atari games, and go). Deep RL is now reaching the stage where it mayoffer value in real world problems, including optimisation of healthcaresystems. One such problem is where to locate ambulances between calls in orderto minimise time from emergency call to ambulance on-scene. This is known asthe Ambulance Location problem. Aim: To develop an OpenAI Gym-compatible framework and simulation environmentfor testing Deep RL agents. Methods: A custom ambulance dispatch simulation environment was developedusing OpenAI Gym and SimPy. Deep RL agents were built using PyTorch. Theenvironment is a simplification of the real world, but allows control over thenumber of clusters of incident locations, number of possible dispatchlocations, number of hospitals, and creating incidents that occur at differentlocations throughout each day. Results: A range of Deep RL agents based on Deep Q networks were tested inthis custom environment. All reduced time to respond to emergency callscompared with random allocation to dispatch points. Bagging Noisy Duelling DeepQ networks gave the most consistence performance. All methods had a tendency tolose performance if trained for too long, and so agents were saved at theiroptimal performance (and tested on independent simulation runs). Conclusion: Deep RL agents, developed using simulated environments, have thepotential to offer a novel approach to optimise the Ambulance Location problem.Creating open simulation environments should allow more rapid progress in thisfield.