Robotic simulators are crucial for academic research and education as well asthe development of safety-critical applications. Reinforcement learningenvironments -- simple simulations coupled with a problem specification in theform of a reward function -- are also important to standardize the development(and benchmarking) of learning algorithms. Yet, full-scale simulators typicallylack portability and parallelizability. Vice versa, many reinforcement learningenvironments trade-off realism for high sample throughputs in toy-likeproblems. While public data sets have greatly benefited deep learning andcomputer vision, we still lack the software tools to simultaneously develop --and fairly compare -- control theory and reinforcement learning approaches. Inthis paper, we propose an open-source OpenAI Gym-like environment for multiplequadcopters based on the Bullet physics engine. Its multi-agent and visionbased reinforcement learning interfaces, as well as the support of realisticcollisions and aerodynamic effects, make it, to the best of our knowledge, afirst of its kind. We demonstrate its use through several examples, either forcontrol (trajectory tracking with PID control, multi-robot flight withdownwash, etc.) or reinforcement learning (single and multi-agent stabilizationtasks), hoping to inspire future research that combines control theory andmachine learning.