The General Video Game AI (GVGAI) competition and its associated softwareframework provides a way of benchmarking AI algorithms on a large number ofgames written in a domain-specific description language. While the competitionhas seen plenty of interest, it has so far focused on online planning,providing a forward model that allows the use of algorithms such as Monte CarloTree Search. In this paper, we describe how we interface GVGAI to the OpenAI Gymenvironment, a widely used way of connecting agents to reinforcement learningproblems. Using this interface, we characterize how widely used implementationsof several deep reinforcement learning algorithms fare on a number of GVGAIgames. We further analyze the results to provide a first indication of therelative difficulty of these games relative to each other, and relative tothose in the Arcade Learning Environment under similar conditions.