At least a quarter of the warming that the Earth is experiencing today is dueto anthropogenic methane emissions. There are multiple satellites in orbit andplanned for launch in the next few years which can detect and quantify theseemissions; however, to attribute methane emissions to their sources on theground, a comprehensive database of the locations and characteristics ofemission sources worldwide is essential. In this work, we develop deep learningalgorithms that leverage freely available high-resolution aerial imagery toautomatically detect oil and gas infrastructure, one of the largestcontributors to global methane emissions. We use the best algorithm, which wecall OGNet, together with expert review to identify the locations of oilrefineries and petroleum terminals in the U.S. We show that OGNet detects manyfacilities which are not present in four standard public datasets of oil andgas infrastructure. All detected facilities are associated with characteristicsknown to contribute to methane emissions, including the infrastructure type andthe number of storage tanks. The data curated and produced in this study isfreely available at http://stanfordmlgroup.github.io/projects/ognet .