Benchmarks, such as COCO, play a crucial role in object detection. However,existing benchmarks are insufficient in scale variation, and their protocolsare inadequate for fair comparison. In this paper, we introduce theUniversal-Scale object detection Benchmark (USB). USB has variations in objectscales and image domains by incorporating COCO with the recently proposed WaymoOpen Dataset and Manga109-s dataset. To enable fair comparison, we propose USBprotocols by defining multiple thresholds for training epochs and evaluationimage resolutions. By analyzing methods on the proposed benchmark, we designedfast and accurate object detectors called UniverseNets, which surpassed allbaselines on USB and achieved state-of-the-art results on existing benchmarks.Specifically, UniverseNets achieved 54.1% AP on COCO test-dev with 20 epochstraining, the top result among single-stage detectors on the Waymo Open DatasetChallenge 2020 2D detection, and the first place in the NightOwls DetectionChallenge 2020 all objects track. The code is available athttps://github.com/shinya7y/UniverseNet .