Abstract
Evaluation in machine learning is usually informed by past choices, forexample which datasets or metrics to use. This standardization enables thecomparison on equal footing using leaderboards, but the evaluation choicesbecome sub-optimal as better alternatives arise. This problem is especiallypertinent in natural language generation which requires ever-improving suitesof datasets, metrics, and human evaluation to make definitive claims. To makefollowing best model evaluation practices easier, we introduce GEMv2. The newversion of the Generation, Evaluation, and Metrics Benchmark introduces amodular infrastructure for dataset, model, and metric developers to benefitfrom each others work. GEMv2 supports 40 documented datasets in 51 languages.Models for all datasets can be evaluated online and our interactive data cardcreation and rendering tools make it easier to add new datasets to the livingbenchmark.