Abstract
Computation-intensive pretrained models have been taking the lead of manynatural language processing benchmarks such as GLUE. However, energy efficiencyin the process of model training and inference becomes a critical bottleneck.We introduce HULK, a multi-task energy efficiency benchmarking platform forresponsible natural language processing. With HULK, we compare pretrainedmodels' energy efficiency from the perspectives of time and cost. Baselinebenchmarking results are provided for further analysis. The fine-tuningefficiency of different pretrained models can differ a lot among differenttasks and fewer parameter number does not necessarily imply better efficiency.We analyzed such phenomenon and demonstrate the method of comparing themulti-task efficiency of pretrained models. Our platform is available athttps://sites.engineering.ucsb.edu/~xiyou/hulk/.