Optimal Subarchitecture Extraction For BERT

  • 2020-10-20 17:53:01
  • Adrian de Wynter, Daniel J. Perry
  • 62

Abstract

We extract an optimal subset of architectural parameters for the BERTarchitecture from Devlin et al. (2018) by applying recent breakthroughs inalgorithms for neural architecture search. This optimal subset, which we referto as "Bort", is demonstrably smaller, having an effective (that is, notcounting the embedding layer) size of $5.5\%$ the original BERT-largearchitecture, and $16\%$ of the net size. Bort is also able to be pretrained in$288$ GPU hours, which is $1.2\%$ of the time required to pretrain thehighest-performing BERT parametric architectural variant, RoBERTa-large (Liu etal., 2019), and about $33\%$ of that of the world-record, in GPU hours,required to train BERT-large on the same hardware. It is also $7.9$x faster ona CPU, as well as being better performing than other compressed variants of thearchitecture, and some of the non-compressed variants: it obtains performanceimprovements of between $0.3\%$ and $31\%$, absolute, with respect toBERT-large, on multiple public natural language understanding (NLU) benchmarks.

 

Quick Read (beta)

loading the full paper ...