Abstract
The leading approaches in language modeling are all obsessed with TV shows ofmy youth - namely Transformers and Sesame Street. Transformers this,Transformers that, and over here a bonfire worth of GPU-TPU-neuromorphic waferscale silicon. We opt for the lazy path of old and proven techniques with afancy crypto inspired acronym: the Single Headed Attention RNN (SHA-RNN). Theauthor's lone goal is to show that the entire field might have evolved adifferent direction if we had instead been obsessed with a slightly differentacronym and slightly different result. We take a previously strong languagemodel based only on boring LSTMs and get it to within a stone's throw of astone's throw of state-of-the-art byte level language model results on enwik8.We also achieve state-of-the-art on WikiText-103 - or do we? This work hasundergone no intensive hyperparameter optimization and lived entirely on acommodity desktop machine that made the author's small studio apartment far toowarm in the midst of a San Franciscan summer. The final results are achievablein plus or minus 24 hours on a single GPU as the author is impatient. Theattention mechanism is also readily extended to large contexts and requiresminimal computation. Take that Sesame Street.