Abstract

This paper describes an efficient, open source PyTorch implementation of theSpectral Transform Unit. We investigate sequence prediction tasks over severalmodalities including language, robotics, and simulated dynamical systems. Wefind that for the same parameter count, the STU and its variants outperform theTransformer as well as other leading state space models across variousmodalities.

 

Quick Read (beta)