Abstract
We propose Token Turing Machines (TTM), a sequential, autoregressiveTransformer model with memory for real-world sequential visual understanding.Our model is inspired by the seminal Neural Turing Machine, and has an externalmemory consisting of a set of tokens which summarise the previous history(i.e., frames). This memory is efficiently addressed, read and written using aTransformer as the processing unit/controller at each step. The model's memorymodule ensures that a new observation will only be processed with the contentsof the memory (and not the entire history), meaning that it can efficientlyprocess long sequences with a bounded computational cost at each step. We showthat TTM outperforms other alternatives, such as other Transformer modelsdesigned for long sequences and recurrent neural networks, on two real-worldsequential visual understanding tasks: online temporal activity detection fromvideos and vision-based robot action policy learning.