Token Turing Machines

  • 2022-11-16 18:59:18
  • Michael S. Ryoo, Keerthana Gopalakrishnan, Kumara Kahatapitiya, Ted Xiao, Kanishka Rao, Austin Stone, Yao Lu, Julian Ibarz, Anurag Arnab
  • 36


We propose Token Turing Machines (TTM), a sequential, autoregressiveTransformer model with memory for real-world sequential visual understanding.Our model is inspired by the seminal Neural Turing Machine, and has an externalmemory consisting of a set of tokens which summarise the previous history(i.e., frames). This memory is efficiently addressed, read and written using aTransformer as the processing unit/controller at each step. The model's memorymodule ensures that a new observation will only be processed with the contentsof the memory (and not the entire history), meaning that it can efficientlyprocess long sequences with a bounded computational cost at each step. We showthat TTM outperforms other alternatives, such as other Transformer modelsdesigned for long sequences and recurrent neural networks, on two real-worldsequential visual understanding tasks: online temporal activity detection fromvideos and vision-based robot action policy learning.


Quick Read (beta)

loading the full paper ...