Chess as a Testbed for Language Model State Tracking

  • 2022-05-13 22:40:30
  • Shubham Toshniwal, Sam Wiseman, Karen Livescu, Kevin Gimpel
  • 0

Abstract

Transformer language models have made tremendous strides in natural languageunderstanding tasks. However, the complexity of natural language makes itchallenging to ascertain how accurately these models are tracking the worldstate underlying the text. Motivated by this issue, we consider the task oflanguage modeling for the game of chess. Unlike natural language, chessnotations describe a simple, constrained, and deterministic domain. Moreover,we observe that the appropriate choice of chess notation allows for directlyprobing the world state, without requiring any additional probing-relatedmachinery. We find that: (a) With enough training data, transformer languagemodels can learn to track pieces and predict legal moves with high accuracywhen trained solely on move sequences. (b) For small training sets providingaccess to board state information during training can yield significantimprovements. (c) The success of transformer language models is dependent onaccess to the entire game history i.e. "full attention". Approximating thisfull attention results in a significant performance drop. We propose thistestbed as a benchmark for future work on the development and analysis oftransformer language models.

 

Quick Read (beta)

loading the full paper ...