Dodrio: Exploring Transformer Models with Interactive Visualization

  • 2021-03-26 17:39:37
  • Zijie J. Wang, Robert Turko, Duen Horng Chau
  • 70

Abstract

Why do large pre-trained transformer-based models perform so well across awide variety of NLP tasks? Recent research suggests the key may lie inmulti-headed attention mechanism's ability to learn and represent linguisticinformation. Understanding how these models represent both syntactic andsemantic knowledge is vital to investigate why they succeed and fail, what theyhave learned, and how they can improve. We present Dodrio, an open-sourceinteractive visualization tool to help NLP researchers and practitionersanalyze attention mechanisms in transformer-based models with linguisticknowledge. Dodrio tightly integrates an overview that summarizes the roles ofdifferent attention heads, and detailed views that help users compare attentionweights with the syntactic structure and semantic information in the inputtext. To facilitate the visual comparison of attention weights and linguisticknowledge, Dodrio applies different graph visualization techniques to representattention weights with longer input text. Case studies highlight how Dodrioprovides insights into understanding the attention mechanism intransformer-based models. Dodrio is available athttps://poloclub.github.io/dodrio/.

 

Quick Read (beta)

loading the full paper ...