On the Validity of Self-Attention as Explanation in Transformer Models

  • 2019-08-12 15:48:34
  • Gino Brunner, Yang Liu, Damián Pascual, Oliver Richter, Roger Wattenhofer
  • 9


Explainability of deep learning systems is a vital requirement for manyapplications. However, it is still an unsolved problem. Recent self-attentionbased models for natural language processing, such as the Transformer or BERT,offer hope of greater explainability by providing attention maps that can bedirectly inspected. Nevertheless, by just looking at the attention maps oneoften overlooks that the attention is not over words but over hiddenembeddings, which themselves can be mixed representations of multipleembeddings. We investigate to what extent the implicit assumption made in manyrecent papers - that hidden embeddings at all layers still correspond to theunderlying words - is justified. We quantify how much embeddings are mixedbased on a gradient based attribution method and find that already after thefirst layer less than 50% of the embedding is attributed to the underlyingword, declining thereafter to a median contribution of 7.5% in the last layer.While throughout the layers the underlying word remains as the one contributingmost to the embedding, we argue that attention visualizations are misleadingand should be treated with care when explaining the underlying deep learningsystem.


Quick Read (beta)

On the Validity of Self-Attention as
Explanation in Transformer Models

Gino Brunner, Yang Liu, Damián Pascual, Oliver Richter, Roger Wattenhofer
Department of Electrical Engineering and Information Technology
ETH Zurich
Zurich, Switzerland
Authors in alphabetical order.

Explainability of deep learning systems is a vital requirement for many applications. However, it is still an unsolved problem. Recent self-attention based models for natural language processing, such as the Transformer or BERT, offer hope of greater explainability by providing attention maps that can be directly inspected. Nevertheless, by just looking at the attention maps one often overlooks that the attention is not over words but over hidden embeddings, which themselves can be mixed representations of multiple embeddings. We investigate to what extent the implicit assumption made in many recent papers - that hidden embeddings at all layers still correspond to the underlying words - is justified. We quantify how much embeddings are mixed based on a gradient based attribution method and find that already after the first layer less than 50% of the embedding is attributed to the underlying word, declining thereafter to a median contribution of 7.5% in the last layer. While throughout the layers the underlying word remains as the one contributing most to the embedding, we argue that attention visualizations are misleading and should be treated with care when explaining the underlying deep learning system.

On the Validity of Self-Attention as
Explanation in Transformer Models

Gino Brunner, Yang Liu, Damián Pascual, Oliver Richter, Roger Wattenhofer thanks: Authors in alphabetical order.
Department of Electrical Engineering and Information Technology
ETH Zurich
Zurich, Switzerland

1 Introduction

Deep learning is at the core of the recent outstanding development of artificial intelligence. However, its indisputable empirical success in a huge variety of tasks, ranging from image classification (LeCun et al., 1989; Krizhevsky et al., 2012; He et al., 2016; Szegedy et al., 2017) and image generation (Zhu et al., 2017; Karras et al., 2017) over natural language processing (Vaswani et al., 2017; Radford et al., 2018; 2019; Devlin et al., 2019) to playing complex games (Mnih et al., 2015; Silver et al., 2018), is confronted with lack of explainability (Shwartz-Ziv and Tishby, 2017; Saxe et al., 2018). Deep learning models are known to be “black boxes” (Shwartz-Ziv and Tishby, 2017) whose decisions are hard to explain. Although there have been attempts to quantify the inner operations of neural networks (Shwartz-Ziv and Tishby, 2017), follow up work has shown that these results merely reflect numerical approximation errors (Saxe et al., 2018), while the actual mechanisms are difficult to understand.

Figure 1: Visualization of how information about the word “grass” can end up through multiple paths (in red) in the second embedding in layer 2, which many papers implicitly associate exclusively with the second word, here “cow”. Therefore, if an attention distribution in the following layer attends to this “cow”-associated embedding, it indirectly also attends to the word “grass”.

In the particular case of natural language processing (NLP), the state-of-the-art models proposed in the last year are based on a deep architecture, the Transformer (Vaswani et al., 2017; Devlin et al., 2019), which heavily relies on an operation called self-attention. On top of achieving spectacular success in the most relevant NLP benchmarks, e.g., GLUE (Wang et al., 2019), the Transformer architecture shows promising signs towards a greater interpretability than other deep models. However, looking at the attention distributions brings the pitfall of interpreting it as words attending to each other, when in fact the attention is between (hidden) embeddings, which can themselves be arbitrary mixtures of the underlying words. To see this, take a look at the example in Figure 1: Information about the word “grass” can end up in the representation at position 2, which many papers implicitly associate exclusively with the word “cow”. This implicit assumption can lead to flawed interpretations and diagnostics. Specifically, we identified three use cases, where the assumption is made implicitly without justification:

  • Visualizations/Interpretations that link the attention weights to attention between words, when in fact the attention is between embeddings, i.e., mixtures of multiple words (Vaswani et al., 2017; Devlin et al., 2019; Vig, 2019; Clark et al., 2019; Raganato and Tiedemann, 2018; Voita et al., 2019; Tang et al., 2018; Wangperawong, 2018; Padigela et al., 2019; Baan et al., 2019; Dehghani et al., 2019; Zenkel et al., 2019)

  • Attention accumulation methods that sum the attention to a specific sequence position over layers and/or attention heads, when the given position might encode a different mixture of inputs in each layer (Clark et al., 2019; Baan et al., 2019; Klein and Nabi, 2019; Coenen et al., 2019)

  • Using classifiers to probe embeddings for word specific aspects without investigating to which extent the word is still represented in the hidden embedding (Lin et al., 2019; Peters et al., 2018)

As the amount of research in this area is quite substantial, we aim to provide a quantitative analysis of the degree to which hidden embeddings correspond to given input tokens. We find that the median contribution drops below 50% already after the first layer and decreases further thereafter. Nevertheless, we further find that the word at position i contributes the most to the hidden embeddings at position i throughout all layers, justifying terms such as “contextual word embedding” and the use of embedding probing classifiers (Lin et al., 2019; Tenney et al., 2019; Peters et al., 2018). However, attention visualizations and methods based on accumulated attention weights are harder to justify since they do not account for the mixing.

Our aim with this work is not to put a hold on attention as explanation since insights into deep learning systems can be gained by looking at the learned attention distributions of some architectures (Xu et al., 2015; Mott et al., 2019). However, interpretations should be drawn carefully and should be backed up by attribution methods. Further, we appeal to readers and reviewers of explanatory papers to keep a critical eye regarding the conclusions drawn.

We contribute in this paper with a new attribution method to quantify the mix of hidden embeddings with respect to the input tokens as well as an analysis of BERT (Devlin et al., 2019), measuring the influence of input tokens to the contextual embeddings at the same sentence position. We are currently working on backing these insights with more evidence.

2 Background

The original Transformer model was introduced by Vaswani et al. (2017) and consists of an encoder and a decoder, both in turn consisting of a series of multi-head self-attention layers. At the time, the Transformer brought a major leap forward in neural machine translation performance compared to recurrent and convolutional baselines. In this paper we focus on a Transformer based model called BERT (Devlin et al., 2019), although our analysis can be easily extended to other models such as GPT (Radford et al., 2018; 2019). BERT is an encoder-only Transformer model trained on a large text corpus to predict masked-out words. This unsupervised masked language modeling pre-training task effectively allows BERT to model language bi-directionally. Additionally, BERT is pre-trained on a next sentence prediction task that helps it model language structure beyond single sentence boundaries. This kind of unsupervised pre-training has been shown to allow the model to be fine-tuned on a suite of natural language task and achieve state of the art performance.

In a self-attention layer, as used in all Transformer architectures, every token of the input (word) sequence projects its embedding to a query, key and value vector. The output of the attention layer is then computed for each token as the weighted sum over the value vectors of all tokens, where the weights are calculated as softmax over the dot products between token keys and token queries. More formally, let Qds×dq be the query matrix, Kds×dq the key matrix and Vds×dv the value matrix, where ds is the sequence length and dq and dv the dimension of the query and the value vectors, respectively. Then the output of the attention layer is given by (Vaswani et al., 2017):

Attention(Q,K,V)=AV  withA=softmax(QKTdq)

Note that the attention matrix Ads×ds calculates for each token in the sequence how much the hidden embedding at this sequence position attends to each of the other (hidden) embeddings. It is therefore tempting to assume that each embedding corresponds to the word at that given position, such that the attention distribution reveals an interpretation of how much the words attend to each other. This interpretation is however flawed, as already after the first attention layer, the second attention layer attends over hidden embeddings which are themselves a mixture of the inputs.

3 Methods

To quantitatively analyze how much a given input token ti at position i contributes to the embedding ejl at position j after layer l, we turn to input attribution methods. Specifically, we use a gradient based attribution method (Simonyan et al., 2014) to investigate the model’s sensitivity with respect to its inputs.

Gradient based attribution approximates the neural network function f(x) around a given input x by the linear part of the Taylor expansion. With this, the network sensitivity is analyzed by looking at how small changes at the input correlate with changes at the output. Since in the linear approximation this change is given by the gradient δf(x)δxi for a change in the i-th input token xi of x, the attribution of how much input token xi affects the network output f(x) can be approximated by the L2 norm of the gradient


Note, that if we are interested in discovering which input token led to a given output, the L2 norm is a bad choice (Pörner et al., 2018), since it does not take into account whether the input token contributed positively or negatively to the network decision. However, since our focus is on quantifying how much each input token contributed, irrespective of the output, the L2 norm of the gradient is a natural choice.

To investigate how much a given input token contributes to a hidden embedding, we adapt the input gradient attribution method as follows: When calculating the gradient of the network output f(x) with respect to the input token xi, we only consider gradient information back-propagated through the target embedding ejl. That is as to say, we are interested in how sensitive the output is with respect to input token xi through embedding ejl. Note that this is equivalent to calculating an attribution gradient for each dimension of embedding ejl and then weighting these attribution gradients according to how sensitive the output is to the corresponding embedding dimension. After calculating a gradient for every input token xi we take the L2 norm and normalize over all input tokens to get a percentage estimate of how much input token xi contributed to embedding ejl. Formally, we define the contribution ci,jl as

ci,jl=||gi,jl||2k=0ds||gk,jl||2  withgi,jl=δf(x)δejlδejlδxi

As network output f(x) we take the logit output of the model and calculate the gradients with respect to the position augmented input embeddings xi:=ei0.

4 Experimental Results

We target in our evaluation the pre-trained uncased BERT base model as provided by Devlin et al. (2019)11 1 https://github.com/google-research/bert and use the data from the Microsoft Research Paraphrase Corpus (MRPC) dataset (Dolan and Brockett, 2005). For the subsequent experiments we fine-tune BERT on MRPC using the code of Devlin et al. Note that we will repeat the experiments on data from Wikipedia using the non-fine-tuned BERT to be in line with prior work  (Vig, 2019; Clark et al., 2019; Klein and Nabi, 2019; Coenen et al., 2019; Lin et al., 2019).

Figure 2: (a) Contribution and (b) ranking of the contribution of the input token to the embedding at the same position per layer. The orange line represents the median value. Outliers are not shown.

In our experiments, we calculate the gradients of the logits on the MRPC evaluation set, which consists of 408 examples with a sequence length ds that ranges from 26 to 92 tokens, with 58 tokens on average. We then use these gradients to compute the value of the token contribution ci,jl for each embedding in each layer in each example. We report results for layers 1 to 11 since in layer 12 only the embedding at the CLS token position is used to output the logit and therefore no gradient information exists for the other embeddings.

In Figure 1(a) we report the contribution of the tokens xi given at the input of the model to their corresponding hidden embeddings eil at the same position i for each layer l. We observe that already after the first layer the median contribution of the input token is less than half (42.6%). Furthermore, the contribution decreases monotonically in deeper layers, at layer 5 the median contribution is only 15.8% and after the last layer the median is 7.5%. These results show that self-attention strongly mixes the information contained in the input tokens and after few layers the original input only contributes weakly to the embedding.

Despite the low median contribution value, we are also interested in knowing whether the corresponding input token is the token with the largest contribution when compared to all other input tokens. To test this statement, we calculate the ranking of the contribution of each input token xi to its corresponding embeddings eil in each layer. This ranking illustrates the contribution of the token xi in comparison to all other tokens xk, k0,,ds and ki. Therefore, if an input token xi has the largest contribution to its corresponding embedding eil, its ranking is 1. However, if there are other input tokens xk that contribute more to eil, the ranking value will be higher. Figure 1(b) displays how the ranking of contributions is distributed in each layer. Indeed, down to layer 8 all the tokens have ranking 1, i.e., the largest contribution to the embedding eil comes from the corresponding input token xi. Only for the last three layers there is variability, with some tokens having higher ranking values.

Figure 3: Contribution per POS tag in (a) layer 1 and (b) layer 11 ordered according to contribution in layer 11.

The results presented so far give an idea of how the information is overall mixed at each layer of BERT. Additionally, different types of words may behave differently. To test this hypothesis we perform Part-Of-Speech (POS) tagging on the whole MRPC evaluation set using the Stanford POS tagger (Toutanova et al., 2003), which employs the Penn Treebank tag set (Marcus et al., 1993). Since some symbols are not included in the original Penn Treebank tag set, we add the following symbols to the symbol tag (SYM): dollar sign, dot, comma, colon and quotation marks. Furthermore, we add two categories, CLS and SEP, which respectively correspond to the starting and separation tokens in the MRPC and next sentence prediction tasks and have no semantic meaning, and we remove the categories “UH”, “NNPS” and “WP$”, since they do not occur in the dataset. Then, we run the same experiments as above on each of the subsets of words with a given tag.

Figure 3 shows the token contribution per POS tag in layers 1 and 11 (results for other layers are reported in the Appendix 7). We observe that in the first layer no obvious pattern exist and as we move towards deeper layers the general trend of decreasing contribution affects all POS tags, with no word type having a larger median contribution than 13.8% in the last layer. However, we also see that in deeper layers the four major word classes of English22 2 As defined by the Cambridge Dictionary keep a larger contribution than other word classes; specifically, the POS tags corresponding to nouns (NN, NNP, NNS), adjectives (JJ, JJR, JJS), adverbs (RB, RBR) and some types of verbs (VB, VBG, VBN). Besides the major word classes, the seemingly less important word types of cardinal numbers (CD) and list item markers (LS) also keep a large contribution. Similarly, Figure 4 shows that the ranking for the mentioned POS tags is always 1 in the last layer, with the exception of NNP. Finally, the SEP token has a much larger median ranking value (27), which points towards SEP gathering contextual information from the whole sentence or possibly being used as a no-op (Clark et al., 2019).

Figure 4: Ranking per POS tag in layer 11, SEP median value is 27 and lays outside the plot to avoid distortion.

The results from this section demonstrate that input information is heavily mixed by the successive self-attention layers of BERT. The fact that the ranking of the corresponding input token is mostly 1 points towards the identity of the token being somehow preserved and the embeddings constituting “contextual embeddings”. However, regardless of the word type the contribution of the original token is so small that the intermediate embeddings cannot be assumed to be equivalent to the original input tokens and therefore, interpreting the behavior of BERT, and more broadly of Transformers, by looking at these intermediate representations should be done with care. Specifically, looking at attention maps beyond the first layer to draw conclusions about which word pays attention to which word or how much attention a word gets is not justified, since the embeddings are a mixture of many words.

5 Related Work

The success of BERT (Devlin et al., 2019) on a wide range of NLP tasks has spawned a line of work trying to understand the inner workings of the model. There exists a range of explanation methods that could be directly applied to models such as BERT. In general, these methods aim at explaining a model’s prediction with respect to the input, and we refer to (Pörner et al., 2018) for a comparison of multiple explanation methods for NLP. In our work, we use input gradient attribution (Simonyan et al., 2014) to quantify the influence an input token has on the hidden embeddings at each layer of the transformer. To the best of our knowledge, we propose a novel way to apply input gradient attribution to multi-layer multi-head self-attention models in order to quantify the mixing of tokens across layers.

While calculating input gradient attributions is relatively straightforward, it nevertheless represents an additional layer of complexity. Intuitively, it makes sense to directly look at what a model is “attending” to and use the attention distribution directly as explanation. However, (Jain and Wallace, 2019) show empirically that for bi-directional LSTM and CNN based models, attention does not provide explanation of model behaviour, as the attention distribution generally does not correlate well with gradient-based input attribution. Nonetheless, these results do not necessarily generalize to Transformer architectures, since there are a few crucial differences between these models. For one, transformers use self-attention, where every token computes an attention distribution over all tokens. Second, while attention is like an add-on to RNN and CNN based sequence to sequence models, it is the core part of the transformer architecture and makes up a considerable amount of its parameters. Additionally, self-attention has also been called “self-explanatory” (Pörner et al., 2018). While it is tempting to assume that one can directly interpret self-attention and use it to explain a models decision, it is by no means obvious that this is the case. Especially since self-attention is a highly non-local operation and thus tokens could become mixed in arbitrarily complex and non-interpretable ways, especially in deeper layers.

Some works on investigating transformer architectures have acknowledged this issue, but have either ignored it for the sake of simplicity (Baan et al., 2019) or only partially addressed it (Marecek and Rosa, 2018). In particular, Marecek and Rosa (2018) try to aggregate the attention over the layers to account for the mixing. They make the assumption that the skip connections “boost” self-attention and then heuristically adapt the attention distribution to take mixing into account. In contrast, we use a theoretically justified gradient-based attribution to quantify how the tokens are mixed at each layer. (Peters et al., 2018) compute the cosine similarity between hidden embeddings for bi-LSTMS, bi-CNNs and a version of a bi-directional Transformer. Their visualizations show that as embeddings are passed through the layers, they generally become more similar to one another, giving a hint that the embeddings are mixed and become increasingly contextualized. However, it is not possible with this technique to directly quantify how much each input token contributes to the hidden embeddings.

6 Conclusion

In this work we provide insights into how information propagates through Transformer models. To this end, we propose a new attribution method based on gradient attribution that traces the contribution of the input tokens to the intermediate embeddings. Our results on the BERT model demonstrate that the input information is rapidly mixed and after few layers the input tokens contribute weakly to their corresponding embedding. This points out that directly visualizing hidden attention distributions is misleading since the hidden embeddings no longer represent the input token but a mixture of different tokens. Instead, one should rely on the input gradient attribution as presented in this paper to evaluate the contribution of input tokens to hidden embeddings. We further show that an input token at position i contributes the most to the hidden embeddings at position i compared to all other input tokens. This result suggests that one can still treat a hidden embedding as corresponding to the original word, at least to the degree quantified by our gradient attribution method. As part of our ongoing work on this topic we will conduct further experiments to strengthen our claims.


The authors would like to thank Massimiliano Ciaramita for the initial discussions on the topic that sparked the investigations presented here.


  • J. Baan, M. ter Hoeve, M. van der Wees, A. Schuth, and M. de Rijke (2019) Do transformer attention heads provide transparency in abstractive summarization?. CoRR abs/1907.00570. External Links: 1907.00570 Cited by: 1st item, 2nd item, §5.
  • K. Clark, U. Khandelwal, O. Levy, and C. D. Manning (2019) What does BERT look at? an analysis of bert’s attention. CoRR abs/1906.04341. External Links: 1906.04341 Cited by: 1st item, 2nd item, §4, §4.
  • A. Coenen, E. Reif, A. Yuan, B. Kim, A. Pearce, F. B. Viégas, and M. Wattenberg (2019) Visualizing and measuring the geometry of BERT. CoRR abs/1906.02715. External Links: 1906.02715 Cited by: 2nd item, §4.
  • M. Dehghani, S. Gouws, O. Vinyals, J. Uszkoreit, and L. Kaiser (2019) Universal transformers. See DBLP:conf/iclr/2019, Cited by: 1st item.
  • J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019) BERT: pre-training of deep bidirectional transformers for language understanding. See DBLP:conf/naacl/2019-1, pp. 4171–4186. Cited by: 1st item, §1, §1, §1, §2, §4, §5.
  • W. B. Dolan and C. Brockett (2005) Automatically constructing a corpus of sentential paraphrases. See DBLP:conf/acl-iwp/2005, Cited by: §4.
  • K. He, X. Zhang, S. Ren, and J. Sun (2016) Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770–778. Cited by: §1.
  • S. Jain and B. C. Wallace (2019) Attention is not explanation. See DBLP:conf/naacl/2019-1, pp. 3543–3556. Cited by: §5.
  • T. Karras, T. Aila, S. Laine, and J. Lehtinen (2017) Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196. Cited by: §1.
  • T. Klein and M. Nabi (2019) Attention is (not) all you need for commonsense reasoning. See DBLP:conf/acl/2019-1, pp. 4831–4836. Cited by: 2nd item, §4.
  • A. Krizhevsky, I. Sutskever, and G. E. Hinton (2012) ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (Eds.), pp. 1097–1105. Cited by: §1.
  • Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel (1989) Backpropagation applied to handwritten zip code recognition. Neural computation 1 (4), pp. 541–551. Cited by: §1.
  • Y. Lin, Y. C. Tan, and R. Frank (2019) Open sesame: getting inside bert’s linguistic knowledge. arXiv preprint arXiv:1906.01698. Cited by: 3rd item, §1, §4.
  • M. P. Marcus, B. Santorini, and M. A. Marcinkiewicz (1993) Building a large annotated corpus of english: the penn treebank. Computational Linguistics 19 (2), pp. 313–330. Cited by: §4.
  • D. Marecek and R. Rosa (2018) Extracting syntactic trees from transformer encoder self-attentions. See DBLP:conf/emnlp/2018blackbox, pp. 347–349. Cited by: §5.
  • V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. A. Riedmiller, A. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wierstra, S. Legg, and D. Hassabis (2015) Human-level control through deep reinforcement learning. Nature. External Links: Document Cited by: §1.
  • A. Mott, D. Zoran, M. Chrzanowski, D. Wierstra, and D. J. Rezende (2019) Towards interpretable reinforcement learning using attention augmented agents. CoRR abs/1906.02500. External Links: 1906.02500 Cited by: §1.
  • H. Padigela, H. Zamani, and W. B. Croft (2019) Investigating the successes and failures of BERT for passage re-ranking. CoRR abs/1905.01758. External Links: 1905.01758 Cited by: 1st item.
  • M. E. Peters, M. Neumann, L. Zettlemoyer, and W. Yih (2018) Dissecting contextual word embeddings: architecture and representation. See DBLP:conf/emnlp/2018, pp. 1499–1509. Cited by: 3rd item, §1, §5.
  • N. Pörner, H. Schütze, and B. Roth (2018) Evaluating neural network explanation methods using hybrid documents and morphosyntactic agreement. See DBLP:conf/acl/2018-1, pp. 340–350. Cited by: §3, §5, §5.
  • A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever (2018) Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/researchcovers/languageunsupervised/language understanding paper. pdf. Cited by: §1, §2.
  • A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever (2019) Language models are unsupervised multitask learners. OpenAI Blog 1 (8). Cited by: §1, §2.
  • A. Raganato and J. Tiedemann (2018) An analysis of encoder representations in transformer-based machine translation. See DBLP:conf/emnlp/2018blackbox, pp. 287–297. Cited by: 1st item.
  • A. M. Saxe, Y. Bansal, J. Dapello, M. Advani, A. Kolchinsky, B. D. Tracey, and D. D. Cox (2018) On the information bottleneck theory of deep learning. See DBLP:conf/iclr/2018, Cited by: §1.
  • R. Shwartz-Ziv and N. Tishby (2017) Opening the black box of deep neural networks via information. ArXiv abs/1703.00810. Cited by: §1.
  • D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez, M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Simonyan, and D. Hassabis (2018) A general reinforcement learning algorithm that masters chess, shogi, and go through self-play. Science 362 (6419), pp. 1140–1144. External Links: Document, ISSN 0036-8075, https://science.sciencemag.org/content/362/6419/1140.full.pdf Cited by: §1.
  • K. Simonyan, A. Vedaldi, and A. Zisserman (2014) Deep inside convolutional networks: visualising image classification models and saliency maps. See DBLP:conf/iclr/2014w, Cited by: §3, §5.
  • C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A. Alemi (2017) Inception-v4, inception-resnet and the impact of residual connections on learning. In Thirty-First AAAI Conference on Artificial Intelligence, Cited by: §1.
  • G. Tang, R. Sennrich, and J. Nivre (2018) An analysis of attention mechanisms: the case of word sense disambiguation in neural machine translation. See DBLP:conf/wmt/2018, pp. 26–35. Cited by: 1st item.
  • I. Tenney, P. Xia, B. Chen, A. Wang, A. Poliak, R. T. McCoy, N. Kim, B. V. Durme, S. R. Bowman, D. Das, and E. Pavlick (2019) What do you learn from context? probing for sentence structure in contextualized word representations. See DBLP:conf/iclr/2019, Cited by: §1.
  • K. Toutanova, D. Klein, C. D. Manning, and Y. Singer (2003) Feature-rich part-of-speech tagging with a cyclic dependency network. See DBLP:conf/naacl/2003, Cited by: §4.
  • A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin (2017) Attention is all you need. See DBLP:conf/nips/2017, pp. 5998–6008. Cited by: 1st item, §1, §1, §2, §2.
  • J. Vig (2019) Visualizing attention in transformer-based language representation models. CoRR abs/1904.02679. External Links: 1904.02679 Cited by: 1st item, §4.
  • E. Voita, D. Talbot, F. Moiseev, R. Sennrich, and I. Titov (2019) Analyzing multi-head self-attention: specialized heads do the heavy lifting, the rest can be pruned. See DBLP:conf/acl/2019-1, pp. 5797–5808. Cited by: 1st item.
  • A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman (2019) GLUE: A multi-task benchmark and analysis platform for natural language understanding. See DBLP:conf/iclr/2019, Cited by: §1.
  • A. Wangperawong (2018) Attending to mathematical language with transformers. CoRR abs/1812.02825. External Links: 1812.02825 Cited by: 1st item.
  • K. Xu, J. Ba, R. Kiros, K. Cho, A. C. Courville, R. Salakhutdinov, R. S. Zemel, and Y. Bengio (2015) Show, attend and tell: neural image caption generation with visual attention. See DBLP:conf/icml/2015, pp. 2048–2057. Cited by: §1.
  • T. Zenkel, J. Wuebker, and J. DeNero (2019) Adding interpretable attention to neural translation models improves word alignment. CoRR abs/1901.11359. External Links: 1901.11359 Cited by: 1st item.
  • J. Zhu, T. Park, P. Isola, and A. A. Efros (2017) Unpaired image-to-image translation using cycle-consistent adversarial networks. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 2242–2251. Cited by: §1.

7 Appendix

Here we show the contribution per POS tag in all layers, i.e., 1 to 11, order according to the contribution in layer 11, as in the main text.

Figure 5: Layer 1
Figure 6: Layer 2
Figure 7: Layer 3
Figure 8: Layer 4
Figure 9: Layer 5
Figure 10: Layer 6
Figure 11: Layer 7
Figure 12: Layer 8
Figure 13: Layer 9
Figure 14: Layer 10
Figure 15: Layer 11