AST-Probe: Recovering abstract syntax trees from hidden representations of pre-trained language models

  • 2022-06-23 15:09:05
  • José Antonio Hernández López, Martin Weyssow, Jesús Sánchez Cuadrado, Houari Sahraoui
  • 9

Abstract

The objective of pre-trained language models is to learn contextualrepresentations of textual data. Pre-trained language models have becomemainstream in natural language processing and code modeling. Using probes, atechnique to study the linguistic properties of hidden vector spaces, previousworks have shown that these pre-trained language models encode simplelinguistic properties in their hidden representations. However, none of theprevious work assessed whether these models encode the whole grammaticalstructure of a programming language. In this paper, we prove the existence of a\textit{syntactic subspace}, lying in the hidden representations of pre-trainedlanguage models, which contain the syntactic information of the programminglanguage. We show that this subspace can be extracted from the models'representations and define a novel probing method, the AST-Probe, that enablesrecovering the whole abstract syntax tree (AST) of an input code snippet. Inour experimentations, we show that this syntactic subspace exists in fivestate-of-the-art pre-trained language models. In addition, we highlight thatthe middle layers of the models are the ones that encode most of the ASTinformation. Finally, we estimate the optimal size of this syntactic subspaceand show that its dimension is substantially lower than those of the models'representation spaces. This suggests that pre-trained language models use asmall part of their representation spaces to encode syntactic information ofthe programming languages.

 

Quick Read (beta)

loading the full paper ...