The objective of pre-trained language models is to learn contextualrepresentations of textual data. Pre-trained language models have becomemainstream in natural language processing and code modeling. Using probes, atechnique to study the linguistic properties of hidden vector spaces, previousworks have shown that these pre-trained language models encode simplelinguistic properties in their hidden representations. However, none of theprevious work assessed whether these models encode the whole grammaticalstructure of a programming language. In this paper, we prove the existence of asyntactic subspace, lying in the hidden representations of pre-trained languagemodels, which contain the syntactic information of the programming language. Weshow that this subspace can be extracted from the models' representations anddefine a novel probing method, the AST-Probe, that enables recovering the wholeabstract syntax tree (AST) of an input code snippet. In our experimentations,we show that this syntactic subspace exists in five state-of-the-artpre-trained language models. In addition, we highlight that the middle layersof the models are the ones that encode most of the AST information. Finally, weestimate the optimal size of this syntactic subspace and show that itsdimension is substantially lower than those of the models' representationspaces. This suggests that pre-trained language models use a small part oftheir representation spaces to encode syntactic information of the programminglanguages.