Towards Unsupervised Content Disentanglement in Sentence Representations via Syntactic Roles

  • 2022-06-22 16:50:01
  • Ghazi Felhi, Joseph Le Roux, Djam√© Seddah
  • 0


Linking neural representations to linguistic factors is crucial in order tobuild and analyze NLP models interpretable by humans. Among these factors,syntactic roles (e.g. subjects, direct objects,$\dots$) and their realizationsare essential markers since they can be understood as a decomposition ofpredicative structures and thus the meaning of sentences. Starting from a deepprobabilistic generative model with attention, we measure the interactionbetween latent variables and realizations of syntactic roles and show that itis possible to obtain, without supervision, representations of sentences wheredifferent syntactic roles correspond to clearly identified different latentvariables. The probabilistic model we propose is an Attention-DrivenVariational Autoencoder (ADVAE). Drawing inspiration from Transformer-basedmachine translation models, ADVAEs enable the analysis of the interactionsbetween latent variables and input tokens through attention. We also develop anevaluation protocol to measure disentanglement with regard to the realizationsof syntactic roles. This protocol is based on attention maxima for the encoderand on latent variable perturbations for the decoder. Our experiments on rawEnglish text from the SNLI dataset show that $\textit{i)}$ disentanglement ofsyntactic roles can be induced without supervision, $\textit{ii)}$ ADVAEseparates syntactic roles better than classical sequence VAEs and TransformerVAEs, $\textit{iii)}$ realizations of syntactic roles can be separatelymodified in sentences by mere intervention on the associated latent variables.Our work constitutes a first step towards unsupervised controllable contentgeneration. The code for our work is publicly available.


Quick Read (beta)

loading the full paper ...