A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks

  • 2018-11-14 19:42:03
  • Victor Sanh, Thomas Wolf, Sebastian Ruder
  • 22

Abstract

Much efforts has been devoted to evaluate whether multi-task learning can beleveraged to learn rich representations that can be used in various NaturalLanguage Processing (NLP) down-stream applications. However, there is still alack of understanding of the settings in which multi-task learning has asignificant effect. In this work, we introduce a hierarchical model trained ina multi-task learning setup on a set of carefully selected semantic tasks. Themodel is trained in a hierarchical fashion to introduce an inductive bias bysupervising a set of low level tasks at the bottom layers of the model and morecomplex tasks at the top layers of the model. This model achievesstate-of-the-art results on a number of tasks, namely Named Entity Recognition,Entity Mention Detection and Relation Extraction without hand-engineeredfeatures or external NLP tools like syntactic parsers. The hierarchicaltraining supervision induces a set of shared semantic representations at lowerlayers of the model. We show that as we move from the bottom to the top layersof the model, the hidden states of the layers tend to represent more complexsemantic information.

 

Quick Read (beta)

loading the full paper ...