Abstract
Despite the well-developed cut-edge representation learning for language,most language representation models usually focus on specific level oflinguistic unit, which cause great inconvenience when being confronted withhandling multiple layers of linguistic objects in a unified way. Thus this workintroduces and explores the universal representation learning, i.e., embeddingsof different levels of linguistic unit in a uniform vector space through atask-independent evaluation. We present our approach of constructing analogydatasets in terms of words, phrases and sentences and experiment with multiplerepresentation models to examine geometric properties of the learned vectorspace. Then we empirically verify that well pre-trained Transformer modelsincorporated with appropriate training settings may effectively yield universalrepresentation. Especially, our implementation of fine-tuning ALBERT on NLI andPPDB datasets achieves the highest accuracy on analogy tasks in differentlanguage levels. Further experiments on the insurance FAQ task showeffectiveness of universal representation models in real-world applications.