Integrating Knowledge Graph embedding and pretrained Language Models in Hypercomplex Spaces

  • 2022-08-04 17:18:16
  • Mojtaba Nayyeri, Zihao Wang, Mst. Mahfuja Akter, Mirza Mohtashim Alam, Md Rashad Al Hasan Rony, Jens Lehmann, Steffen Staab
  • 17

Abstract

Knowledge Graphs, such as Wikidata, comprise structural and textual knowledgein order to represent knowledge. For each of the two modalities dedicatedapproaches for graph embedding and language models learn patterns that allowfor predicting novel structural knowledge. Few approaches have integratedlearning and inference with both modalities and these existing ones could onlypartially exploit the interaction of structural and textual knowledge. In ourapproach, we build on existing strong representations of single modalities andwe use hypercomplex algebra to represent both, (i), single-modality embeddingas well as, (ii), the interaction between different modalities and theircomplementary means of knowledge representation. More specifically, we suggestDihedron and Quaternion representations of 4D hypercomplex numbers to integratefour modalities namely structural knowledge graph embedding, word-levelrepresentations (e.g.\ Word2vec, Fasttext), sentence-level representations(Sentence transformer), and document-level representations (sentencetransformer, Doc2vec). Our unified vector representation scores theplausibility of labelled edges via Hamilton and Dihedron products, thusmodeling pairwise interactions between different modalities. Extensiveexperimental evaluation on standard benchmark datasets shows the superiority ofour two new models using abundant textual information besides sparse structuralknowledge to enhance performance in link prediction tasks.

 

Quick Read (beta)

loading the full paper ...