Multi-label Zero-shot Classification by Learning to Transfer from External Knowledge

  • 2020-07-30 17:26:46
  • He Huang, Yuanwei Chen, Wei Tang, Wenhao Zheng, Qing-Guo Chen, Yao hu, Philip Yu
  • 11


Multi-label zero-shot classification aims to predict multiple unseen classlabels for an input image. It is more challenging than its single-labelcounterpart. On one hand, the unconstrained number of labels assigned to eachimage makes the model more easily overfit to those seen classes. On the otherhand, there is a large semantic gap between seen and unseen classes in theexisting multi-label classification datasets. To address these difficultissues, this paper introduces a novel multi-label zero-shot classificationframework by learning to transfer from external knowledge. We observe thatImageNet is commonly used to pretrain the feature extractor and has a large andfine-grained label space. This motivates us to exploit it as external knowledgeto bridge the seen and unseen classes and promote generalization. Specifically,we construct a knowledge graph including not only classes from the targetdataset but also those from ImageNet. Since ImageNet labels are not availablein the target dataset, we propose a novel PosVAE module to infer their initialstates in the extended knowledge graph. Then we design a relational graphconvolutional network (RGCN) to propagate information among classes and achieveknowledge transfer. Experimental results on two benchmark datasets demonstratethe effectiveness of the proposed approach.


Quick Read (beta)

loading the full paper ...