Abstract
Computer Vision is driven by the many datasets which can be used for trainingor evaluating novel methods. However, each dataset has different set of classlabels, visual definition of classes, images following a specific distribution,annotation protocols, etc. In this paper we explore the automatic discovery ofvisual-semantic relations between labels across datasets. We want to understandhow the instances of a certain class in a dataset relate to the instances ofanother class in another dataset. Are they in an identity, parent/child,overlap relation? Or is there no link between them at all? To find relationsbetween labels across datasets, we propose methods based on language, onvision, and on a combination of both. Our methods can effectively discoverlabel relations across datasets and the type of the relations. We use theseresults for a deeper inspection on why instances relate, find missing aspectsof a class, and use our relations to create finer-grained annotations. Weconclude that label relations cannot be established by looking at the names ofclasses alone, as they depend strongly on how each of the datasets wasconstructed.