Abstract
Current major approaches to visual recognition follow an end-to-endformulation that classifies an input image into one of the pre-determined setof semantic categories. Parametric softmax classifiers are a common choice forsuch a closed world with fixed categories, especially when big labeled data isavailable during training. However, this becomes problematic for open-setscenarios where new categories are encountered with very few examples forlearning a generalizable parametric classifier. We adopt a non-parametricapproach for visual recognition by optimizing feature embeddings instead ofparametric classifiers. We use a deep neural network to learn the visualfeature that preserves the neighborhood structure in the semantic space, basedon the Neighborhood Component Analysis (NCA) criterion. Limited by itscomputational bottlenecks, we devise a mechanism to use augmented memory toscale NCA for large datasets and very deep networks. Our experiments delivernot only remarkable performance on ImageNet classification for such a simplenon-parametric method, but most importantly a more generalizable featurerepresentation for sub-category discovery and few-shot recognition.