Abstract
We address the problem of jointly learning vision and language to understandthe object in a fine-grained manner. The key idea of our approach is the use ofobject descriptions to provide the detailed understanding of an object. Basedon this idea, we propose two new architectures to solve two related problems:object captioning and natural language-based object retrieval. The goal of theobject captioning task is to simultaneously detect the object and generate itsassociated description, while in the object retrieval task, the goal is tolocalize an object given an input query. We demonstrate that both problems canbe solved effectively using hybrid end-to-end CNN-LSTM networks. Theexperimental results on our new challenging dataset show that our methodsoutperform recent methods by a fair margin, while providing a detailedunderstanding of the object and having fast inference time. The source codewill be made available.