Connecting Vision and Language with Localized Narratives

  • 2020-03-17 15:19:46
  • Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, Vittorio Ferrari
  • 0


This paper proposes Localized Narratives, a new form of multimodal imageannotations connecting vision and language. We ask annotators to describe animage with their voice while simultaneously hovering their mouse over theregion they are describing. Since the voice and the mouse pointer aresynchronized, we can localize every single word in the description. This densevisual grounding takes the form of a mouse trace segment per word and is uniqueto our data. We annotate 628k images with Localized Narratives: the whole COCOdataset and 504k images of the Open Images dataset, which we make publiclyavailable. We provide an extensive analysis of these annotations showing theyare diverse, accurate, and efficient to produce. We also demonstrate theirutility on the application of controlled image captioning.


Quick Read (beta)

1 Dataset Collection, Quality, and Statistics

1.1 Dataset collection


Image Sources and Scale. We annotated images from \coco \cococite and \oid \oidcite. In order to facilitate future comparison to previous work, we re-annotated the full set of \numimagescoco images of \coco (train and validation). For \oid, we annotated \numimagesoid images, selected from the train split. To enable cross-modal applications, we selected images for which object segmentations [benenson19cvpr], bounding boxes or visual relationships [kuznetsova18arxiv] are already available.

Overall, we annotated \numimages images (Tab. LABEL:tab:datasets). For analysis purposes, we annotated \att\num5000 \coco images with replication 5 (5 different annotators annotated each image). Beyond this, we prioritized having a larger set covered, so the rest of images were annotated with replication 1. All analysis in the remainder of this section are done on the \coco dataset.


Annotation Cost. Annotating one image with Localized Narratives takes \annotationtime seconds on average. \newWe consider this a relatively low cost given the amount of information harvested, and it allows data collection at scale (\att\numcaptions annotations so far). Manual transcription takes up the majority of the time (\transcriptiontime sec., \transcriptiontimepercent%), while the narration step only takes \narrationtime seconds (\narrationtimepercent%). In the future, when ASR systems improve further, manual transcription could be skipped and Localized Narratives could become even faster thanks to our core idea of using speech.


To put our timings into perspective, we can roughly compare to \flickr Entities [plummer17ijcv], which is the only work we are aware of that reports annotation times. They first manually identified which words constitute entities, which took 235 seconds per image. In a second stage, annotators drew bounding boxes for these selected entities, taking 408 seconds (8.7 entities per image on average). This yields a total of 643 seconds per image, without counting the time to write the actual captions (not reported). This is 4.3× slower than the total annotation cost of our method, which includes the grounding of \att\num12.1 nouns per image and the writing of the caption. The \vg [krishna17ijcv] dataset was also annotated by a complex multi-stage pipeline, also involving drawing a bounding box for each phrase describing a region in the image.

1.2 Dataset Quality


To ensure high quality, Localized Narratives was made by 126 professional annotators working full time on this project. Annotator managers did frequent manual inspections to keep quality consistently high. In addition, we used an automatic quality control mechanism to ensure that the spoken and written transcriptions match (Sec. LABEL:sec:generation – Automatic quality control). In practice, we placed a high quality bar, which resulted in discarding \att23.5% of all annotations (all dataset statistics reported in this paper are after this step). Below we analyse the quality of the annotations that remained after this automatic discarding step.


Semantic and Transcription Accuracy. In this section we quantify (i) how well the noun phrases and verbs in the caption correctly represent the objects in the image (Semantic accuracy) and (ii) how well the manually transcribed caption matches the voice recording (Transcription accuracy). We manually check every word in \att\num100 randomly selected Localized Narratives annotations and log each of these two types of errors. This checking was performed carefully by experts (\iethe authors of this paper), not by the annotators themselves (and hence are an independent source).

In terms of semantic accuracy, we check every noun and verb in a caption and assess whether that object or action is indeed present in the corresponding image. We allow generality up to a base class name (\eg we count either “dog” or “Chihuahua” as correct for a Chihuahua in the image) and we strictly enforce correctness (\eg we count “skating” as incorrect when the correct term is “snowboarding” or “bottle” in the case of a “jar”). Under these criteria, semantic accuracy is very high: \att\num98.0% of the nouns and verbs are accurate.

In terms of transcription accuracy, we listen to the voice recordings and compare them to the manual transcriptions. We count every instance of (i) a missing word in the transcription, (ii) an extra word in the transcription, and (iii) a word with typographical errors. We normalize the number of words with errors by the total number of words in the 100 captions. This results in \att\num3.8% for type (i), \att\num1.5% for (ii), and \att\num1.9% for (iii), showing transcription accuracy is high.

Localization Accuracy. To analyze how well the mouse traces match the location of actual objects in the image, we extract all instances of any of the 80 \coco object classes in our captions (exact string matching). We recover \att\num146723 instances. We then associate each mouse trace segment to the closest ground-truth box of its corresponding class. Figure 2 displays the 2D histogram of the positions of all trace segment points with respect to the closest box (2), normalized by box size. We observe that most of the trace points are within the correct bounding box.

We attribute the trace points that fall outside the box to two different effects. First, circling around the objects is commonly used by annotators (Fig. LABEL:fig:loc_narr_intro_example and Fig. 3). This causes the mouse traces to be close to the box, but not inside it. Second, some annotators sometimes start moving the mouse before they describe the object, or vice versa. We see both cases as a research opportunity to better understand the connection between vision and language.

1.3 Dataset Statistics


Richness. The mean length of the captions we produced is \captionlength words (Tab. LABEL:tab:datasets), substantially longer than previous captioning datasets (\eg\att\num4× longer than \newthe individual \coco captions). We also compare in terms of the average number of nouns, pronouns, adjectives, verbs, and adpositions (prepositions and postpositions, Tab. 1). We determined this using the spaCy [spacy] part-of-speech tagger. Localized Narratives has a higher occurrence \newper caption for each of these categories compared to previous datasets, which indicates that our annotations provide richer use of natural language in connection to the images they describe.

\topruleDataset Words Nouns Pronouns Adjectives Adpositions Verbs
\midrule\coco Captions [chen15arxiv] \num10.5 \num3.6 \num0.2 \num0.8 \num1.7 \num0.9
\flickr [young14tacl] \num12.4 \num3.9 \num0.3 \num1.1 \num1.8 \num1.4
\new\vg [krishna17ijcv] \num5.1 \num1.9 \num0.0 \num0.6 \num0.7 \num0.3
Loc. Narratives (Ours) \captionlength \att\num12.1 \att\num3.8 \att\num2.0 \att\num5.3 \att\num4.2

Table 1: Richness of individual captions of Localized Narratives versus previous works. \newPlease note that since \coco Captions and Flickr30K have replication 5 (and \vg also has a high replication), counts per image would be higher in these datasets. However, many of them would be duplicates. We want to highlight the richness of captions as units and thus we show word counts averaged over individual captions.

{tikzpicture}{axis} [width=1.4height=1.4view=090, point meta min=0, point meta max=10100, axis equal=true, no markers, colorbar, colorbar style=ytick=, xtick=-1,0,1, xticklabels=-1,0,1, ytick=-1,0,1, yticklabels=-1,0,1] \addplot3 [surf, mesh/rows=49, shader=interp] file figures/data/2D_hist_trace_location.txt; \addplot+[line width=1.5pt,red,dashed] coordinates (-1,-1) (-1,1) (1,1) (1,-1)(-1,-1);    {tikzpicture}{axis} [width=11cm, height=55mm, ylabel=Percentage of captions (%),ylabel shift=-3pt,xlabel=Number of nouns per caption,xlabel shift=-3pt, legend pos = north east, minor grid style=white!85!black, major grid style=white!60!black, xmin=0, xmax=35, ymin=0, ymax=55, ytick=0,5,…,100, xtick=0,5,…,45, minor ytick=0,1,…,100, minor xtick=0,1,…,100, clip marker paths=true, enlargelimits=false,grid=both,grid style=densely dotted] \addplot+[blue,solid,mark=*, mark size=0.8, line width=1.5pt] coordinates (0,3.187053918450993) (1,24.962900148399406) (2,52.44859020563918) (3,16.405201045862484) (4,2.6535227192424564) (5,0.3427319624054837) ; \addlegendentry\vg \addplot +[black,solid,mark=*, mark size=0.8, line width=1.5pt] coordinates (0,0.030006001200240048) (1,0.9701940388077616) (2,13.432686537307461) (3,37.82756551310262) (4,31.00620124024805) (5,11.88237647529506) (6,3.4906981396279257) (7,0.9401880376075216) (8,0.24004800960192038) (9,0.14002800560112022) (10,0.040008001600320066) ; \addlegendentry\coco \addplot +[olive,solid,mark=*, mark size=0.8, line width=1.5pt, mark options=fill=olive] coordinates (0,0.048001920076803074) (1,1.5260610424416976) (2,15.14660586423457) (3,28.547141885675426) (4,25.245009800392015) (5,15.16060642425697) (6,7.536301452058082) (7,3.5401416056642265) (8,1.6600664026561063) (9,0.8200328013120526) (10,0.3820152806112245) (11,0.18400736029441178) (12,0.10400416016640666) (13,0.04000160006400256) (14,0.02800112004480179) (15,0.016000640025601025) (16,0.006000240009600384) (17,0.006000240009600384) (18,0.004000160006400256) ; \addlegendentry\flickr \addplot +[red,solid,mark=*, mark size=0.8, line width=2.5pt, mark options=fill=red] coordinates (0,0.0) (1,0.03769114796467801) (2,0.27460693517122553) (3,1.346112427309929) (4,2.902218393280207) (5,4.522937755761361) (6,5.9982769760930434) (7,7.215162610381219) (8,7.344389403402972) (9,8.087443463278053) (10,7.721300883049752) (11,7.624380788283437) (12,7.145164764161102) (13,6.278268360973509) (14,5.653672194701701) (15,4.528322205470601) (16,4.0383372819297865) (17,3.1283652810682745) (18,3.096058582812837) (19,2.5468447124703855) (20,1.9114796467800992) (21,1.4268791729485246) (22,1.3138057290544907) (23,1.1845789360327375) (24,0.9422786991169503) (25,0.7592074090027999) (26,0.5438294206332113) (27,0.4522937755761361) (28,0.32845143226362267) (29,0.35537368080982124) (30,0.33383588197286235) (31,0.22614688778806805) (32,0.18307129011415035) (33,0.09153564505707518) (34,0.04307559767391773) (35,0.059228946801636866) (36,0.0646133965108766) (37,0.0699978462201163) (38,0.0699978462201163) (39,0.0323066982554383) (40,0.02692224854619858) (41,0.0323066982554383) (42,0.010768899418479432) (43,0.01615334912771915) (44,0.010768899418479432) (45,0.021537798836958864) ; \addlegendentryLocalized Narratives (Ours)

Figure 1: Histogram of mouse trace segment locations with respect to the closest box of the relevant class (2).
Figure 2: Distribution of number of nouns per caption. \JordiAdded next sentence.As in Table 1, these counts are per individual caption, not adding up all nouns across replicated captions.

Diversity. To illustrate the diversity of our captions, we plot the distribution of the number of nouns per caption, and compare it to the distributions obtained over previous datasets (Fig. 2). We observe that the \newrange of number of nouns is significantly higher in Localized Narratives (up to \att\num45 nouns in some images). This poses an additional challenge for captioning methods: automatically adapting the length of the descriptions to each image, as a function of the richness of its content. Beyond nouns, Localized Narratives provide visual grounding for every word (verbs, prepositions, etc.). This is especially interesting for relationship words, \eg“woman holding ballon” (Fig. LABEL:fig:loc_narr_intro_example) or “with a hand under his chin” (Fig. LABEL:fig:sample_datasets(d)). This opens the door to a new venue of research: understanding how humans naturally ground visual relationships.

Diversity in Localized Narratives is present not only in the language modality, but also in the visual modality, such as the different ways to indicate the spatial location of objects in an image. In contrast to previous works, where the grounding is in the form of a bounding box, our instructions lets the annotator hover the mouse over the object in any way they feel natural. This leads to diverse styles of creating trace segments (Fig. 3): circling around an object (sometimes without even intersecting it), scribbling over it, underlining in case of text, etc. This diversity also presents another challenge: detect and adapt to different trace styles in order to make full use of them.

Ship Open land with some grass on it Main stairs

Figure 3: Examples of mouse trace segments and their corresponding word(s) in the caption with different pointing styles: circling, scribbling, and underlining.