REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environments

  • 2020-01-06 01:38:43
  • Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang, Chunhua Shen, Anton van den Hengel
  • 0


One of the long-term challenges of robotics is to enable robots to interactwith humans in the visual world via natural language, as humans are visualanimals that communicate through language. Overcoming this challenge requiresthe ability to perform a wide variety of complex tasks in response tomultifarious instructions from humans. In the hope that it might drive progresstowards more flexible and powerful human interactions with robots, we propose adataset of varied and complex robot tasks, described in natural language, interms of objects visible in a large set of real images. Given an instruction,success requires navigating through a previously-unseen environment to identifyan object. This represents a practical challenge, but one that closely reflectsone of the core visual problems in robotics. Several state-of-the-artvision-and-language navigation, and referring-expression models are tested toverify the difficulty of this new task, but none of them show promising resultsbecause there are many fundamental differences between our task and previousones. A novel Interactive Navigator-Pointer model is also proposed thatprovides a strong baseline on the task. The proposed model especially achievesthe best performance on the unseen test split, but still leaves substantialroom for improvement compared to the human performance.


Quick Read (beta)

REVERIE: Remote Embodied Visual Referring Expression in
Real Indoor Environments

Yuankai Qi Australia Centre for Robotic Vision, The University of Adelaide Qi Wu corresponding author; [email protected] Australia Centre for Robotic Vision, The University of Adelaide Peter Anderson Georgia Institute of Technology Xin Wang University of California, Santa Barbara William Yang Wang University of California, Santa Barbara Chunhua Shen Australia Centre for Robotic Vision, The University of Adelaide Anton van den Hengel Australia Centre for Robotic Vision, The University of Adelaide

One of the long-term challenges of robotics is to enable robots to interact with humans in the visual world via natural language, as humans are visual animals that communicate through language. Overcoming this challenge requires the ability to perform a wide variety of complex tasks in response to multifarious instructions from humans. In the hope that it might drive progress towards more flexible and powerful human interactions with robots, we propose a dataset of varied and complex robot tasks, described in natural language, in terms of objects visible in a large set of real images. Given an instruction, success requires navigating through a previously-unseen environment to identify an object. This represents a practical challenge, but one that closely reflects one of the core visual problems in robotics. Several state-of-the-art vision-and-language navigation, and referring-expression models are tested to verify the difficulty of this new task, but none of them show promising results because there are many fundamental differences between our task and previous ones. A novel Interactive Navigator-Pointer model is also proposed that provides a strong baseline on the task. The proposed model especially achieves the best performance on the unseen test split, but still leaves substantial room for improvement compared to the human performance.

1 Introduction

Figure 1: REVERIE task: an agent is given a natural language instruction referring to a remote object (here in the red bounding box) in a photo-realistic 3D environment. The agent must navigate to an appropriate location and identify the object from multiple distracting candidates. The blue discs indicate nearby navigable viewpoints provided by the simulator.

You can ask a 10-year-old child to bring you a cushion, and there is a good chance that they will succeed (even in an unfamiliar environment), while the probability that a robot will achieve the same task is significantly lower. Children have a wealth of knowledge learned from similar environments that they can easily apply to such tasks in an unfamiliar environment, including the facts that cushions generally inhabit couches, that couches inhabit lounge rooms, and that lounge rooms are often connected to the rest of a building through hallways. Children are also able to interpret natural language instructions and associate them with the visual world. However, the fact that robots currently lack these capabilities dramatically limits their domain of application.

1. Fold the towel in the bathroom with the fishing theme.
2. Enter the bedroom with the letter E over the bed and turn the light switch off.
3. Go to the blue family room and bring the framed picture of a person on a horse at the top left corner above the TV.
4. Push in the bar chair, in the kitchen, by the oven.
5. Windex the mirror above the sink, in the bedroom with the large, stone fireplace.
6. Could you please dust the light above the toilet in the bathroom that is near the entry way?
7. At the top of the stairs, the first set of potted flowers in front of the stairs need to be dusted off.
8. To the right at the end of the hall, where the large blue table foot stool is, there is a mirror that needs to be wiped.
9. Go to the hallway area where there are three pictures side by side and get me the one on the right.
10. There is a bottle in the office alcove next to the piano. It is on the shelf above the sink on the extreme right. Please bring it here.
Table 1: Indicative instruction examples from the REVERIE dataset illustrating various interesting linguistic phenomena such as dangling modifiers (e.g. 1), spatial relations (e.g. 3), imperatives (e.g. 9), co-references (e.g. 10), etc. Note that the agent in our task is required to identify the referent object, but is not required to complete any manipulation tasks (such as folding the towel).

Therefore, to equip robots with such abilities and to advance real-world vision-and-language research, we introduce a new problem, which we refer to as Remote Embodied Visual referring Expression in Real Indoor EnvironmentsREVERIE. An example of the REVERIE task is illustrated in Fig. 1. A robot spawns at a starting location and is given a natural language instruction that refers to a remote target object at another location within the same building. To carry out the task, the agent is required to navigate closer to the object and return a bounding box encompassing the target object specified by the instruction. It demands the robot to infer the probable location of the object using knowledge of the environments, and explicitly identify the object according to the language instruction.

In distinction to other embodied tasks such as Vision-and-Language Navigation (VLN) [1] and Embodied Question Answering (EQA) [7], REVEVIE evaluates the success based on explicit object grounding rather than the point navigation in VLN or the question answering in EQA. This more clearly reflects the necessity of robots’ capability of natural language understanding, visual navigation, and object grounding. More importantly, the concise instructions in REVERIE represent more practical tasks that humans would ask a robot to perform (see Tab. 1). Those high-level instructions are fundamentally different from the fine-grained visuomotor instructions in VLN, and would empower high-level reasoning and real-world applications. Moreover, compared to the task of Referring Expression (RefExp) [14, 22] that selects the desired object from a single image, REVERIE is far more challenging in the sense that the target object is not visible in the initial view and needs to be discovered by actively navigating in the environment. Hence, in REVERIE, there are at least an order of magnitude more object candidates to choose from than RefExp.

We build the REVERIE dataset upon the Matterport3D Simulator [1, 4], which provides panoramas of all the navigable locations and the connectivity graph in a building. To provide object-level information of the environments, we have extended the simulator to incorporate object annotations, including labels and bounding boxes from Chang et al[4]. The extended simulator can project bounding boxes onto images of different viewpoints and angles, thus able to accommodate evaluation on every possible location. The REVERIE dataset comprises 10,567 panoramas within 90 buildings containing 4,140 target objects, and 21,702 crowd-sourced instructions with an average length of 18 words. Tab. 1 demonstrates sample instructions from the dataset, which illustrate various linguistic phenomena, including spatial relations, multiple long and dangling modifiers, and coreferences, etc.

We investigate the difficulty of the REVERIE task by directly combining state-of-the-art (SoTA) navigation methods and referring expression methods, and none of them shows promising results. We then propose an Interactive Navigator-Pointer model serving as a strong baseline for the REVERIE task. We also provide the human performance of the REVERIE task on the test set to quantify the machine-human gap.

In summary, our main contributions are:

  1. 1.

    A new embodied vision-and-language problem, Remote Embodied Visual referring Expressions in Real 3D Indoor Environments (REVERIE), where given a natural language instruction that represents a practical task to perform, an agent must navigate and identify a remote object in real indoor environments.

  2. 2.

    The first benchmark dataset for the REVERIE task, which contains large-scale human-annotated instructions and extends the Matterport3D Simulator [1] with additional object annotations.

  3. 3.

    A novel interactive navigator-pointer model that provides strong baselines for the REVERIE dataset under several evaluation metrics.

Dataset Language Context Visual Context Goal
Human Main Content Unamb Guidance Level BBox Real-world Temporal
EQA [7], IQA [10] QA-pair Dynamic QA
MARCO [21], DRIF [3] Nav-Instruction Detailed Dynamic Navigation
R2R [1] Nav-Instruction Detailed Dynamic Navigation
TouchDown [5] Nav-Instruction Detailed Dynamic Navigation
VLNA [24], HANNA[23] Nav-Dialog High Dynamic Find Object
TtW [8] Nav-Dialog High Dynamic Navigation
CVDN [25] Nav-Dialog High Dynamic Find Room
ReferCOCO [30] RefExp Static Localise Object
REVERIE Remote RefExp High Dynamic Localise Remote Object
Table 2: Compared to existing datasets involving embodied vision and language tasks. Symbol instruction: ‘QA’: ‘Question-Answer’, ‘Unamb’: ‘Unambiguous’, ‘BBox’: ‘Bounding Box’, ‘Dynamic’/‘Static’: visual context temporally changed or not.

2 Related Work

Referring Expression Comprehension.

The referring expression comprehension task requires an agent to localise an object in an image given a natural language expression. Recent work casts this task as looking for the object that can generate its paired expressions [13, 18, 30] or jointly embedding the image and expression for matching estimation [6, 12, 16, 29]. Yu et al[30] propose to compute the appearance difference of the same category objects to enhance the visual features for expression generation. Instead of treating each expression as a unit, [29] learns to decompose an expression into appearance, location, and object relationship three components.

Different from referring expression, REVERIE introduces three new challenges: i) The refereed object is not visible in the initial scene and only can be accessed after navigating to a closed location. ii) In contrast to previous referring expression tasks that select the target object from a single image, object candidates in REVERIE come from panoramas of all the possible viewpoints. iii) The objects in referring expression are normally captured from the front view, while in our setting, the visual appearances of objects may vary largely due to different observation angles and viewpoints.

Vision-and-Language Navigation.

Vision-and-language navigation (VLN) is the task where an agent is to navigate to a goal location in a 3D simulator given detailed natural language instructions such as “Turn right and go through the kitchen. Walk past the couches on the right and into the hallway on the left. Go straight until you get to a room that is to the left of the pictures of children on the wall. Turn left and go into the bathroom. Wait near the sink.” [1]. A range of VLN methods [9, 15, 20, 19, 27, 28] have been proposed to address this VLN task.

Although the proposed REVERIE task also requires an agent to navigate to a goal location, it differs from existing VLN tasks in two important aspects: i) The challenge is much more closely related to the overarching objective of enabling natural language robot tasking because the goal is to localise a target object specified in an instruction, not just a location. This removes the artificial constraint that the instruction is restricted to solely to navigation, and reflects the reality of the fact that most objects can be seen from multiple viewpoints. ii) Our collected navigation instructions are semantic-level commands which better reflect the way humans communicate. They are thus closer to ‘the cold tap in the first bedroom on level two’ rather than step by step navigation instructions such as ‘go to the top of the stairs then turn left and walk along the hallway and stop at the first bedroom on your right’.

The most closely related challenge to that proposed here is that addressed in [23, 24, 25] whereby an agent must identify an object by requesting and interpreting natural language assistance. The instructions are of the form ‘Find a mug’, and the assumption is that there is an oracle following the agent around the environment willing to provide natural language assistance. The question is then whether the agent can effectively exploit the assistance provided by the omniscient oracle. REVERIE, in contrast, evaluates whether the agent can carry out a natural-language instruction alone. Another closely related work is TOUCHDOWN [5], that requires an agent to find a location in an urban outdoor environment on the basis of detailed navigation instructions.

Embodied Question Answering.

Embodied question answering (EQA) [7] requires an agent to answer a question about an object or a room in a synthetic environment. Gordon et al[10] introduce an interactive version of the EQA task, where the agent may need to interact with the environment/objects to correctly answer questions. Our REVERIE task differs from previous works that only output a simple answer or a series of actions, as we ask the agent to output a bounding box around a target object. This is a more challenging but realistic setting because if we want a robot to carry out a task that relates to an object, we need its precise location. Tab. 2 displays the difference between our task and other related embodied vision-language tasks.

3 The REVERIE Dataset

We now describe the REVERIE task and dataset, including the task definition, evaluation metrics, simulator, data collection policy, and analysis of the collected instructions.

3.1 The REVERIE Task

As shown in Fig. 1, our REVERIE task requires an intelligent agent to correctly localise a remote target object (can not be observed at starting location) specified by a concise high-level natural language instruction (see samples in Tab. 1). It is worth noting that our instructions are much closer to practical scenarios in daily life than the detailed instructions in VLN [1], because the latter ones are so complex and long that are unrealistic for human to command robots. Since the target object is in a different room from the starting one, the agent needs first to navigate to the goal location.

Formally, at the beginning of each episode, the agent is given as input a high-level natural language instruction 𝒳=w1,w2,,wL, where L is the length of the instruction and wi is a single word token. Following the common practice in VLN, the agent has access to surrounding panoramic images 𝒱0={v0,k,k1,,36} and navigable viewpoints at the current location, where v0,k is determined by the agent’s states comprising a tuple of 3D position, heading and elevation s0,k=p0,ϕ0,k,θ0,k (3 elevation and 12 heading angles are used). Then the agent needs to make a sequence of actions a0,,aT to reach the goal location, where each action is choosing one of the navigable viewpoints or choosing the current viewpoint which means to stop. The action can also be a ‘detecting’ action that outputs the target object bounding-box refereed by the instruction. It is worth noting that the agent can attempt to localise the target at any step, which is totally up to algorithm design. But we only allow the agent output once in each episode, which means the agent only can guess the answer once in a single run. If the agent ‘thinks’ it has localised the target object and decides to output it, it is required to output a bounding box or choose from several candidates provided by the simulator. A bounding box is denoted as bx,by,bw,bh, where bx and by are the coordinate of the left-top point, bw and bh denote the width and height of the bounding box, respectively. The episode ends after the agent outputs the target bounding box.

3.2 Evaluation Metrics

The performance of a model is mainly measured by REVERIE success rate, which is the number of successful tasks over the total number of tasks. A task is considered successful if it selects the correct bounding box of the target object from a set of candidates (or the IoU between the predicted bounding box and the ground-truth bounding box 0.5, when candidate objects bounding boxes are not given). Because the target object can be observed at different viewpoints or camera views, we treat it as a success as long as the agent can identify the target within 3 meters, regardless of from different viewpoints or views. We also measure the navigation performance with four kinds of metrics, including success rate, oracle success rate, success rate weighted by path length (SPL), and path length (in meters) [1]. Please note that in our task, a navigation is considered successful only when the agent stops at a location within 3 meters from the target object. More details of evaluation metrics can be found in supplementary materials.

3.3 The REVERIE Simulator

Our simulator is based on the Matterport3D Simulator [1], a large-scale interactive environment constructed from the Matterport3D dataset [4]. In the simulator, an embodied agent is able to virtually ‘move’ throughout each building by iteratively selecting adjacent nodes from the graph of panoramic viewpoints and adjusting the camera pose at each viewpoint. At each viewpoint, it returns a rendered colour image that captures the current view, as shown in Fig. 1.

Figure 2: Object bounding boxes (BBox) in our simulator. The BBox size and aspect ratio of the same object may change after the agent moves to another viewpoint or changes its camera view.

Adding Object-level Annotations.

Object bounding boxes are needed in our proposed task, which are either provided as object hypotheses or used to assess the agent’s ability to localise the object that is referred to by a natural expression. The main challenge of adding the object bounding boxes into the simulator is that we need to handle the changes in visibility and coordinate of 2D bounding boxes as the camera moves or rotates.

To address these issues, we calculate the overlap between bounding boxes and object depth in each view. If a bounding box is fully covered by another one and it has a larger depth, we treat it as an occluded case. Specifically, for each building the Matterport3D dataset provides all the objects appearing in it with centre point position 𝐜=cx,cy,cz, three axis directions 𝐝i=dix,diy,diz,i{1,2,3}, and three radii ri, one for each axis direction. To correctly render objects in the web simulator, we first calculate the eight vertexes using 𝐜,𝐝i and ri. Then these vertexes are projected into the camera space by the camera pose provided by Matterport3D dataset. Both C++ and web simulators will be released with the code. Fig. 2 presents an example of projected bounding boxes. Note that the target object may be observed at multiple viewpoints in one room, but we expect a robot can reach the target in a short distance. Thus, we only preserve objects within three meters to a viewpoint. For each object, a class label and a bounding box are associated. But it is worth noting that we adjust the size and aspect-ratio accordingly as the viewpoint and camera angle change. In total, we obtain 20k object annotations.

3.4 Data Collection

Our goal is to collect high-level human daily commands that may be assigned to a home robot in future, such as ‘Open the left window in the kitchen’ or ‘Go to my bedroom and bring me a pillow’. We develop an interactive 3D WebGL simulator to collect such instructions on Amazon Mechanical Turk (AMT). The web simulator first shows a path animation and then randomly highlights one object at the goal location for workers to provide instructions to find or operate with. There is no style limitation of the command as long as it can lead the robot to the target object. Assistant room and object information are provided to workers facilitating them to provide unambiguous instructions if there are similar rooms or objects. The workers can look around at the goal location to learn the environment. For each target object, we collect three referring expressions. The full collection interface (see in supplementary) is the result of several rounds of experimentation. Over 1,000 workers took part in the data collection, totally contributing around 2,648 hours of annotation time. Examples of the collected data can be found in Tab. 1, and more are in supplementary.

Figure 3: The distribution of the number of words (left) and objects (right) in each instruction.

3.5 Dataset Analysis

The REVERIE dataset contains totally 21,702 instructions and a vocabulary of over 1,600 words. The average length of collected instructions is 18 words involving both navigation and referring expression information. Considering the detailed navigation instructions provided in the R2R [1] with an average length of 29 words and the previous largest dataset RefCOCOg [30] contains an average of 8 words, our instructions command is much more concise and natural, and thus more challenging.

Fig. 3 (left) displays the length distribution of the collected instructions, which shows that most instructions have 1022 words while the shortest annotation could be only 3 words, such as ‘Flush the toilet’. Fig. 4 (left) presents the relative amount of words used in instructions in the form of word cloud. It shows that people prefer ‘go’ for navigation, and most instructions involve ‘bathroom’. We also compute the number of mentioned objects in instructions and its distribution is presented in Fig. 3 (right). It shows that 56% instructions mention 3 or more objects, 28% instructions mention 2 objects, and the remaining 15% instructions mention 1 object. On average, there are 7 objects with 50 bounding boxes at each target viewpoint. There are 4,140 target objects in the dataset, falling into 489 categories, which are 6 times more than the 80 categories in ReferCOCO [30], a most popular referring expression dataset at present. Fig. 4 (right) shows the relative amount of target objects in different categories.

Figure 4: Word cloud of instructions (left) and target objects (right) in the REVERIE dataset. The bigger the font, the more percentage it occupies.

Data Splits

We follow the same train/val/test split strategy as the R2R [1] datasets. The training set consists of 59 scenes and 10,466 instructions over 2,353 objects. The validation set including seen and unseen splits totally contains 63 scenes, 953 objects, and 4,944 instructions, of which 10 scenes and 3,573 instructions over 525 objects are reserved for val unseen split. For the test set, we collect 6,292 instructions involving 834 objects randomly scattered in 16 scenes. All the test data are unseen during training and validation procedures. The ground truth for the test set will not be released, and we will host an evaluation server where agent trajectories and detected bounding boxes can be uploaded for scoring.

Figure 5: Our Interactive Navigator-Pointer Model

4 The Interactive Navigator-Pointer Model

As our REVERIE task requires an agent to navigate to the goal location and point out the target object, a naive solution is to employ state-of-the-art navigation (as a navigator) and referring expression comprehension (as a pointer) methods jointly. However, it is of great importance how the navigator and pointer should work together. Ideally, we want the navigator and pointer to benefit each other. Here, we propose a simple yet effective interactive manner that achieves the best performance as a strong baseline. Specifically, our module includes three parts: a navigator module that decides the action to take for the next step, a pointer module that attempts to localise the target object according to the language guidance, and an interaction module that responses for sending the referring expression comprehension information obtained from the pointer to the navigator to guild it to make more accurate action prediction.

4.1 The Navigator Module

The backbone of our navigator module is a ‘short’ version of FAST [15], which uses a sequence-to-sequence LSTM architecture with an attention mechanism and a backtracking mechanism to increase the action accuracy. Specifically, let 𝑿L×512 denote instruction features obtained from 𝒳 by an LSTM, and 𝑽=[𝒗t,1;;𝒗t,K]K×4736 denote updated visual features obtained by our interactive module (Sec. 4.3) for panoramic images 𝒱t at step t. FAST-short learns the local logit lt signal, which is calculated by a visual and textual co-grounding model adopted from [19]. First, grounded text 𝒙^t=𝜶t𝑿 and grounded visual 𝒗^t=𝜷t𝑽 are learned by

𝜶t =softmax(PE(𝑿)(𝑾x𝒉t-1)) (1)
𝜷t =softmax(g(𝑽)(𝑾v𝒉t-1)) (2)

where 𝜶tL×1 is textual attention weight, 𝜷tK×1 is visual attention weight, 𝑾x and 𝑾v are learnable parameters, PE() is the positional encoding [26] that captures the relative position between each word within an instruction, g() is a one-layer Multi-Layer Perceptron (MLP), 𝒉t-1512×1 is previous encoder context. The new context is updated by an LSTM taking as input the newly grounded text and visual features as well as previous selected action

(𝒉t,𝒄t)=LSTM([𝒙^t,𝒗^t,𝒂t-1],(𝒉t-1,𝒄t-1)). (3)

Then the logit lt can be computed via an inner-product between each candidate’s encoded context and instruction by

lt,k=(𝑾a[𝒉t,𝒙^t])g(𝒗t,k) (4)

where 𝑾a is a learnable parameter matrix.

Based on logit lt, FAST-short maintains one candidate queue and one ending queue. All navigable viewpoints (including the current viewpoint) at the current location are pushed into the candidate queue, but only the viewpoint with the largest accumulated logit τ=0tlτ is popped out as the selected next step. Each passed viewpoint is pushed into the ending queue. One episode ends if the current viewpoint is selected or the candidate queue is empty or the maximum step is reached. Finally, the viewpoint with the largest accumulated logits is chosen as the actual stop location.

4.2 The Pointer Module

We use MAttNet [29] as our pointer because of its good generalisation ability. It decomposes an expression into three modular components related to subject appearance, location, and relationship to other objects via the attention mechanism qm=j=1Lam,jej, where m{subj,loc,rel}, ej is the embedding of each word in the expression/instruction 𝒳. am,j is the attention on each word for each module.

Then three kinds of matching scores S(oi|qm) are computed for each object oi conditioned on each modular phrase embedding qm. Specifically, S(oi|qsubj)=F(v~isubj,qsubj), S(oi|qloc)=F(l~iloc,qloc), and S(oi|qrel)=maxjiF(v~ijrel,qrel), where F() is a two-layer MLP, v~isubj is a ‘in-box’ attended feature for each object using a 14×14 grid. l~iloc is the location representation of object oi obtained by a fully-connected layer taking as input the relative position offset and area ratio to its up to five surrounding objects of the same category. v~ijrel is the visual representation of the surrounding object oj regardless of categories.

The final matching score of object oi and the instruction 𝒳 is a weighted sum:

S=S(oi|qm)wm (5)

where wm=softmax(WmL[h0,hL]+bm).

4.3 The Interaction Module

Intuitively, we want the Navigator and Pointer to interact with each other so that both navigation and referring expression accuracy can be improved. For example, the navigator can use the visual grounding information to decide when and where to stop, and the pointer accuracy can be improved if the navigator can reach the correct target location. To this end, we propose an interaction module that can plug the pointer’s output into the navigator. Specifically, we first perform referring expression comprehension using the above pointer module to select the top-3 matching objects in each candidate view. Then we use a trainable bi-direction LSTM to encode the category labels of these selected objects 𝒳O={Labelitop3}

𝒙t,ko=bi_LSTM(𝒳O) (6)

as the textual representation for the k-th candidate viewpoint. In addition, the averaged output of ResNet FC7 layer of these object regions is used as the visual representation 𝒗t,ko. Finally, we update the candidate viewpoint feature by concatenation

𝒗t,k=[𝒗t,k,𝒙t,ko,𝒗t,ko] (7)

which is send to the navigator (see Equ. 2 and 4). The pointer in such an interaction serves as hard attention for each candidate viewpoint, which highlights the most target-related objects for the navigator to take into account.

4.4 Loss Functions

Our final loss consists of two parts, the navigation loss Lnav and referring expression loss Lexp. The Lnav is a cross-entropy loss for action selection and a mean squared error loss for progress monitor:

Lnav=-λ1t=1Tytalog(lt,k)-(1-λ1)t=1T(ytpm-ptpm)2 (8)

where yta is the ground truth action at step t, λ1=0.5 is the weight balancing the two loss, ytpm[0,1] is the normalised distance in units of length from the current viewpoint to the goal, ptpm is the predicted progress.

The referring expression loss Lexp is a ranking loss:

Lexp=i[λ2max(0,δ+S(oi|rj)-S(oi|ri))+λ3max(0,δ+S(ok|ri)-S(oi|ri))] (9)

where λ2=1.0, λ3=1.0, (oi,ri) is a positive (object, expression) pair, (oi,rj) and (ok,ri) are negative (object, expression) pairs, δ is the distance margin between positive and negative pairs. All of the losses are summarised together:

L=Lnav+λ4Lexp (10)

to train our Interactive Navigator-Pointer model. We set λ4 to 1.0 by default.

Methods Val Seen Val UnSeen Test (Unseen)
Navigation Acc. REVERIE Navigation Acc. REVERIE Navigation Acc. REVERIE
Succ. OSucc. SPL Length Succ. Succ. OSucc. SPL Length Succ. Succ. OSucc. SPL Length Succ.
Random 2.74 8.92 1.91 11.99 1.97 1.76 11.93 1.01 10.76 0.96 2.30 8.88 1.44 10.34 1.18
Shortest 100 100 100 10.46 68.45 100 100 100 9.47 56.63 100 100 100 9.39 48.98
R2R-TF [1] 7.38 10.75 6.40 11.19 4.22 3.21 4.94 2.80 11.22 2.02 3.94 6.40 3.30 10.07 2.32
R2R-SF [1] 29.59 35.70 24.01 12.88 18.97 4.20 8.07 2.84 11.07 2.16 3.99 6.88 3.09 10.89 2.00
RCM [27] 23.33 29.44 21.82 10.70 16.23 9.29 14.23 6.97 11.98 4.89 7.84 11.68 6.67 10.60 3.67
SelfMonitor [19] 41.25 43.29 39.61 7.54 30.07 8.15 11.28 6.44 9.07 4.54 5.80 8.39 4.53 9.23 3.10
FAST-Short [15] 45.12 49.68 40.18 13.22 31.41 10.08 20.48 6.17 29.70 6.24 14.18 23.36 8.74 30.69 7.07
FAST-Lan-Only 8.36 23.61 3.67 49.43 5.97 9.37 29.76 3.65 45.03 5.00 8.15 28.45 2.88 46.19 4.34
Ours 50.53 55.17 45.50 16.35 31.97 14.40 28.20 7.19 45.28 7.84 19.88 30.63 11.61 39.05 11.28
Human 81.51 86.83 53.66 21.18 77.84
Table 3: REVERIE success rate achieved by combining state-of-the-art navigation methods with the RefExp method MAttNet [29].

5 Experiments

In this section, we first present the training details of the interactive navigator–pointer model. Then, we provide extensive evaluation and analysis.

5.1 Implementation Details

The simulator image resolution is set to 640×480 pixels with a vertical field of view of 60 degrees. For each instruction in the train split, images and object bounding boxes at the goal viewpoint (for the views where the target object is visible) are organised following the format as in MAttNet for pointer training. With the trained pointer, assistant object information is provided for the navigator as described in Section 4.3 to train the navigator. The code and dataset will be released.

5.2 REVERIE Experimental Results

We first evaluate several baseline models and state-of-the-art (SoTA) navigation models, combined with the MattNet, i.e., the pointer module. After the navigation models decide to stop, the pointer module is used to predict the target object. In addition, we also test human performance using an interactive web interface (see details in the supplementary).

Below is a brief introduction of the evaluated baseline and state-of-the-art models. There are four baseline models, which are:

  • Random exploits the characteristics of the dataset by randomly choosing a path with random steps (maximum 10) and then randomly choose an object as the predicted target.

  • Shortest always follows the shortest path to the goal.

  • R2R-TF and R2R-SF  [1] are the first batch of navigation baselines, which trains a basic LSTM [11] with attention mechanism [2]. The difference between R2R-TF and R2R-SF is that R2R-TF is trained with the ground truth action at each step (Teacher-Forcing, TF) while R2R-SF adopts an action sampled from the predicted probability over its action space (Student-Forcing, SF).

The evaluated four SoTA navigation models are:

  • SelfMonitor [19] uses a visual-textual co-grounding module to highlight the instruction for the next action and a progress monitor to reflect the progress.

  • RCM [27] employs reinforcement learning to encourage global matching between instructions and trajectories, and performs cross-model grounding.

  • FAST-Short [15] introduces backtracking into SelfMonitor.

  • FAST-Lan-Only employs above FAST-Short model but we only input the language instruction without any visual input. This model is used to check whether our task/dataset has a bias on language input.

Results. The detailed experimental results are presented in Tab. 3, of which the first four rows are results for baselines, the following four rows are for SoTA methods, and the last two rows are for our model and human performance.

According to the baseline section in Tab. 3, the Random model only achieves a REVERIE success around 1%, which indicates the REVERIE task has a huge solution space. The sequence-to-sequence baselines R2R-TF and R2R-SF [1] achieve good results on the Val-Seen split but decrease a lot on the unseen splits. Student-Forcing is generally better than Teacher-Forcing. The Shortest model achieves the perfect performance because the ground-truth path to the goal is directly given.

In the second part, the best REVERIE success rate is achieved by the combination of SoTA navigation (FAST) and referring expression (MAttNet) models. However, the REVERIE success rate is only 7.07% on the test split, falling far behind human performance 77.84%. The navigation-only accuracy of these SoTA navigation models indicates the challenge of our navigation task. Nearly 30% drops on the unseen splits are observed compared to the performance on previous R2R [1]. For example, the navigation SPL score of FAST-Short [15] on Val-UnSeen split drops from 43% on the R2R dataset to 6.17% on REVERIE.

To test whether our dataset has strong language bias, i.e., whether a language-only model can achieve good performance, we implement a FAST-Lan-Only model with only instructions as its input. We observe a big drop on both seen and unseen splits, which suggests jointly considering language and visual information is necessary to our task.

Overall, these SoTA model results show that a simple combination of SoTA navigation and referring expression methods would not necessarily lead to the best performance as failures from either the navigator or the pointer would decrease the overall success. In this paper, we make the first attempt to enable the navigator and pointer to work interactively as described in Sec. 4.3. The results in Tab. 3 show that our interactive model achieves consistently better results than non-interactive ones. The FAST-Short can be treated as our ablated model that without our proposed interaction module. Our final model achieves a gain of 4.2% on the test split.

Referring Expression-Only.

We also report the Referring Expression-Only performance. In this setting, the agent is placed at the ground-truth target location, and then referring expression comprehension models are tested.

We test the SoTA models such as MattNet [29] and CM-Erase [17] as well as a simple CNN-RNN baseline model with triplet ranking loss. Tab. 4 presents the results with human performance. It shows that the SoTA models achieve around 50% accuracy on the test split11 1 These SoTA models achieve 80% accuracy on ReferCOCO [30], a golden benchmark for referring expression. which are far more better than the results when jointly considering the navigation and referring expression shown in Tab. 3. Even though, there is still a 40% gap to human performance, suggesting that our proposed REVERIE task is challenging.

Val Seen Val UnSeen Test
Baseline 30.69 18.63 16.18
MAttNet [29] 68.45 56.63 48.98
CM-Erase [17] 65.21 54.02 45.25
Human 90.76
Table 4: Referring expression comprehension success rate (%) at the ground truth goal viewpoint of our REVERIE dataset.

6 Conclusion

Enable human-robots collaboration is a long-term goal. In this paper, we make a step further towards this goal by proposing a Remote Embodied Visual referring Expressions in Real Indoor Environments (REVERIE) task and dataset. The REVERIE is the first one to evaluate the capability of an agent to follow high-level natural languages instructions to navigate and identify the target object in previously unseen real images rendered buildings. We investigate several baselines and an interactive Navigator-Pointer agent model, of which the performance consistently demonstrate the significant necessity of further researches in this field.

We reach three main conclusions: First, REVERIE is interesting because existing vision and language methods can be easily plugged in. Second, the challenge of understanding and executing high-level instructions is significant. Finally, the combination of instruction navigation and referring expression comprehension is a challenging task due to the large gap to human performance.

7 Acknowledgments

We thank Sam Bahrami and Phil Roberts for their great help in the building of the REVERIE dataset.


  • [1] P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. D. Reid, S. Gould, and A. van den Hengel (2018) Vision-and-language navigation: interpreting visually-grounded navigation instructions in real environments. In CVPR, pp. 3674–3683. Cited by: item 2, Table 2, §1, §1, §2, §3.1, §3.2, §3.3, §3.5, §3.5, Table 3, 3rd item, §5.2, §5.2.
  • [2] D. Bahdanau, K. Cho, and Y. Bengio (2015) Neural machine translation by jointly learning to align and translate. In ICLR, Cited by: 3rd item.
  • [3] V. Blukis, D. K. Misra, R. A. Knepper, and Y. Artzi (2018) Mapping navigation instructions to continuous control actions with position-visitation prediction. In 2nd Annual Conference on Robot Learning, CoRL 2018, Zürich, Switzerland, 29-31 October 2018, Proceedings, pp. 505–518. Cited by: Table 2.
  • [4] A. X. Chang, A. Dai, T. A. Funkhouser, M. Halber, M. Nießner, M. Savva, S. Song, A. Zeng, and Y. Zhang (2017) Matterport3D: learning from RGB-D data in indoor environments. In 2017 International Conference on 3D Vision, 3DV 2017, Qingdao, China, October 10-12, 2017, pp. 667–676. Cited by: §1, §3.3.
  • [5] H. Chen, A. Suhr, D. Misra, N. Snavely, and Y. Artzi (2019) Touchdown: natural language navigation and spatial reasoning in visual street environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12538–12547. Cited by: Table 2, §2.
  • [6] K. Chen, R. Kovvuri, and R. Nevatia (2017) Query-guided regression network with context policy for phrase grounding. In ICCV, pp. 824–832. Cited by: §2.
  • [7] A. Das, S. Datta, G. Gkioxari, S. Lee, D. Parikh, and D. Batra (2018) Embodied question answering. In CVPR, pp. 1–10. Cited by: Table 2, §1, §2.
  • [8] H. de Vries, K. Shuster, D. Batra, D. Parikh, J. Weston, and D. Kiela (2018) Talk the walk: navigating new york city through grounded dialogue. CoRR abs/1807.03367. Cited by: Table 2.
  • [9] D. Fried, R. Hu, V. Cirik, A. Rohrbach, J. Andreas, L. Morency, T. Berg-Kirkpatrick, K. Saenko, D. Klein, and T. Darrell (2018) Speaker-follower models for vision-and-language navigation. In NeurIPS, pp. 3318–3329. Cited by: §2.
  • [10] D. Gordon, A. Kembhavi, M. Rastegari, J. Redmon, D. Fox, and A. Farhadi (2018) IQA: visual question answering in interactive environments. In CVPR, pp. 4089–4098. Cited by: Table 2, §2.
  • [11] S. Hochreiter and J. Schmidhuber (1997) Long short-term memory. Neural Computation 9 (8), pp. 1735–1780. Cited by: 3rd item.
  • [12] R. Hu, M. Rohrbach, J. Andreas, T. Darrell, and K. Saenko (2017) Modeling relationships in referential expressions with compositional modular networks. In CVPR, pp. 4418–4427. Cited by: §2.
  • [13] R. Hu, H. Xu, M. Rohrbach, J. Feng, K. Saenko, and T. Darrell (2016) Natural language object retrieval. In CVPR, pp. 4555–4564. Cited by: §2.
  • [14] S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg (2014) Referitgame: referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 787–798. Cited by: §1.
  • [15] L. Ke, X. Li, Y. Bisk, A. Holtzman, Z. Gan, J. Liu, J. Gao, Y. Choi, and S. S. Srinivasa (2019) Tactical rewind: self-correction via backtracking in vision-and-language navigation. In CVPR, pp. 6741–6749. Cited by: §2, §4.1, Table 3, 3rd item, §5.2.
  • [16] J. Liu, L. Wang, and M. Yang (2017) Referring expression generation and comprehension via attributes. In ICCV, pp. 4866–4874. Cited by: §2.
  • [17] X. Liu, Z. Wang, J. Shao, X. Wang, and H. Li (2019) Improving referring expression grounding with cross-modal attention-guided erasing. In CVPR, pp. 1950–1959. Cited by: §5.2, Table 4.
  • [18] R. Luo and G. Shakhnarovich (2017) Comprehension-guided referring expressions. In CVPR, pp. 3125–3134. Cited by: §2.
  • [19] C. Ma, J. Lu, Z. Wu, G. AlRegib, Z. Kira, R. Socher, and C. Xiong (2019) Self-monitoring navigation agent via auxiliary progress estimation. In ICLR, Cited by: §2, §4.1, Table 3, 1st item.
  • [20] C. Ma, Z. Wu, G. AlRegib, C. Xiong, and Z. Kira (2019) The regretful agent: heuristic-aided navigation through progress estimation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pp. 6732–6740. Cited by: §2.
  • [21] M. MacMahon, B. Stankiewicz, and B. Kuipers (2006) Walk the talk: connecting language, knowledge, and action in route instructions. In Proceedings, The Twenty-First National Conference on Artificial Intelligence and the Eighteenth Innovative Applications of Artificial Intelligence Conference, July 16-20, 2006, Boston, Massachusetts, USA, pp. 1475–1482. Cited by: Table 2.
  • [22] J. Mao, J. Huang, A. Toshev, O. Camburu, A. Yuille, and K. Murphy (2016) Generation and comprehension of unambiguous object descriptions. In CVPR, Cited by: §1.
  • [23] K. Nguyen and H. Daumé III (2019) Help, anna! visual navigation with natural multimodal assistance via retrospective curiosity-encouraging imitation learning. arXiv preprint arXiv:1909.01871. Cited by: Table 2, §2.
  • [24] K. Nguyen, D. Dey, C. Brockett, and B. Dolan (2019) Vision-based navigation with language-based assistance via imitation learning with indirect intervention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 12527–12537. Cited by: Table 2, §2.
  • [25] J. Thomason, M. Murray, M. Cakmak, and L. Zettlemoyer (2019) Vision-and-dialog navigation. CoRR abs/1907.04957. Cited by: Table 2, §2.
  • [26] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin (2017) Attention is all you need. In NeurIPS, pp. 5998–6008. Cited by: §4.1.
  • [27] X. Wang, Q. Huang, A. Çelikyilmaz, J. Gao, D. Shen, Y. Wang, W. Y. Wang, and L. Zhang (2018) Reinforced cross-modal matching and self-supervised imitation learning for vision-language navigation. CoRR abs/1811.10092. Cited by: §2, Table 3, 2nd item.
  • [28] X. Wang, W. Xiong, H. Wang, and W. Y. Wang (2018) Look before you leap: bridging model-free and model-based reinforcement learning for planned-ahead vision-and-language navigation. In ECCV, pp. 38–55. Cited by: §2.
  • [29] L. Yu, Z. Lin, X. Shen, J. Yang, X. Lu, M. Bansal, and T. L. Berg (2018) MAttNet: modular attention network for referring expression comprehension. In CVPR, pp. 1307–1315. Cited by: §2, §4.2, Table 3, §5.2, Table 4.
  • [30] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg (2016) Modeling context in referring expressions. In ECCV, pp. 69–85. Cited by: Table 2, §2, §3.5, §3.5, footnote 1.


In this supplementary material, we provide detailed explanation of evaluation metrics, examples of collected data, data collecting tools, how human test are performed and visualisation of several REVERIE results.

1 Evaluation Metrics

  • Navigation Success: a navigation is considered successful only if the target object can be observed at the stop viewpoint. Please note that to encourage the agent to approach closer to the target object, we set the objects visible if they are within 3 meters away from the current location.

  • Navigation Oracle Success: a navigation is considered oracle successful if the target object can be observed at one of its passed viewpoints.

  • Navigation SPL: it is the navigation success weighted by the length of navigation path, which is

    1Ni=1NSiimax(i,pi) (1)

    where N is the number of tasks, Si{0,1} is a binary indicator of success of task i, i is the shortest length between the starting viewpoint and the goal viewpoint of task i, and pi is the path length of an agent for task i.

  • Navigation Length: trajectory length in meters.

  • REVERIE Success: a task is considered REVERIE successful if the output bounding box has an IoU (intersection over union) 0.5 with the ground truth.

2 Typical Samples of The REVERIE Task

In Figure 1, we present several typical samples of the proposed REVERIE task. It shows the diversity in object category, goal region, path instruction, and target object referring expression.

3 Data Collecting Tools

To collect data for the REVERIE task, we develop a WebGL based data collecting tool as shown in Figure 2 and Figure 3. To facilitate the workers, we provide real-time updated reference information in the web page according to the location of the agent, including the current level/total level, the current region, and the number of regions in the build having the same function. At the goal location, in addition to highlighting the target object with a red 3D rectangle, we also provide the label of the target object and the number of objects falling in the same category with the target object. Text and video instructions are provided for workers to understand how to make high quality annotations as shown in Figure 2.

4 Human Performance Test

To obtain the machine-human performance gap, we develop a WebGL based tool as shown in Figure 4 to test human performance. In the tool, we show an instruction about a remote object to the worker. Then the worker needs to navigate to the goal location and select one object as the target object from a range of object candidates. The worker can look around and go forward/backword by dragging or clicking.

5 Visualisation of REVERIE Results

In Figure 5, we provide the visualisation of several REVERIE results obtained by the typical state-of-the-art method, FAST-short, and the typical baseline method, R2R-SF.

Figure 1: Several typical samples of the collected dataset, which involves various object category, goal region, path instruction, and object referring expression.

Figure 2: Data collecting interface part I: instructions for AMT workers.

Figure 3: Data collecting interface Part II: assistant information and user input field.

Figure 4: Human test interface. Workers need first to navigate to the goal location by clicking or dragging mouse, and then identify the target object. Only objects within 3 meters from the current location are highlighted. Different colors are used to facilitate workers to distinguish one object from others.
Figure 5: Visualisation of several REVERIE results, including trajectories and referring expression grounding of three typical methods. Colorized dots denote reachable locations, and different colors mark locations belonging to different regions according to the Matterport dataset.