Abstract
Optical flow, semantic segmentation, and surface normals represent differentinformation modalities, yet together they bring better cues for sceneunderstanding problems. In this paper, we study the influence between the threemodalities: how one impacts on the others and their efficiency in combination.We employ a modular approach using a convolutional refinement network which istrained supervised but isolated from RGB images to enforce joint modalityfeatures. To assist the training process, we create a large-scale syntheticoutdoor dataset that supports dense annotation of semantic segmentation,optical flow, and surface normals. The experimental results show positiveinfluence among the three modalities, especially for objects' boundaries,region consistency, and scene structures.