Abstract
In Augmented Reality (AR) environment, realistic interactions between thevirtual and real objects play a crucial role in user experience. Much of recentadvances in AR has been largely focused on developing geometry-awareenvironment, but little has been done in dealing with interactions at thesemantic level. High-level scene understanding and semantic descriptions in ARwould allow effective design of complex applications and enhanced userexperience. In this paper, we present a novel approach and a prototype systemthat enables the deeper understanding of semantic properties of the real worldenvironment, so that realistic physical interactions between the real and thevirtual objects can be generated. A material-aware AR environment has beencreated based on the deep material learning using a fully convolutional network(FCN). The state-of-the-art dense Simultaneous Localisation and Mapping (SLAM)has been used for the semantic mapping. Together with efficient accelerated 3Dray casting, natural and realistic physical interactions are generated forinteractive AR games. Our approach has significant impact on the futuredevelopment of advanced AR systems and applications.