Abstract
Mixed Reality (MR) is a powerful interactive technology that yields new typesof user experience. We present a semantic based interactive MR framework thatexceeds the current geometry level approaches, a step change in generatinghigh-level context-aware interactions. Our key insight is to build semanticunderstanding in MR that not only can greatly enhance user experience throughobject-specific behaviours, but also pave the way for solving complexinteraction design challenges. The framework generates semantic properties ofthe real world environment through dense scene reconstruction and deep imageunderstanding. We demonstrate our approach with a material-aware prototypesystem for generating context-aware physical interactions between the real andthe virtual objects. Quantitative and qualitative evaluations are carried outand the results show that the framework delivers accurate and fast semanticinformation in interactive MR environment, providing effective semantic levelinteractions.