Natural Language Interaction with Explainable AI Models

  • 2019-03-13 21:29:13
  • Arjun R Akula, Sinisa Todorovic, Joyce Y Chai, Song-Chun Zhu
  • 2


This paper presents an explainable AI (XAI) system that provides explanationsfor its predictions. The system consists of two key components -- namely, theprediction And-Or graph (AOG) model for recognizing and localizing concepts ofinterest in input data, and the XAI model for providing explanations to theuser about the AOG's predictions. In this work, we focus on the XAI modelspecified to interact with the user in natural language, whereas the AOG'spredictions are considered given and represented by the corresponding parsegraphs (pg's) of the AOG. Our XAI model takes pg's as input and providesanswers to the user's questions using the following types of reasoning: directevidence (e.g., detection scores), part-based inference (e.g., detected partsprovide evidence for the concept asked), and other evidences fromspatio-temporal context (e.g., constraints from the spatio-temporal surround).We identify several correlations between user's questions and the XAI answersusing Youtube Action dataset.


Introduction (beta)



Conclusion (beta)