Marrying topic models and language models exposes language understanding to abroader source of document-level context beyond sentences via topics. Whileintroducing topical semantics in language models, existing approachesincorporate latent document topic proportions and ignore topical discourse insentences of the document. This work extends the line of research byadditionally introducing an explainable topic representation in languageunderstanding, obtained from a set of key terms correspondingly for each latenttopic of the proportion. Moreover, we retain sentence-topic associations alongwith document-topic association by modeling topical discourse for everysentence in the document. We present a novel neural composite language modelthat exploits both the latent and explainable topics along with topicaldiscourse at sentence-level in a joint learning framework of topic and languagemodels. Experiments over a range of tasks such as language modeling, word sensedisambiguation, document classification, retrieval and text generationdemonstrate ability of the proposed model in improving language understanding.