Abstract
Despite the effectiveness of sequence-to-sequence framework on the task ofShort-Text Conversation (STC), the issue of under-exploitation of training data(i.e., the supervision signals from query text is \textit{ignored}) stillremains unresolved. Also, the adopted \textit{maximization}-based decodingstrategies, inclined to generating the generic responses or responses withrepetition, are unsuited to the STC task. In this paper, we propose toformulate the STC task as a language modeling problem and tailor-make atraining strategy to adapt a language model for response generation. To enhancegeneration performance, we design a relevance-promoting transformer languagemodel, which performs additional supervised source attention after theself-attention to increase the importance of informative query tokens incalculating the token-level representation. The model further refines the queryrepresentation with relevance clues inferred from its multiple referencesduring training. In testing, we adopt a\textit{randomization-over-maximization} strategy to reduce the generation ofgeneric responses. Experimental results on a large Chinese STC datasetdemonstrate the superiority of the proposed model on relevance metrics anddiversity metrics.\footnote{Code available athttps://ai.tencent.com/ailab/nlp/dialogue/.