Abstract
Despite the superior performance of large language models to generate naturallanguage texts, it is hard to generate texts with correct logic according to agiven task, due to the difficulties for neural models to capture implied rulesfrom free-form texts. In this paper, we propose a novel graph-based languagemodel, Logical-GLM, to infuse logic into language models for more valid textgeneration and interpretability. Specifically, we first capture informationfrom natural language instructions and construct logical bayes graphs thatgenerally describe domains. Next, we generate logical skeletons to guidelanguage model training, infusing domain knowledge into language models.Finally, we alternately optimize the searching policy of graphs and languagemodels until convergence. The experimental results show that Logical-GLM isboth effective and efficient compared with traditional language models, despiteusing smaller-scale training data and fewer parameters. Our approach cangenerate instructional texts with more correct logic owing to the internalizeddomain knowledge. Moreover, the usage of logical graphs reflects the innermechanism of the language models, which improves the interpretability ofblack-box models.