Abstract
Large language models (LLMs) have achieved remarkable successes on variousnatural language tasks. However, recent studies have found that there are stillsignificant challenges to the logical reasoning abilities of LLMs. This papersummarizes and categorizes the main challenges into two aspects: (1) Logicalquestion answering, LLMs often fail to generate the correct answer withincomplex logical problem which requires sophisticated deductive, inductive orabductive reasoning given a collection of premises and constrains. (2) Logicalconsistency, LLMs are prone to producing responses contradicting themselvesacross different questions. For example, a state-of-the-art Macawquestion-answering LLM answers Yes to both questions Is a magpie a bird? andDoes a bird have wings? but answers No to Does a magpie have wings?. Tofacilitate this research direction, we comprehensively investigate the mostcutting-edge methods and propose detailed taxonomies of these methods.Specifically, to accurately answer complex logic questions, previous methodscan be categorized based on reliance on external solvers, prompts, pretraining,and fine-tuning. To avoid logical contradictions, we discuss concepts andsolutions of various logical consistencies, including implication, negation,transitivity, factuality consistency, and their composites. In addition, wereview commonly used benchmark datasets and evaluation metrics, and discusspromising research directions, such as extensions to modal logic to account foruncertainty, and efficient algorithms satisfying multiple logical consistenciessimultaneously.