Abstract
Large language models (LLMs) are highly adept at question answering andreasoning tasks, but when reasoning in situational context, human expectationsvary depending on the relevant cultural common ground. As human languages areassociated with diverse cultures, LLMs should also be culturally-diversereasoners. In this paper, we study the ability of a wide range ofstate-of-the-art multilingual LLMs (mLLMs) to reason with proverbs and sayingsin a conversational context. Our experiments reveal that: (1) mLLMs 'knows'limited proverbs and memorizing proverbs does not mean understanding themwithin a conversational context; (2) mLLMs struggle to reason with figurativeproverbs and sayings, and when asked to select the wrong answer (instead ofasking it to select the correct answer); and (3) there is a "culture gap" inmLLMs when reasoning about proverbs and sayings translated from otherlanguages. We construct and release our evaluation dataset MAPS (MulticultrAlProverbs and Sayings) for proverb understanding with conversational context forsix different languages.