Abstract
Large Language Models (LLMs) are now increasingly widely used to simulatepersonas in virtual environments, leveraging their instruction-followingcapability. However, we discovered that even state-of-the-art LLMs cannotsimulate personas with reversed performance (e.g., student personas with lowproficiency in educational settings), which impairs the simulation diversityand limits the practical applications of the simulated environments. In thiswork, using mathematical reasoning as a representative scenario, we propose thefirst benchmark dataset for evaluating LLMs on simulating personas withreversed performance, a capability that we dub "counterfactual instructionfollowing". We evaluate both open-weight and closed-source LLMs on this taskand find that LLMs, including the OpenAI o1 reasoning model, all struggle tofollow counterfactual instructions for simulating reversedly performingpersonas. Intersectionally simulating both the performance level and the racepopulation of a persona worsens the effect even further. These resultshighlight the challenges of counterfactual instruction following and the needfor further research.