With the explosive influence caused by the success of large language models(LLM) like ChatGPT and GPT-4, there has been an extensive amount of recent workshowing that foundation models can be used to solve a large variety of tasks.However, there is very limited work that shares insights on multi-agentplanning. Multi-agent planning is different from other domains by combining thedifficulty of multi-agent coordination and planning, and making it hard toleverage external tools to facilitate the reasoning needed. In this paper, wefocus on the problem of multi-agent path finding (MAPF), which is also known asmulti-robot route planning, and study the performance of solving MAPF withLLMs. We first show the motivating success on an empty room map withoutobstacles, then the failure to plan on the harder room map and maze map of thestandard MAPF benchmark. We present our position on why directly solving MAPFwith LLMs has not been successful yet, and we use various experiments tosupport our hypothesis. Based on our results, we discussed how researchers withdifferent backgrounds could help with this problem from different perspectives.