Abstract
Understanding and modeling collective intelligence is essential foraddressing complex social systems. Directed graphs called fuzzy cognitive maps(FCMs) offer a powerful tool for encoding causal mental models, but extractinghigh-integrity FCMs from text is challenging. This study presents an approachusing large language models (LLMs) to automate FCM extraction. We introducenovel graph-based similarity measures and evaluate them by correlating theiroutputs with human judgments through the Elo rating system. Results showpositive correlations with human evaluations, but even the best-performingmeasure exhibits limitations in capturing FCM nuances. Fine-tuning LLMsimproves performance, but existing measures still fall short. This studyhighlights the need for soft similarity measures tailored to FCM extraction,advancing collective intelligence modeling with NLP.