Soft Measures for Extracting Causal Collective Intelligence

  • 2024-09-27 17:54:36
  • Maryam Berijanian, Spencer Dork, Kuldeep Singh, Michael Riley Millikan, Ashlin Riggs, Aadarsh Swaminathan, Sarah L. Gibbs, Scott E. Friedman, Nathan Brugnone
  • 0

Abstract

Understanding and modeling collective intelligence is essential foraddressing complex social systems. Directed graphs called fuzzy cognitive maps(FCMs) offer a powerful tool for encoding causal mental models, but extractinghigh-integrity FCMs from text is challenging. This study presents an approachusing large language models (LLMs) to automate FCM extraction. We introducenovel graph-based similarity measures and evaluate them by correlating theiroutputs with human judgments through the Elo rating system. Results showpositive correlations with human evaluations, but even the best-performingmeasure exhibits limitations in capturing FCM nuances. Fine-tuning LLMsimproves performance, but existing measures still fall short. This studyhighlights the need for soft similarity measures tailored to FCM extraction,advancing collective intelligence modeling with NLP.

 

Quick Read (beta)

loading the full paper ...