Abstract
Artificial Intelligence (AI) systems are transforming critical sectors suchas healthcare, finance, and transportation, enhancing operational efficiencyand decision-making processes. However, their deployment in high-stakes domainshas exposed vulnerabilities that can result in significant societal harm. Tosystematically study and mitigate these risk, initiatives like the AI IncidentDatabase (AIID) have emerged, cataloging over 3,000 real-world AI failurereports. Currently, associating a new report with the appropriate AI Incidentrelies on manual expert intervention, limiting scalability and delaying theidentification of emerging failure patterns. To address this limitation, we propose a retrieval-based framework thatautomates the association of new reports with existing AI Incidents throughsemantic similarity modeling. We formalize the task as a ranking problem, whereeach report-comprising a title and a full textual description-is compared topreviously documented AI Incidents based on embedding cosine similarity.Benchmarking traditional lexical methods, cross-encoder architectures, andtransformer-based sentence embedding models, we find that the latterconsistently achieve superior performance. Our analysis further shows thatcombining titles and descriptions yields substantial improvements in rankingaccuracy compared to using titles alone. Moreover, retrieval performanceremains stable across variations in description length, highlighting therobustness of the framework. Finally, we find that retrieval performanceconsistently improves as the training set expands. Our approach provides ascalable and efficient solution for supporting the maintenance of the AIID.