Abstract
Existing works on long-term open-domain dialogues focus on evaluating modelresponses within contexts spanning no more than five chat sessions. Despiteadvancements in long-context large language models (LLMs) and retrievalaugmented generation (RAG) techniques, their efficacy in very long-termdialogues remains unexplored. To address this research gap, we introduce amachine-human pipeline to generate high-quality, very long-term dialogues byleveraging LLM-based agent architectures and grounding their dialogues onpersonas and temporal event graphs. Moreover, we equip each agent with thecapability of sharing and reacting to images. The generated conversations areverified and edited by human annotators for long-range consistency andgrounding to the event graphs. Using this pipeline, we collect LoCoMo, adataset of very long-term conversations, each encompassing 300 turns and 9Ktokens on avg., over up to 35 sessions. Based on LoCoMo, we present acomprehensive evaluation benchmark to measure long-term memory in models,encompassing question answering, event summarization, and multi-modal dialoguegeneration tasks. Our experimental results indicate that LLMs exhibitchallenges in understanding lengthy conversations and comprehending long-rangetemporal and causal dynamics within dialogues. Employing strategies likelong-context LLMs or RAG can offer improvements but these models stillsubstantially lag behind human performance.