Abstract
Forecasting is a critical task in decision making across various domains.While numerical data provides a foundation, it often lacks crucial contextnecessary for accurate predictions. Human forecasters frequently rely onadditional information, such as background knowledge or constraints, which canbe efficiently communicated through natural language. However, the ability ofexisting forecasting models to effectively integrate this textual informationremains an open question. To address this, we introduce "Context is Key" (CiK),a time series forecasting benchmark that pairs numerical data with diversetypes of carefully crafted textual context, requiring models to integrate bothmodalities. We evaluate a range of approaches, including statistical models,time series foundation models, and LLM-based forecasters, and propose a simpleyet effective LLM prompting method that outperforms all other tested methods onour benchmark. Our experiments highlight the importance of incorporatingcontextual information, demonstrate surprising performance when using LLM-basedforecasting models, and also reveal some of their critical shortcomings. Bypresenting this benchmark, we aim to advance multimodal forecasting, promotingmodels that are both accurate and accessible to decision-makers with variedtechnical expertise. The benchmark can be visualized athttps://servicenow.github.io/context-is-key-forecasting/v0/ .