Abstract
Entities and events have long been regarded as the crux of machine reasoning.Specifically, procedural texts have received increasing attention due to thedynamic nature of involved entities and events. Existing work has exclusivelyfocused on entity state tracking (e.g., the temperature of a pan) orcounterfactual event reasoning (e.g., how likely am I to burn myself bytouching the pan), while these two tasks are tightly intertwined. In this work,we propose CREPE, the first benchmark on causal reasoning about eventplausibility based on entity states. We experiment with strong large languagemodels and show that most models including GPT3 perform close to chance of .30F1, lagging far behind the human performance of .87 F1. Inspired by the findingthat structured representations such as programming languages benefits eventreasoning as a prompt to code language models such as Codex, we creativelyinject the causal relations between entities and events through intermediatevariables and boost the performance to .67 to .72 F1. Our proposed eventrepresentation not only allows for knowledge injection, but also marks thefirst successful attempt of chain-of-thought reasoning with code languagemodels.