Abstract
To assist human fact-checkers, researchers have developed automatedapproaches for visual misinformation detection. These methods assign veracityscores by identifying inconsistencies between the image and its caption, or bydetecting forgeries in the image. However, they neglect a crucial point of thehuman fact-checking process: identifying the original meta-context of theimage. By explaining what is actually true about the image, fact-checkers canbetter detect misinformation, focus their efforts on check-worthy visualcontent, engage in counter-messaging before misinformation spreads widely, andmake their explanation more convincing. Here, we fill this gap by introducingthe task of automated image contextualization. We create 5Pils, a dataset of1,676 fact-checked images with question-answer pairs about their originalmeta-context. Annotations are based on the 5 Pillars fact-checking framework.We implement a first baseline that grounds the image in its originalmeta-context using the content of the image and textual evidence retrieved fromthe open web. Our experiments show promising results while highlighting severalopen challenges in retrieval and reasoning. We make our code and data publiclyavailable.