Logical Tasks for Measuring Extrapolation and Rule Comprehension

  • 2022-11-14 19:55:54
  • Ippei Fujisawa, Ryota Kanai
  • 53

Abstract

Logical reasoning is essential in a variety of human activities. Arepresentative example of a logical task is mathematics. Recent large-scalemodels trained on large datasets have been successful in various fields, buttheir reasoning ability in arithmetic tasks is limited, which we reproduceexperimentally. Here, we recast this limitation as not unique to mathematicsbut common to tasks that require logical operations. We then propose a new setof tasks, termed logical tasks, which will be the next challenge to address.This higher point of view helps the development of inductive biases that havebroad impact beyond the solution of individual tasks. We define andcharacterize logical tasks and discuss system requirements for their solution.Furthermore, we discuss the relevance of logical tasks to concepts such asextrapolation, explainability, and inductive bias. Finally, we providedirections for solving logical tasks.

 

Quick Read (beta)

loading the full paper ...