Abstract
Search-Oriented Conversational AI (SCAI) is an established venue thatregularly puts a spotlight upon the recent work advancing the field ofconversational search. SCAI'21 was organised as an independent on-line eventand featured a shared task on conversational question answering. Since all ofthe participant teams experimented with answer generation models for this task,we identified evaluation of answer correctness in this settings as the majorchallenge and a current research gap. Alongside the automatic evaluation, weconducted two crowdsourcing experiments to collect annotations for answerplausibility and faithfulness. As a result of this shared task, the originalconversational QA dataset used for evaluation was further extended withalternative correct answers produced by the participant systems.