Abstract
Retrieval Augmented Generation (RAG) is widely employed to ground responsesto queries on domain-specific documents. But do RAG implementations leave outimportant information when answering queries that need an integrated analysisof information (e.g., Tell me good news in the stock market today.)? To addressthese concerns, RAG developers need to annotate information retrieval (IR) datafor their domain of interest, which is challenging because (1) domain-specificqueries usually need nuanced definitions of relevance beyond shallow semanticrelevance; and (2) human or GPT-4 annotation is costly and cannot cover all(query, document) pairs (i.e., annotation selection bias), thus harming theeffectiveness in evaluating IR recall. To address these challenges, we proposeDIRAS (Domain-specific Information Retrieval Annotation with Scalability), amanual-annotation-free schema that fine-tunes open-sourced LLMs to considernuanced relevance definition and annotate (partial) relevance labels withcalibrated relevance scores. Extensive evaluation shows that DIRAS enablessmaller (8B) LLMs to achieve GPT-4-level performance on annotating and rankingunseen (query, document) pairs, and is helpful for real-world RAG development.All code, LLM generations, and human annotations can be found in\url{https://github.com/EdisonNi-hku/DIRAS}.