Abstract
LLMs enable an exciting new class of data processing applications over largecollections of unstructured documents. Several new programming frameworks haveenabled developers to build these applications by composing them out ofsemantic operators: a declarative set of AI-powered data transformations withnatural language specifications. These include LLM-powered maps, filters,joins, etc. used for document processing tasks such as information extraction,summarization, and more. While systems of semantic operators have achievedstrong performance on benchmarks, they can be difficult to optimize. Anoptimizer for this setting must determine how to physically implement eachsemantic operator in a way that optimizes the system globally. Existingoptimizers are limited in the number of optimizations they can apply, and most(if not all) cannot optimize system quality, cost, or latency subject toconstraint(s) on the other dimensions. In this paper we present Abacus, anextensible, cost-based optimizer which searches for the best implementation ofa semantic operator system given a (possibly constrained) optimizationobjective. Abacus estimates operator performance by leveraging a minimal set ofvalidation examples and, if available, prior beliefs about operatorperformance. We evaluate Abacus on document processing workloads in thebiomedical and legal domains (BioDEX; CUAD) and multi-modal question answering(MMQA). We demonstrate that systems optimized by Abacus achieve 18.7%-39.2%better quality and up to 23.6x lower cost and 4.2x lower latency than the nextbest system.