Abstract
We introduce AutoJudge, a framework that accelerates large language model(LLM) inference with task-specific lossy speculative decoding. Instead ofmatching the original model output distribution token-by-token, we identifywhich of the generated tokens affect the downstream quality of the generatedresponse, relaxing the guarantee so that the "unimportant" tokens can begenerated faster. Our approach relies on a semi-greedy search algorithm to testwhich of the mismatches between target and draft model should be corrected topreserve quality, and which ones may be skipped. We then train a lightweightclassifier based on existing LLM embeddings to predict, at inference time,which mismatching tokens can be safely accepted without compromising the finalanswer quality. We test our approach with Llama 3.2 1B (draft) and Llama 3.1 8B(target) models on zero-shot GSM8K reasoning, where it achieves up to 1.5x moreaccepted tokens per verification cycle with under 1% degradation in answeraccuracy compared to standard speculative decoding and over 2x with small lossin accuracy. When applied to the LiveCodeBench benchmark, our approachautomatically detects other, programming-specific important tokens and showssimilar speedups, demonstrating its ability to generalize across tasks.