Abstract
Conformal prediction (CP) is an emerging uncertainty quantification frameworkthat allows us to construct a prediction set to cover the true label with apre-specified marginal or conditional probability. Although the valid coverageguarantee has been extensively studied for classification problems, CP oftenproduces large prediction sets which may not be practically useful. This issueis exacerbated for the setting of class-conditional coverage on imbalancedclassification tasks with many and/or imbalanced classes. This paper proposesthe Rank Calibrated Class-conditional CP (RC3P) algorithm to reduce theprediction set sizes to achieve class-conditional coverage, where the validcoverage holds for each class. In contrast to the standard class-conditional CP(CCP) method that uniformly thresholds the class-wise conformity score for eachclass, the augmented label rank calibration step allows RC3P to selectivelyiterate this class-wise thresholding subroutine only for a subset of classeswhose class-wise top-k error is small. We prove that agnostic to the classifierand data distribution, RC3P achieves class-wise coverage. We also show thatRC3P reduces the size of prediction sets compared to the CCP method.Comprehensive experiments on multiple real-world datasets demonstrate that RC3Pachieves class-wise coverage and 26.25% reduction in prediction set sizes onaverage.