Abstract
Algorithmic decisions in critical domains such as hiring, college admissions,and lending are often based on rankings. Given the impact of these decisions onindividuals, organizations, and population groups, it is essential tounderstand them - to help individuals improve their ranking position, designbetter ranking procedures, and ensure legal compliance. In this paper, we arguethat explainability methods for classification and regression, such as SHAP,are insufficient for ranking tasks, and present ShaRP - Shapley Values forRankings and Preferences - a framework that explains the contributions offeatures to various aspects of a ranked outcome. ShaRP computes feature contributions for various ranking-specific profitfunctions, such as rank and top-k, and also includes a novel Shapleyvalue-based method for explaining pairwise preference outcomes. We provide aflexible implementation of ShaRP, capable of efficiently and comprehensivelyexplaining ranked and pairwise outcomes over tabular data, in score-basedranking and learning-to-rank tasks. Finally, we develop a comprehensiveevaluation methodology for ranking explainability methods, showing throughqualitative, quantitative, and usability studies that our rank-aware QoIs offercomplementary insights, scale effectively, and help users interpret rankedoutcomes in practice.