Abstract
In the Learning to Defer (L2D) framework, a prediction model can either makea prediction or defer it to an expert, as determined by a rejector. Current L2Dmethods train the rejector to decide whether to reject the entire prediction,which is not desirable when the model predicts long sequences. We present anL2D setting for sequence outputs where the system can defer specific outputs ofthe whole model prediction to an expert in an effort to interleave the expertand machine throughout the prediction. We propose two types of model-basedpost-hoc rejectors for pre-trained predictors: a token-level rejector, whichdefers specific token predictions to experts with next token predictioncapabilities, and a one-time rejector for experts without such abilities, whichdefers the remaining sequence from a specific point onward. In the experiments,we also empirically demonstrate that such granular deferrals achieve bettercost-accuracy tradeoffs than whole deferrals on Traveling salesman solvers andNews summarization models.