Abstract
Deep learning models are often deployed in downstream tasks that the trainingprocedure may not be aware of. For example, models solely trained to achieveaccurate predictions may struggle to perform well on downstream tasks becauseseemingly small prediction errors may incur drastic task errors. The standardend-to-end learning approach is to make the task loss differentiable or tointroduce a differentiable surrogate that the model can be trained on. In thesesettings, the task loss needs to be carefully balanced with the prediction lossbecause they may have conflicting objectives. We propose take the task losssignal one level deeper than the parameters of the model and use it to learnthe parameters of the loss function the model is trained on, which can be doneby learning a metric in the prediction space. This approach does not alter theoptimal prediction model itself, but rather changes the model learning toemphasize the information important for the downstream task. This enables us toachieve the best of both worlds: a prediction model trained in the originalprediction space while also being valuable for the desired downstream task. Wevalidate our approach through experiments conducted in two main settings: 1)decision-focused model learning scenarios involving portfolio optimization andbudget allocation, and 2) reinforcement learning in noisy environments withdistracting states. The source code to reproduce our experiments is availableat https://github.com/facebookresearch/taskmet