System-provided explanations for recommendations are an important componenttowards transparent and trustworthy AI. In state-of-the-art research, this is aone-way signal, though, to improve user acceptance. In this paper, we turn therole of explanations around and investigate how they can contribute toenhancing the quality of generated recommendations themselves. We devise ahuman-in-the-loop framework, called ELIXIR, where user feedback on explanationsis leveraged for pairwise learning of user preferences. ELIXIR leveragesfeedback on pairs of recommendations and explanations to learn user-specificlatent preference vectors, overcoming sparseness by label propagation withitem-similarity-based neighborhoods. Our framework is instantiated usinggeneralized graph recommendation via Random Walk with Restart. Insightfulexperiments with a real user study show significant improvements in movie andbook recommendations over item-level feedback.