Abstract
Sensational headlines are headlines that capture people's attention andgenerate reader interest. Conventional abstractive headline generation methods,unlike human writers, do not optimize for maximal reader attention. In thispaper, we propose a model that generates sensational headlines without labeleddata. We first train a sensationalism scorer by classifying online headlineswith many comments ("clickbait") against a baseline of headlines generated froma summarization model. The score from the sensationalism scorer is used as thereward for a reinforcement learner. However, maximizing the noisysensationalism reward will generate unnatural phrases instead of sensationalheadlines. To effectively leverage this noisy reward, we propose a novel lossfunction, Auto-tuned Reinforcement Learning (ARL), to dynamically balancereinforcement learning (RL) with maximum likelihood estimation (MLE). Humanevaluation shows that 60.8% of samples generated by our model are sensational,which is significantly better than the Pointer-Gen baseline and other RLmodels.