Abstract
Large language models (LLMs) can generate fluent summaries across domainsusing prompting techniques, reducing the need to train models for summarizationapplications. However, crafting effective prompts that guide LLMs to generatesummaries with the appropriate level of detail and writing style remains achallenge. In this paper, we explore the use of salient information extractedfrom the source document to enhance summarization prompts. We show that addingkeyphrases in prompts can improve ROUGE F1 and recall, making the generatedsummaries more similar to the reference and more complete. The number ofkeyphrases can control the precision-recall trade-off. Furthermore, ouranalysis reveals that incorporating phrase-level salient information issuperior to word- or sentence-level. However, the impact on hallucination isnot universally positive across LLMs. To conduct this analysis, we introduceKeyphrase Signal Extractor (CriSPO), a lightweight model that can be finetunedto extract salient keyphrases. By using CriSPO, we achieve consistent ROUGEimprovements across datasets and open-weight and proprietary LLMs without anyLLM customization. Our findings provide insights into leveraging salientinformation in building prompt-based summarization systems.