Abstract
Panoptic Scene Graph Generation (PSG) aims at achieving a comprehensive imageunderstanding by simultaneously segmenting objects and predicting relationsamong objects. However, the long-tail problem among relations leads tounsatisfactory results in real-world applications. Prior methods predominantlyrely on vision information or utilize limited language information, such asobject or relation names, thereby overlooking the utility of languageinformation. Leveraging the recent progress in Large Language Models (LLMs), wepropose to use language information to assist relation prediction, particularlyfor rare relations. To this end, we propose the Vision-Language Prompting(VLPrompt) model, which acquires vision information from images and languageinformation from LLMs. Then, through a prompter network based on attentionmechanism, it achieves precise relation prediction. Our extensive experimentsshow that VLPrompt significantly outperforms previous state-of-the-art methodson the PSG dataset, proving the effectiveness of incorporating languageinformation and alleviating the long-tail problem of relations.