Abstract
Relation extraction (RE) consistently involves a certain degree of labeled orunlabeled data even if under zero-shot setting. Recent studies have shown thatlarge language models (LLMs) transfer well to new tasks out-of-the-box simplygiven a natural language prompt, which provides the possibility of extractingrelations from text without any data and parameter tuning. This work focuses onthe study of exploring LLMs, such as ChatGPT, as zero-shot relation extractors.On the one hand, we analyze the drawbacks of existing RE prompts and attempt toincorporate recent prompt techniques such as chain-of-thought (CoT) to improvezero-shot RE. We propose the summarize-and-ask (\textsc{SumAsk}) prompting, asimple prompt recursively using LLMs to transform RE inputs to the effectivequestion answering (QA) format. On the other hand, we conduct comprehensiveexperiments on various benchmarks and settings to investigate the capabilitiesof LLMs on zero-shot RE. Specifically, we have the following findings: (i)\textsc{SumAsk} consistently and significantly improves LLMs performance ondifferent model sizes, benchmarks and settings; (ii) Zero-shot prompting withChatGPT achieves competitive or superior results compared with zero-shot andfully supervised methods; (iii) LLMs deliver promising performance inextracting overlapping relations; (iv) The performance varies greatly regardingdifferent relations. Different from small language models, LLMs are effectivein handling challenge none-of-the-above (NoTA) relation.