Abstract
ChatGPT has shown strong capabilities in natural language generation tasks,which naturally leads researchers to explore where its abilities end. In thispaper, we examine whether ChatGPT can be used for zero-shot textclassification, more specifically, automatic genre identification. We compareChatGPT with a multilingual XLM-RoBERTa language model that was fine-tuned ondatasets, manually annotated with genres. The models are compared on test setsin two languages: English and Slovenian. Results show that ChatGPT outperformsthe fine-tuned model when applied to the dataset which was not seen before byeither of the models. Even when applied on Slovenian language as anunder-resourced language, ChatGPT's performance is no worse than when appliedto English. However, if the model is fully prompted in Slovenian, theperformance drops significantly, showing the current limitations of ChatGPTusage on smaller languages. The presented results lead us to questioningwhether this is the beginning of an end of laborious manual annotationcampaigns even for smaller languages, such as Slovenian.