Abstract
The rapid growth of social media platforms has led to concerns aboutradicalization, filter bubbles, and content bias. Existing approaches toclassifying ideology are limited in that they require extensive human effort,the labeling of large datasets, and are not able to adapt to evolvingideological contexts. This paper explores the potential of Large LanguageModels (LLMs) for classifying the political ideology of online content in thecontext of the two-party US political spectrum through in-context learning(ICL). Our extensive experiments involving demonstration selection inlabel-balanced fashion, conducted on three datasets comprising news articlesand YouTube videos, reveal that our approach significantly outperformszero-shot and traditional supervised methods. Additionally, we evaluate theinfluence of metadata (e.g., content source and descriptions) on ideologicalclassification and discuss its implications. Finally, we show how providing thesource for political and non-political content influences the LLM'sclassification.