Abstract
Language Models (LMs) are indispensable tools shaping modern workflows, buttheir global effectiveness depends on understanding local socio-culturalcontexts. To address this, we introduce SANSKRITI, a benchmark designed toevaluate language models' comprehension of India's rich cultural diversity.Comprising 21,853 meticulously curated question-answer pairs spanning 28 statesand 8 union territories, SANSKRITI is the largest dataset for testing Indiancultural knowledge. It covers sixteen key attributes of Indian culture: ritualsand ceremonies, history, tourism, cuisine, dance and music, costume, language,art, festivals, religion, medicine, transport, sports, nightlife, andpersonalities, providing a comprehensive representation of India's culturaltapestry. We evaluate SANSKRITI on leading Large Language Models (LLMs), IndicLanguage Models (ILMs), and Small Language Models (SLMs), revealing significantdisparities in their ability to handle culturally nuanced queries, with manymodels struggling in region-specific contexts. By offering an extensive,culturally rich, and diverse dataset, SANSKRITI sets a new standard forassessing and improving the cultural understanding of LMs.