Abstract
For Large Language Models (LLMs) to be effectively deployed in a specificcountry, they must possess an understanding of the nation's culture and basicknowledge. To this end, we introduce National Alignment, which measures analignment between an LLM and a targeted country from two aspects: social valuealignment and common knowledge alignment. Social value alignment evaluates howwell the model understands nation-specific social values, while commonknowledge alignment examines how well the model captures basic knowledgerelated to the nation. We constructed KorNAT, the first benchmark that measuresnational alignment with South Korea. For the social value dataset, we obtainedground truth labels from a large-scale survey involving 6,174 unique Koreanparticipants. For the common knowledge dataset, we constructed samples based onKorean textbooks and GED reference materials. KorNAT contains 4K and 6Kmultiple-choice questions for social value and common knowledge, respectively.Our dataset creation process is meticulously designed and based on statisticalsampling theory and was refined through multiple rounds of human review. Theexperiment results of seven LLMs reveal that only a few models met ourreference score, indicating a potential for further enhancement. KorNAT hasreceived government approval after passing an assessment conducted by agovernment-affiliated organization dedicated to evaluating dataset quality.Samples and detailed evaluation protocols of our dataset can be found inhttps://selectstar.ai/ko/papers-national-alignment