Abstract
As LLMs are increasingly deployed in global applications, the importance ofcultural sensitivity becomes paramount, ensuring that users from diversebackgrounds feel respected and understood. Cultural harm can arise when thesemodels fail to align with specific cultural norms, resulting inmisrepresentations or violations of cultural values. This work addresses thechallenges of ensuring cultural sensitivity in LLMs, especially insmall-parameter models that often lack the extensive training data needed tocapture global cultural nuances. We present two key contributions: (1) Acultural harm test dataset, created to assess model outputs across differentcultural contexts through scenarios that expose potential culturalinsensitivities, and (2) A culturally aligned preference dataset, aimed atrestoring cultural sensitivity through fine-tuning based on feedback fromdiverse annotators. These datasets facilitate the evaluation and enhancement ofLLMs, ensuring their ethical and safe deployment across different culturallandscapes. Our results show that integrating culturally aligned feedback leadsto a marked improvement in model behavior, significantly reducing thelikelihood of generating culturally insensitive or harmful content. Ultimately,this work paves the way for more inclusive and respectful AI systems, fosteringa future where LLMs can safely and ethically navigate the complexities ofdiverse cultural landscapes.