Abstract
The advancements in the Large Language Model (LLM) have helped in solvingseveral problems related to language processing. Most of the researches havefocused on the English language only, because of its popularity and abundanceon the internet. However, a high-performance language model for Hindi and otherIndic languages is lacking in the literature. In this work, we have pre-trainedtwo autoregressive LLM models for the Hindi language, namely HindiLLM-Small andHindiLLM-Medium. We use a two-step process comprising unsupervised pre-trainingand supervised fine-tuning. First, we create a large and high-quality textcorpus for unsupervised pre-training. Next, we train a Byte-Pair Encoding,named HindiLLM tokenizer, using the pre-training text data. We then performtraining on the unlabeled data, known as the pre-training step, to get theHindiLLM base models. Furthermore, we perform fine-tuning of the HindiLLM basemodels for different tasks like sentiment analysis, text classification,natural language inference, and multiple choice question-answer on popularlabeled datasets to measure the real-world performance. The evaluation showsthat the HindiLLM-based fine-tuned models outperform several models in most ofthe language related tasks.