We study the effect of applying a language model (LM) on the output ofAutomatic Speech Recognition (ASR) systems for Indic languages. We fine-tunewav2vec $2.0$ models for $18$ Indic languages and adjust the results withlanguage models trained on text derived from a variety of sources. Our findingsdemonstrate that the average Character Error Rate (CER) decreases by over $28$\% and the average Word Error Rate (WER) decreases by about $36$ \% afterdecoding with LM. We show that a large LM may not provide a substantialimprovement as compared to a diverse one. We also demonstrate that high qualitytranscriptions can be obtained on domain-specific data without retraining theASR model and show results on biomedical domain.