Abstract
We investigate the use of large language models (LLMs) as post-processingmodules for automatic speech recognition (ASR), focusing on their ability toperform error correction for disordered speech. In particular, we proposeconfidence-informed prompting, where word-level uncertainty estimates areembedded directly into LLM training to improve robustness and generalizationacross speakers and datasets. This approach directs the model to uncertain ASRregions and reduces overcorrection. We fine-tune a LLaMA 3.1 model and compareour approach to both transcript-only fine-tuning and post hoc confidence-basedfiltering. Evaluations show that our method achieves a 10% relative WERreduction compared to naive LLM correction on the Speech Accessibility Projectspontaneous speech and a 47% reduction on TORGO, demonstrating theeffectiveness of confidence-aware fine-tuning for impaired speech.