Abstract
Large language models (LLMs) are useful tools with the capacity forperforming specific types of knowledge work at an effective scale. However, LLMdeployments in high-risk and safety-critical domains pose unique challenges,notably the issue of ``hallucination,'' where LLMs can generate fabricatedinformation. This is particularly concerning in settings such as drug safety,where inaccuracies could lead to patient harm. To mitigate these risks, we havedeveloped and demonstrated a proof of concept suite of guardrails specificallydesigned to mitigate certain types of hallucinations and errors for drugsafety, and potentially applicable to other medical safety-critical contexts.These guardrails include mechanisms to detect anomalous documents to preventthe ingestion of inappropriate data, identify incorrect drug names or adverseevent terms, and convey uncertainty in generated content. We integrated theseguardrails with an LLM fine-tuned for a text-to-text task, which involvesconverting both structured and unstructured data within adverse event reportsinto natural language. This method was applied to translate individual casesafety reports, demonstrating effective application in a pharmacovigilanceprocessing task. Our guardrail framework offers a set of tools with broadapplicability across various domains, ensuring LLMs can be safely used inhigh-risk situations by eliminating the occurrence of key errors, including thegeneration of incorrect pharmacovigilance-related terms, thus adhering tostringent regulatory and quality standards in medical safety-criticalenvironments.