Abstract
Recent artificial intelligence (AI) systems have reached milestones in "grandchallenges" ranging from Go to protein-folding. The capability to retrievemedical knowledge, reason over it, and answer medical questions comparably tophysicians has long been viewed as one such grand challenge. Large language models (LLMs) have catalyzed significant progress in medicalquestion answering; Med-PaLM was the first model to exceed a "passing" score inUS Medical Licensing Examination (USMLE) style questions with a score of 67.2%on the MedQA dataset. However, this and other prior work suggested significantroom for improvement, especially when models' answers were compared toclinicians' answers. Here we present Med-PaLM 2, which bridges these gaps byleveraging a combination of base LLM improvements (PaLM 2), medical domainfinetuning, and prompting strategies including a novel ensemble refinementapproach. Med-PaLM 2 scored up to 86.5% on the MedQA dataset, improving upon Med-PaLMby over 19% and setting a new state-of-the-art. We also observed performanceapproaching or exceeding state-of-the-art across MedMCQA, PubMedQA, and MMLUclinical topics datasets. We performed detailed human evaluations on long-form questions along multipleaxes relevant to clinical applications. In pairwise comparative ranking of 1066consumer medical questions, physicians preferred Med-PaLM 2 answers to thoseproduced by physicians on eight of nine axes pertaining to clinical utility (p< 0.001). We also observed significant improvements compared to Med-PaLM onevery evaluation axis (p < 0.001) on newly introduced datasets of 240 long-form"adversarial" questions to probe LLM limitations. While further studies are necessary to validate the efficacy of these modelsin real-world settings, these results highlight rapid progress towardsphysician-level performance in medical question answering.