Physician and AI evaluators diverge when assessing LLM clinical case responses

Physician evaluations of LLM responses on real clinical cases showed substantial heterogeneity by seniority and setting, shifting model rankings. AI agents provided efficient but insufficiently nuanced assessments, unable to replace human judgment.

• Multicenter study with >400 physicians across 7 specialties and matched AI agents evaluating LLM responses to de-identified clinical cases.
• AI agents produced directionally aligned assessments and show promise as assistive triage tools to reduce human burden.
• AI evaluations did not capture the full nuance of human clinical judgment, limiting their standalone use.

RadPigeon summaries are original and for information only. They are not clinical advice.