AI-generated radiology flashcards miss 1% of critical errors, exceeding safety threshold in validation study

Multi-reader validation of a Gemini-based LLM pipeline for radiology board prep found 1.00% of flashcards had critical errors (95% CI 0.50–1.78%), exceeding the 0.3% safety threshold. Attending radiologists detected more errors than residents (OR 4.52). Automated feedback improv…

• Two-phase evaluation of 1,284 flashcards across 11 subspecialties by 20 readers (9 residents, 11 attendings); the false-negative rate for blocking errors was 1.00% (95% CI 0.50–1.78%).
• Fifth-stage feedback was associated with increased error detection (McNemar p=0.003) and lower technical accuracy/educational quality scores (both p<0.001), but the single-arm design prevents isolating useful cueing from anchoring.
• Key limitation: conditional safety estimates, low-reliability reference standard, rater-dependent patterns, and exploratory underpowered image pilot; results do not support fully automated deployment.

RadPigeon summaries are original and for information only. They are not clinical advice.