Quality Rubric for AI-Generated Patient-Friendly Radiology Reports: Almost-Perfect Interrater Agreement

A rubric to assess artificial intelligence (AI)-generated patient-friendly radiology reports was developed. Lay & radiologist raters (n=3 each) had near-perfect agreement (α=0.87). Reports scored 1 on any attribute (except verbosity) were deemed unsafe.

• Safety rule: reports rated 1 on any core attribute (except verbosity) were deemed unsafe for distribution.
• Additional lay (n=19) and radiologist (n=12) raters had moderate (α=0.51) and substantial (α=0.65) interreader agreement; subjective decisions aligned with rule-based decisions in 91.2% and 95.8% of cases.
• In wider field testing (80 lay, 480 reports), ChatGPT-5 grading agreed moderately with reference standard (κ=0.44), but AI rule-based safety decisions matched reference-standard decisions in 88.1% of cases.

Automated summary

RadPigeon summaries are original and for information only. They are not clinical advice.