Three Methodologic Concerns About GPT-4o for Resident Report Drafting Feedback

While GPT-4o reliably identified errors in breast imaging drafts and gave helpful feedback, a comment highlights 3 methodological concerns: construct circularity, selection bias from Levenshtein filtering, and a drop in attending interrater agreement when GPT-4o joined the panel.

• The original study reported GPT-4o's accuracy in identifying reporting errors and its feedback rated helpful by residents and attendings.
• The commentary raises: (1) error categories were derived from the same data used to evaluate the model, creating circularity; (2) a Levenshtein distance filter likely excluded more challenging reports, biasing the sample; (3) attending interrater agreement decreased after GPT-4o was introduced, suggesting the model may have influenced human judgment.
• This is a commentary, not new data; the concerns are methodological and require further study before positioning the tool as a scalable educational adjunct.

RadPigeon summaries are original and for information only. They are not clinical advice.