Sex- Stratified Benchmarking of Multimodal LLMs for Cardiopulmonary Findings on Chest Radiographs

Three commercial multimodal LLMs showed differing zero-shot operating profiles on chest radiographs. GPT-5.4 had highest discrimination (AUC-ROC 0.836-0.883) but sensitivity as low as 0.043, while Gemini 2.5 Pro favored sensitivity (up to 0.973). None are suitable for autonomous…

• In 4500 evaluations from the CheXpert dataset, GPT-5.4 achieved near-perfect specificity (0.977-0.997) at the cost of very low sensitivity for cardiomegaly, edema, and effusion.
• Sex-stratified analyses showed consistently higher AUC estimates for male patients with cardiomegaly and pulmonary edema, supporting subgroup-aware performance reporting.
• Claude Opus 4.5 showed the most balanced profile with sensitivity ranging from 0.396 to 0.876 and specificity from 0.461 to 0.863.

Automated summary

RadPigeon summaries are original and for information only. They are not clinical advice.