LLM Extraction from Reports Enables Scalable AI Monitoring in ACR's Assess-AI Registry

In ACR's Assess-AI registry, LLM prompts for extracting findings from radiology reports achieved 0.985 and 0.997 agreement on development cohorts for ICH and PE, but independent validation is lacking.

• LLM prompts were developed for nine use cases, with high agreement on internal tuning cohorts (0.985 for intracranial hemorrhage, 0.997 for pulmonary embolism).
• No independent validation was performed; the reported agreement comes from the same cohorts used to refine prompts and reference labels.
• The workflow demonstrates feasibility of scalable, report-anchored AI performance monitoring, but clinical utility remains unestablished.

Automated summary

RadPigeon summaries are original and for information only. They are not clinical advice.