Enhancing diagnostic precision : utilising a large language model to extract U scores from thyroid sonography reports
Watts, Emma Pournik, Omid Allington, Rose Ding, Xuefei Boelaert, Kristien Sharma, Neil Ghalichi, Leila Arvanitis, Theodoros N
Watts, Emma
Pournik, Omid
Allington, Rose
Ding, Xuefei
Boelaert, Kristien
Sharma, Neil
Ghalichi, Leila
Arvanitis, Theodoros N
Affiliation
University of Birmingham; University Hospitals Birmingham
Other Contributors
Publication date
2025-06-26
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
This study evaluates the performance of ChatGPT-4, a Large Language Model (LLM), in automatically extracting U scores from free-text thyroid ultrasound reports collected from University Hospitals Birmingham (UHB), UK, between 2014 and 2024. The LLM was provided with guidelines on the U classification system and extracted U scores independently from 14,248 de-identified reports, without access to human-assigned scores. The LLM-extracted scores were compared to initial clinician-assigned and refined U scores provided by expert reviewers. The LLM achieved 97.7% agreement with refined human U scores, successfully identifying the highest U score in 98.1% of reports with multiple nodules. Most discrepancies (2.5%) were linked to ambiguous descriptions, multi-nodule reports, and cases with human-documented uncertainty. While the results demonstrate the potential for LLMs to improve reporting consistency and reduce manual workload, ethical and governance challenges such as transparency, privacy, and bias must be addressed before routine clinical deployment. Embedding LLMs into reporting workflows, such as Online Analytical Processing (OLAP) tools, could further enhance reporting quality and consistency.
Citation
Watts E, Pournik O, Allington R, Ding X, Boelaert K, Sharma N, Ghalichi L, Arvanitis TN. Enhancing Diagnostic Precision: Utilising a Large Language Model to Extract U Scores from Thyroid Sonography Reports. Stud Health Technol Inform. 2025 Jun 26;328:56-60. doi: 10.3233/SHTI250672.
Type
Article