Large language models approach expert-level clinical knowledge and reasoning in ophthalmology : a head-to-head cross-sectional study
Thirunavukarasu, Arun James Mahmood, Shathar Malem, Andrew Foster, William Paul Sanghera, Rohan Hassan, Refaat Zhou, Sean Wong, Shiao Wei Wong, Yee Ling
Thirunavukarasu, Arun James
Mahmood, Shathar
Malem, Andrew
Foster, William Paul
Sanghera, Rohan
Hassan, Refaat
Zhou, Sean
Wong, Shiao Wei
Wong, Yee Ling
Citations
Altmetric:
Affiliation
University of Cambridge School of Clinical Medicine; University of Oxford; Cleveland Clinic Abu Dhabi; Sandwell and West Birmingham NHS Trust; et al.
Other Contributors
Publication date
2024-04-17
Subject
Collections
Research Projects
Organizational Units
Journal Issue
Abstract
Large language models (LLMs) underlie remarkable recent advanced in natural language processing, and they are beginning to be applied in clinical contexts. We aimed to evaluate the clinical potential of state-of-the-art LLMs in ophthalmology using a more robust benchmark than raw examination scores. We trialled GPT-3.5 and GPT-4 on 347 ophthalmology questions before GPT-3.5, GPT-4, PaLM 2, LLaMA, expert ophthalmologists, and doctors in training were trialled on a mock examination of 87 questions. Performance was analysed with respect to question subject and type (first order recall and higher order reasoning). Masked ophthalmologists graded the accuracy, relevance, and overall preference of GPT-3.5 and GPT-4 responses to the same questions. The performance of GPT-4 (69%) was superior to GPT-3.5 (48%), LLaMA (32%), and PaLM 2 (56%). GPT-4 compared favourably with expert ophthalmologists (median 76%, range 64-90%), ophthalmology trainees (median 59%, range 57-63%), and unspecialised junior doctors (median 43%, range 41-44%). Low agreement between LLMs and doctors reflected idiosyncratic differences in knowledge and reasoning with overall consistency across subjects and types (p>0.05). All ophthalmologists preferred GPT-4 responses over GPT-3.5 and rated the accuracy and relevance of GPT-4 as higher (p<0.05). LLMs are approaching expert-level knowledge and reasoning skills in ophthalmology. In view of the comparable or superior performance to trainee-grade ophthalmologists and unspecialised junior doctors, state-of-the-art LLMs such as GPT-4 may provide useful medical advice and assistance where access to expert ophthalmologists is limited. Clinical benchmarks provide useful assays of LLM capabilities in healthcare before clinical trials can be designed and conducted.
Citation
Thirunavukarasu AJ, Mahmood S, Malem A, Foster WP, Sanghera R, Hassan R, Zhou S, Wong SW, Wong YL, Chong YJ, Shakeel A, Chang YH, Tan BKJ, Jain N, Tan TF, Rauz S, Ting DSW, Ting DSJ. Large language models approach expert-level clinical knowledge and reasoning in ophthalmology: A head-to-head cross-sectional study. PLOS Digit Health. 2024 Apr 17;3(4):e0000341. doi: 10.1371/journal.pdig.0000341
Type
Article