Recommendations for the development and use of imaging test sets to investigate the test performance of artificial intelligence in health screening.
Name:
Publisher version
View Source
Access full-text PDFOpen Access
View Source
Check access options
Check access options
Author
Chalkidou, AnastasiaShokraneh, Farhad
Kijauskaite, Goda
Taylor-Phillips, Sian
Halligan, Steve
Wilkinson, Louise
Glocker, Ben
Garrett, Peter
Denniston, Alastair K
Mackie, Anne
Seedat, Farah
Publication date
2022-12Subject
Public health. Health statistics. Occupational health. Health educationRadiology
Ophthalmology
Metadata
Show full item recordAbstract
Rigorous evaluation of artificial intelligence (AI) systems for image classification is essential before deployment into health-care settings, such as screening programmes, so that adoption is effective and safe. A key step in the evaluation process is the external validation of diagnostic performance using a test set of images. We conducted a rapid literature review on methods to develop test sets, published from 2012 to 2020, in English. Using thematic analysis, we mapped themes and coded the principles using the Population, Intervention, and Comparator or Reference standard, Outcome, and Study design framework. A group of screening and AI experts assessed the evidence-based principles for completeness and provided further considerations. From the final 15 principles recommended here, five affect population, one intervention, two comparator, one reference standard, and one both reference standard and comparator. Finally, four are appliable to outcome and one to study design. Principles from the literature were useful to address biases from AI; however, they did not account for screening specific biases, which we now incorporate. The principles set out here should be used to support the development and use of test sets for studies that assess the accuracy of AI within screening programmes, to ensure they are fit for purpose and minimise bias.Citation
Chalkidou A, Shokraneh F, Kijauskaite G, Taylor-Phillips S, Halligan S, Wilkinson L, Glocker B, Garrett P, Denniston AK, Mackie A, Seedat F. Recommendations for the development and use of imaging test sets to investigate the test performance of artificial intelligence in health screening. Lancet Digit Health. 2022 Dec;4(12):e899-e905. doi: 10.1016/S2589-7500(22)00186-8Type
ArticleAdditional Links
https://www.sciencedirect.com/journal/the-lancet-digital-healthPMID
36427951Journal
The Lancet Digital HealthPublisher
Elsevierae974a485f413a2113503eed53cd6c53
10.1016/S2589-7500(22)00186-8
Scopus Count
Collections
Related items
Showing items related by title, author, creator and subject.
-
Role of the Guidelines Evidence Specialist in the streamlining of Guidelines at a large acute NHS TrustHeer, Mandeep; Heer, Mandeep; Heer, Mandeep; CEBIS; CEBIS (University Hospitals Coventry and Warwickshire NHS Trust, 2023-11)Role of the Guidelines Evidence Specialist in the streamlining of Guidelines at a large acute NHS Trust
-
Faculty development: clinical dermatology for medical secretaries and administrative staffAgrawal, Rishi; Browne, Rachel; Baldwin, Nicola; Scott, H.; Tso, Simon; Agrawal, R.; Browne, R.; Baldwin, N.; Scott, H.; Tso, S.; et al. (Oxford University Press, 2020-06)A study investigating the potential benefits of specialty-specific clinical inductions for medical secretaries and administrative staff.
-
Mapping inpatient care pathways for patients with COPD: an observational study using routinely collected electronic hospital record data.Evison, Felicity; Cooper, Rachel; Gallier, Suzy; Missier, Paolo; Sayer, Avan A; Sapey, Elizabeth; Witham, Miles D; Evison, Felicity; Gallier, Suzy; Research and Development; et al. (European Respiratory Society, 2023-10-16)Introduction: Respiratory specialist ward care is associated with better outcomes for patients with COPD exacerbations. We assessed patient pathways and associated factors for people admitted to hospital with COPD exacerbations. Methods: We analysed routinely collected electronic health data for patients admitted with COPD exacerbation in 2018 to Queen Elizabeth Hospital, Birmingham, UK. We extracted data on demographics, deprivation index, Elixhauser comorbidities, ward moves, length of stay, and in-hospital and 1-year mortality. We compared care pathways with recommended care pathways (transition from initial assessment area to respiratory wards or discharge). We used Markov state transition models to derive probabilities of following recommended pathways for patient subgroups. Results: Of 42 555 patients with unplanned admissions during 2018, 571 patients were admitted at least once with an exacerbation of COPD. The mean±sd age was 51±11 years; 313 (55%) were women, 337 (59%) lived in the most deprived neighbourhoods and 45 (9%) were from non-white ethnic backgrounds. 428 (75.0%) had ≥4 comorbidities. Age >70 years was associated with higher in-hospital and 1-year mortality, more places of care (wards) and longer length of stay; having ≥4 comorbidities was associated with higher mortality and longer length of stay. Older age was associated with a significantly lower probability of following a recommended pathway (>70 years: 0.514, 95% CI 0.458-0.571; ≤70 years: 0.636, 95% CI 0.572-0.696; p=0.004). Conclusions: Only older age was associated with a lower chance of following recommended hospital pathways of care. Such analyses could help refine appropriate care pathways for patients with COPD exacerbations.