AI Efficacy as a Function of Trainee Interpreter Proficiency: Lessons from a Randomized Controlled Trial

BACKGROUND AND PURPOSE: Recently, AI tools have been deployed with increasing speed in educational and clinical settings. However, the use of AI by trainees across different levels of experience has not been well studied. This study investigates the impact of AI assistance on diagnostic accuracy for intracranial hemorrhage (ICH) and large vessel occlusion (LVO) by medical students (MS) and resident trainees (RT).

MATERIALS AND METHODS: This prospective study was conducted between March 2023 and October 2023. MS and RT were asked to identify ICH and LVO in 100 non-contrast head CTs and 100 head CTAs, respectively. One group received diagnostic aid simulating AI for ICH only (n = 26), the other for LVO only (n = 28). Primary outcomes included accuracy, sensitivity, and specificity for ICH / LVO detection without and with aid. Study interpretation time was a secondary outcome. Individual responses were pooled and analyzed with chi-square; differences in continuous variables were assessed with ANOVA.

RESULTS: 48 participants completed the study, generating 10,779 ICH or LVO interpretations. With diagnostic aid, MS accuracy improved 11.0 points (P < .001) and RT accuracy showed no significant change. ICH interpretation time increased with diagnostic aid for both groups (P < .001) while LVO interpretation time decreased for MS (P < .001). Despite worse performance in detection of the smallest vs. the largest hemorrhages at baseline, MS were not more likely to accept a true positive AI result for these more difficult tasks. Both groups were considerably less accurate when disagreeing with the AI or when supplied with an incorrect AI result.

CONCLUSIONS: This study demonstrated greater improvement in diagnostic accuracy with AI for MS compared to RT. However, MS were less likely than RT to overrule incorrect AI interpretations and were less accurate, even with diagnostic aid, than the AI was by itself.

ABBREVIATIONS: ICH = intracranial hemorrhage; LVO = large vessel occlusion; MS = medical students; RT = resident trainees.