Radiologist-level Scaphoid Fracture Detection: Next Steps for Clinical Application
2021; Radiological Society of North America; Volume: 3; Issue: 4 Linguagem: Inglês
10.1148/ryai.2021210111
ISSN2638-6100
Autores Tópico(s)Radiology practices and education
ResumoHomeRadiology: Artificial IntelligenceVol. 3, No. 4 PreviousNext CommentaryFree AccessRadiologist-level Scaphoid Fracture Detection: Next Steps for Clinical ApplicationMatthew D. Li , Martin TorrianiMatthew D. Li , Martin TorrianiAuthor AffiliationsFrom the Division of Musculoskeletal Imaging and Intervention, Department of Radiology, Massachusetts General Hospital, Harvard Medical School, 55 Fruit St, Yawkey 6E, Boston, MA 02114.Address correspondence to M.D.L. (e-mail: [email protected]).Matthew D. Li Martin TorrianiPublished Online:Jun 23 2021https://doi.org/10.1148/ryai.2021210111MoreSectionsPDF ToolsImage ViewerAdd to favoritesCiteTrack CitationsPermissionsReprints ShareShare onFacebookTwitterLinked In See also the article by Hendrix et al in this issue.Matthew D. Li, MD, is a diagnostic radiology resident and former chief resident in the department of radiology at Massachusetts General Hospital and Harvard Medical School. His research focuses on the development of deep learning techniques for improving the evaluation of disease severity and change in medical imaging, with grant support from the Radiological Society of North America (RSNA) R&E Presidents Circle. Dr Li serves on the trainee editorial board of Radiology: Artificial Intelligence and the RSNA RadLex Steering Subcommittee.Download as PowerPointOpen in Image Viewer Martin Torriani, MD, MMSc, is a musculoskeletal radiologist with over 24 years of clinical experience. He is an associate professor of radiology at Harvard Medical School. Dr Torriani works at Massachusetts General Hospital focusing on clinical musculoskeletal imaging and research, as well as metabolic imaging. He is director of the NIH-funded MGH Metabolic Imaging Core, using advanced imaging to study human metabolism and developing artificial intelligence tools to enhance quantitative analyses.Download as PowerPointOpen in Image Viewer For radiologists and other clinicians involved in the care of patients with upper extremity trauma, the scaphoid is often scrutinized in detail on radiographs. Scaphoid fractures are often subtle, even radiographically occult in many cases, and a missed fracture has important clinical consequences, due to the risk of nonunion. A high-performance artificial intelligence (AI)–based system for automated detection of scaphoid fractures could be useful for improving the accuracy of radiographic diagnoses.In this issue of Radiology: Artificial Intelligence, Hendrix et al achieve this goal with radiologist-level performance, using a convolutional neural network–based pipeline trained to automatically segment scaphoid bones and detect scaphoid fractures on conventional radiographs of the hand, wrist, and scaphoid (1). The test-set performance of their AI model was higher than the average performance of 11 radiologists (area under the receiver operating characteristic curves of 0.87 vs 0.83). Although this difference was not statistically significant, the estimate of radiologist performance may be higher than what would be expected in routine clinical practice. In this study, the participating radiologists examined 190 test-set radiographs specifically for scaphoid fractures. However, in a real-world clinical workflow, the scaphoid is one of many anatomic structures that are assessed on hand and wrist radiographs. Further, the clinical history frequently does not direct attention to the scaphoid, and a growing list of radiology studies to be read may decrease the amount of time that the radiologist can spend inspecting the study. Thus, the AI model developed by Hendrix et al (1) has the potential to help improve the performance of radiologists and other clinicians involved in interpreting these studies. This work is a substantial improvement over existing published AI models, achieving high performance by training on a large number of radiographs from two institutions, including 1039 for the scaphoid segmentation model and 3000 for the fracture detection model. Previously published scaphoid detection AI models used relatively small datasets for training and validation, ranging from 200 to 290 images, and also required manual cropping of the images for the scaphoid before passing images to convolutional neural networks (2,3), a step automated using U-Net segmentation in this study.Clearly, the automated scaphoid fracture detection model created by Hendrix et al performs well, but what will be the next steps to make such a tool clinically useful? This is the quintessential question we ask as an increasing number of published works applying AI models show expert-level or even higher performance for interpretative tasks in medical imaging. These next steps may involve: (a) building the AI inference pipeline into the picture archiving and communication system, radiology information system, and/or electronic health record, (b) designing how the AI-derived information will be presented and used, and (c) implementing quality control and quality improvement processes. For the first of these steps, there are technical issues related to the routing of Digital Imaging and Communications in Medicine (DICOM) files and image series and view selection (eg, the model in this study only accepts anteroposterior or posteroanterior radiographs as inputs, which the authors acknowledge as a limitation). However, a more fundamental question remains, how does a scaphoid fracture detection AI model fit into a more generally applied AI inference platform? Scaphoid fractures, while the most common carpal bone fracture, are only seen in a small minority of hand or wrist radiographs. Depending on the practice setting, radiographs will often have no fracture at all, let alone a scaphoid fracture. Thus, routing DICOM images to a scaphoid fracture detector, a distal radius fracture detector, a trapezius fracture detector, and so on, would be highly inefficient. One potential solution would be to create an inference pipeline where the image of interest is first routed to a general wrist fracture detector, like one previously published (4), calibrated for high sensitivity. If the general fracture detector AI detects a fracture, then the DICOM could be routed to fracture detector AIs for specific bones, including the scaphoid. Many AI models in medical imaging have the similar problem of having a narrow use case. A system of rule-based sequential AI inference may be a potential way to deal with this technical issue in AI deployment.In addition to such technical issues, the AI model must fit into the clinical workflow to be useful. If an AI scaphoid fracture detection system detects a fracture, who receives that information and how? If used as a decision-support tool for radiologists or other clinicians like emergency physicians and orthopedic surgeons, interpretability of the AI output becomes important (5). Hendrix et al (1) used a class activation saliency map approach to attempt to localize the site of fractures, which qualitatively identified fracture sites. However, emerging quantitative research studies on saliency maps have found that these AI visualization techniques can be unreliable for localization of abnormalities in medical imaging (6,7). For the test set, the 11 radiologists in this study also provided confidence scores for their assessments of scaphoid fractures. In some cases, the AI model assigned a high probability for a scaphoid fracture, while the radiologists assigned very low probabilities, and vice versa. The example class activation maps for such cases do not help to explain this disparity, highlighting the black box nature of the model. Further research into the interpretability of such visualizations of AI outputs is warranted.Another consideration for the clinical workflow is how to design the system in a way to improve clinical outcomes. A review of medical negligence legal cases related to missed scaphoid fractures found that scaphoid fractures were most often missed because the frontline clinician did not consider the possibility of a scaphoid fracture, and in 25% of cases of missed scaphoid fractures, radiographs were not obtained at all (8). Thus, although this AI tool may help improve scaphoid fracture detection on radiographs, if a radiograph is not obtained in the first place, that will limit the potential benefit of such a system. Furthermore, even if the radiologist or model performs perfectly, radiography also has limited sensitivity for detection of scaphoid fractures (eg, 66%, using multidetector CT as a reference standard [9]), as initial radiographs can be normal. Hendrix et al (1) trained their AI fracture detection model using labels derived from the radiology reports; thus, the model was not trained to detect radiographically occult fractures. Future work may benefit from using a different reference standard such as CT or MRI for training, to see if an AI model can detect occult fractures, a task that would have more added value to the clinical workflow.If this scaphoid fracture detection AI model is eventually deployed in clinical practice, quality control will also be important to manage risk and assess the actual efficacy of the model in a nontest environment. The performance of the model must be tracked and reevaluated, as its performance metrics will likely be different when presented new data. Analysis of false-positive and false-negative edge cases will be important to identify reasons for failure of the AI system and its consequences. In the case of scaphoid fractures, a false-positive diagnosis can lead to unnecessary casting and have economic impacts for both the patient and the health care system. However, this risk may be mitigated if CT or MRI can be performed in a timely manner to verify or disprove the diagnosis. However, false-negative cases, where the AI model lulls the user into a false sense of security that there is no fracture, are more concerning. This asymmetric risk must be considered with caution. Systematic approaches are being developed to help deal with such issues, which will be useful for clinical departments deploying and evaluating such AI tools (10).Hendrix et al (1) have trained and tested a promising model for automated radiographic scaphoid fracture detection, with radiologist-level performance that may potentially exceed real-world performance. This AI model could be helpful to augment or ensure the quality of interpretation of radiographs for scaphoid fractures by radiologists and other clinicians. We look forward to the next steps to bring such a tool into clinical practice for the benefit of our patients.Disclosures of Conflicts of Interest: M.D.L. Activities related to the present article: member of the Radiology: Artificial Intelligence trainee editorial board. Activities not related to the present article: disclosed no relevant relationships. Other relationships: disclosed no relevant relationships.M.T. disclosed no relevant relationships.References1. Hendrix N, Scholten E, Vernhout B, et al. Development and Validation of a Convolutional Neural Network for Automated Detection of Scaphoid Fractures on Conventional Radiographs. Radiol Artif Intell 2021;3(4):e200260. Link, Google Scholar2. Langerhuizen DWG, Bulstra AEJ, Janssen SJ, et al. Is Deep Learning On Par with Human Observers for Detection of Radiographically Visible and Occult Fractures of the Scaphoid?. Clin Orthop Relat Res 2020;478(11):2653–2659. Crossref, Medline, Google Scholar3. Ozkaya E, Topal FE, Bulut T, Gursoy M, Ozuysal M, Karakaya Z. Evaluation of an artificial intelligence system for diagnosing scaphoid fracture on direct radiography. Eur J Trauma Emerg Surg 2020.10.1007/s00068-020-01468-0. Published online August 30, 2020. Crossref, Medline, Google Scholar4. Lindsey R, Daluiski A, Chopra S, et al. Deep neural network improves fracture detection by clinicians. Proc Natl Acad Sci U S A 2018;115(45):11591–11596. Crossref, Medline, Google Scholar5. Reyes M, Meier R, Pereira S, et al. On the Interpretability of Artificial Intelligence in Radiology: Challenges and Opportunities. Radiol Artif Intell 2020;2(3):e190043. Link, Google Scholar6. Arun N, Gaw N, Singh P, et al. Assessing the (un)trustworthiness of saliency maps for localizing abnormalities in medical imaging. [preprint] medRxiv 2020; 2020.07.28.20163899. Google Scholar7. Saporta A, Gui X, Agrawal A, et al. Deep learning saliency maps do not accurately highlight diagnostically relevant regions for medical image interpretation. [preprint] medRxiv 2021; 2021.02.28.21252634. Google Scholar8. Jamjoom BA, Davis TRC. Why scaphoid fractures are missed. A review of 52 medical negligence cases. Injury 2019;50(7):1306–1308. Crossref, Medline, Google Scholar9. Balci A, Basara I, Çekdemir EY, et al. Wrist fractures: sensitivity of radiography, prevalence, and patterns in MDCT. Emerg Radiol 2015;22(3):251–256. Crossref, Medline, Google Scholar10. Mahajan V, Venugopal VK, Murugavel M, Mahajan H. The Algorithmic Audit: Working with Vendors to Validate Radiology-AI Algorithms-How We Do It. Acad Radiol 2020;27(1):132–135. Crossref, Medline, Google ScholarArticle HistoryReceived: Apr 24 2021Revision requested: Apr 24 2021Revision received: Apr 29 2021Accepted: Apr 29 2021Published online: June 23 2021 FiguresReferencesRelatedDetailsAccompanying This ArticleDevelopment and Validation of a Convolutional Neural Network for Automated Detection of Scaphoid Fractures on Conventional RadiographsApr 28 2021Radiology: Artificial IntelligenceRecommended Articles Development and Validation of a Convolutional Neural Network for Automated Detection of Scaphoid Fractures on Conventional RadiographsRadiology: Artificial Intelligence2021Volume: 3Issue: 4Convolutional Neural Networks for Automated Fracture Detection and Localization on Wrist RadiographsRadiology: Artificial Intelligence2019Volume: 1Issue: 1Additional Value of Dual-Energy CT for Patients with Wrist TraumaRadiology2020Volume: 296Issue: 3pp. 603-604Dual-Energy CT for Suspected Radiographically Negative Wrist Fractures: A Prospective Diagnostic Test Accuracy StudyRadiology2020Volume: 296Issue: 3pp. 596-602Assessment of an AI Aid in Detection of Adult Appendicular Skeletal Fractures by Emergency Physicians and Radiologists: A Multicenter Cross-sectional Diagnostic StudyRadiology2021Volume: 300Issue: 1pp. 120-129See More RSNA Education Exhibits Carpal Injury from a Fall on the Outstretched HandDigital Posters2019Many Hands Make Light Workâ99mTc-MDP Bone SPECT/CT Scintigraphy of the Hands and Wrists: Pearls and PitfallsDigital Posters2018Carpal Scaphoid Fractures Radiologist Role and Surgical PerspectivesDigital Posters2018 RSNA Case Collection Right dorsal triquetral fracture with hook of hamate fractureRSNA Case Collection2020Lunate DislocationRSNA Case Collection2020Scaphoid FractureRSNA Case Collection2020 Vol. 3, No. 4 Metrics Downloaded 414 times Altmetric Score PDF download
Referência(s)