Performance of an artificial intelligence tool with real-time clinical workflow integration – Detection of intracranial hemorrhage and pulmonary embolism
2021; Elsevier BV; Volume: 83; Linguagem: Inglês
10.1016/j.ejmp.2021.03.015
ISSN1724-191X
AutoresNico Buls, Nina Watté, Koenraad Nieboer, Bart Ilsen, Johan De Mey,
Tópico(s)Radiation Dose and Imaging
Resumo•Intra-cranial hemorrhage and pulmonary embolism are life-threatening pathologies.•CT imaging is essential to confirm diagnosis.•In a real-time clinical setting AI shows the potential to rule out ICH and PE.•The positive predictive value of AI remains moderate.•AI has the potential to assist radiologists and serve as real-time clinical adjunct. IntroductionAcute pathologies require early detection with prompt communication of critical findings to ensure adequate clinical management. Intra-cranial hemorrhage (ICH) and pulmonary embolism (PE) are two of such frequent life-threatening pathologies, with significant morbidity and mortality, where misdiagnosis can lead to adverse outcome [1van Asch C.J. Luitse M.J. Rinkel G.J. van der Tweel I. Algra A. Klijn C.J. Incidence, case fatality, and functional outcome of intracerebral haemorrhage over time, according to age, sex, and ethnic origin: a systematic review and meta-analysis.Lancet Neurol. 2010; 9: 167-176https://doi.org/10.1016/S1474-4422(09)70340-0Abstract Full Text Full Text PDF PubMed Scopus (1595) Google Scholar, 2Heit J.J. Iv M. Wintermark M. Imaging of intracranial hemorrhage.J Stroke. 2017; 19: 11-27https://doi.org/10.5853/jos.2016.00563Crossref PubMed Scopus (99) Google Scholar, 3Morales H. Pitfalls in the imaging interpretation of intracranial hemorrhage.Semin Ultrasound CT MR. 2018; 39: 457-468https://doi.org/10.1053/j.sult.2018.07.001Crossref PubMed Scopus (6) Google Scholar]. A non-contrast head CT scan is essential to confirm diagnosis and risk stratification of ICH, while contrast enhanced Computed Tomography Pulmonary Angiography (CTPA) is a standard scan for detecting and locating PE [3Morales H. Pitfalls in the imaging interpretation of intracranial hemorrhage.Semin Ultrasound CT MR. 2018; 39: 457-468https://doi.org/10.1053/j.sult.2018.07.001Crossref PubMed Scopus (6) Google Scholar, 4Estrada-Y-Martin R.M. Oldham S.A. CTPA as the gold standard for the diagnosis of pulmonary embolism.Int J Comput Assist Radiol Surg. 2011; 6: 557-563https://doi.org/10.1007/s11548-010-0526-4Crossref PubMed Scopus (40) Google Scholar].Advances in CT technology have led to the improvement of image quality and reduction of radiation dose, which allows the diagnosis of more subtle lesions. However, the increasing volume in number of examinations and images per examination, can have a disproportionate effect on the radiologist' work stream. McDonald et al., calculated in their study on the influence of technological advancements of cross-sectional imaging on the radiology workflow, that a radiologist analyses an average of one image every three seconds [[5]McDonald R.J. Schwartz K.M. Eckel L.J. et al.The effects of changes in utilization and technological advancements of cross-sectional imaging on radiologist workload.Acad Radiol. 2015; 22: 1191-1198https://doi.org/10.1016/j.acra.2015.05.007Abstract Full Text Full Text PDF PubMed Scopus (149) Google Scholar]. This time-intensive encumbrance on the practicing radiologist, can accrue an increase in false negative results and misdiagnosis [6Grob D. Smit E. Oostveen L.J. et al.Image quality of iodine maps for pulmonary embolism: A comparison of subtraction CT and dual-energy CT [published online ahead of print, 2019 Mar 12].AJR Am J Roentgenol. 2019; 1–7https://doi.org/10.2214/AJR.18.20786Crossref Scopus (7) Google Scholar, 7Brady A.P. Error and discrepancy in radiology: inevitable or avoidable?.Insights Imaging. 2017; 8: 171-182https://doi.org/10.1007/s13244-016-0534-1Crossref PubMed Scopus (193) Google Scholar, 8Sokolovskaya E. Shinde T. Ruchman R.B. et al.The effect of faster reporting speed for imaging studies on the number of misses and interpretation errors: A pilot study.J Am Coll Radiol. 2015; 12: 683-688https://doi.org/10.1016/j.jacr.2015.03.040Abstract Full Text Full Text PDF PubMed Scopus (53) Google Scholar]. Real-time double reading by a peer is often done, which has been proved to aid in lowering the prevalence of misdiagnosis, however it is very labor-intensive. In addition, retrospective peer reviewing of cases does not immediate improve the patient's clinical outcome, especially not in an acute setting [9Geijer H. Geijer M. Added value of double reading in diagnostic radiology, a systematic review.Insights Imaging. 2018; 9: 287-301https://doi.org/10.1007/s13244-018-0599-0Crossref PubMed Scopus (39) Google Scholar, 10Muroff L.R. Berlin L. Speed versus interpretation accuracy: Current thoughts and literature review.AJR Am J Roentgenol. 2019; 213: 490-492https://doi.org/10.2214/AJR.19.21290Crossref PubMed Scopus (8) Google Scholar, 11Babiarz L.S. Yousem D.M. Quality control in neuroradiology: discrepancies in image interpretation among academic neuroradiologists.AJNR Am J Neuroradiol. 2012; 33: 37-42https://doi.org/10.3174/ajnr.A2704Crossref PubMed Scopus (29) Google Scholar].Given the potential adverse outcome in case of misdiagnosis of ICH or PE, the increasing radiology workload, the constant development of new advanced computed tomography techniques and nowadays pandemics that effect our health care system, artificial intelligence (AI) technologies can assist radiologists and serve as a real-time clinical adjunct to diagnose ICH and PE. Using convolutional neural networks (CNN) based on deep learning, AI algorithms are becoming accessible, which can detect those life-threatening lesions [12Arbabshirani M.R. Fornwalt B.K. Mongelluzzo G.J. et al.Advanced machine learning in action: identification of intracranial hemorrhage on computed tomography scans of the head with clinical workflow integration.NPJ Digit Med. 2018; 1 (Published 2018 Apr 4): 9https://doi.org/10.1038/s41746-017-0015-zCrossref PubMed Scopus (183) Google Scholar, 13Prevedello L.M. Erdal B.S. Ryu J.L. et al.Automated critical test findings identification and online notification system using artificial intelligence in imaging.Radiology. 2017; 285: 923-931https://doi.org/10.1148/radiol.2017162664Crossref PubMed Scopus (141) Google Scholar, 14Chang P.D. Kuoy E. Grinband J. et al.Hybrid 3D/2D convolutional neural network for hemorrhage evaluation on head CT.AJNR Am J Neuroradiol. 2018; 39: 1609-1616https://doi.org/10.3174/ajnr.A5742Crossref PubMed Scopus (121) Google Scholar]. AI technologies have multiple potential roles such as quality assurance and productivity enhancement. However, certain roles within specific pathologies have not yet been fully investigated. Implementing an AI tool during a real-time radiology work stream, has the potential to react earlier and/or even notice lesions that can be easily overlooked by a radiologist [15Chilamkurthy S. Ghosh R. Tanamala S. et al.Deep learning algorithms for detection of critical findings in head CT scans: a retrospective study.Lancet. 2018; 392: 2388-2396https://doi.org/10.1016/S0140-6736(18)31645-3Abstract Full Text Full Text PDF PubMed Scopus (360) Google Scholar, 13Prevedello L.M. Erdal B.S. Ryu J.L. et al.Automated critical test findings identification and online notification system using artificial intelligence in imaging.Radiology. 2017; 285: 923-931https://doi.org/10.1148/radiol.2017162664Crossref PubMed Scopus (141) Google Scholar, 16Paiva O.A. Prevedello L.M. The potential impact of artificial intelligence in radiology.Radiol Bras. 2017; 50: V-VIhttps://doi.org/10.1590/0100-3984.2017.50.5e1Crossref PubMed Scopus (15) Google Scholar, 17Ojeda P, Zawaideh M, Mossa-Basha M, et al. The utility of deep learning: evaluation of a convolutional neural network for detection of intracranial bleeds on non-contrast head computed tomography studies. SPIE Medical Imaging, 2019, Proceedings Volume 10949, Medical Imaging 2019: Image Processing; 109493J.Google Scholar, 18Weikert T. Winkel D.J. Bremerich J. et al.Automated detection of pulmonary embolism in CT pulmonary angiograms using an AI-powered algorithm.Eur Radiol. 2020; 30: 6545-6553https://doi.org/10.1007/s00330-020-06998-0Crossref PubMed Scopus (27) Google Scholar].Much research in recent years has focused on such AI solutions, which has indicated a sensitivity of 0.95, specificity of 0.99, negative predictive value (NPV) of 0.98 and positive predictive value (PPV) of 0.98 with an overall accuracy of 0.98 for ICH detection. Rao et al. applied a AI solution to negative-by-report ICH cases. They found a false-negative rate of radiologists for ICH detection at 1.6%, and thus the technology could serve by minimizing false negatives [[19]Rao B. Zohrabian V. Cedeno P. Saha A. Pahade J. Davis M.A. Utility of artificial intelligence tool as a prospective radiology peer reviewer - detection of unreported intracranial hemorrhage [published online ahead of print, 2020 Feb 24].Acad Radiol. 2020; S1076–6332: 30084-30092https://doi.org/10.1016/j.acra.2020.01.035Abstract Full Text Full Text PDF Scopus (22) Google Scholar]. Weikert et al. found a high degree of diagnostic accuracy for PE detection on CTPAs, and a balanced sensitivity and specificity of 0.93 and 0.96 [[18]Weikert T. Winkel D.J. Bremerich J. et al.Automated detection of pulmonary embolism in CT pulmonary angiograms using an AI-powered algorithm.Eur Radiol. 2020; 30: 6545-6553https://doi.org/10.1007/s00330-020-06998-0Crossref PubMed Scopus (27) Google Scholar]. Also with PE, sensitivity, specificity, positive and negative predictive values and accuracy compared with gold standard senior radiologists were reported, 0.85, 0.97, 0.85, 0.97 and 0.95 respectively [[18]Weikert T. Winkel D.J. Bremerich J. et al.Automated detection of pulmonary embolism in CT pulmonary angiograms using an AI-powered algorithm.Eur Radiol. 2020; 30: 6545-6553https://doi.org/10.1007/s00330-020-06998-0Crossref PubMed Scopus (27) Google Scholar].The purpose of this study was to assess the performance of a commercially available AI tool as a second reader in detecting ICH and PE in a diverse clinical setting (e.g., emergency, routine, inpatient, and outpatient) in real-time assessment, by evaluation of the number processed studies by the AI tool and calculation of the diagnostic performance respectively.Materials and methodsThis retrospective study was conducted with approval by our local institutional medical ethics committee with a waiver of informed consent. Our study was bipartite and focused on two acute pathologies: Intra-cranial hemorrhage (ICH) and pulmonary embolism (PE). Each subdivision consisted of 4 stages: (1) Dataset collection; (2) Image data processing by an automated AI tool; (3) Quality control and discrepancy revision by registered radiologists with certificate of added qualification in neuroradiology or thorax radiology; and (4) diagnostic performance analysis.Dataset collectionRandom case collection was performed from a consecutive database of patients referred to our radiology department for an non-contrast head CT or CTPA, blinded to clinical data regarding antecedents, diagnose, therapy or outcome. Both for the brain and lung study, patients under the age of 18 were excluded. CT exams were pseudo-anonymized, retaining solely an identification code to link each report to its respective study. Control CTs were excluded, resulting in eliminating duplicate exams and a final cohort of unique patients and scans.A total of 500 consecutive non-contrast CT exams of the head performed over 31 days from September 1, 2019 until October 1, 2019, were included. This consecutive case collection varies considerably in terms of neurologic pathology signs such as hemorrhage, mass effect, hydrocephalus, suspected acute infarct, encephalomalacia or no evidence of intracranial disease. Cases also differ markedly in hemorrhage age and attenuation on CT (respectively hypo-, iso- and hyperattenuating), hemorrhage size and location (epidural, subdural, subarachnoid and intraparenchymal). Scans with movement artifacts, sloped and postoperative studies remained included to represent standard practice.Secondly, we considered 500 consecutive CTPA scans performed between July 1, 2019 and February 1,2020. This data set consists of pulmonary emboli (central, segmental and subsegmental), common diseases such as interstitial pneumonias, acute respiratory distress syndrome, sarcoidosis, lymphangitic carcinomatosis, cardiogenic pulmonary edema and normal findings. Scans containing moderate breathing or beam hardening artifacts, were included as well to represent routine radiology practice.CT examinations were performed on one of our four different scanners, of which one used dual-energy technology (DECT). Table 1 specifies information with reference to the utilized scanner vendor, model, tube voltage, single collimation width, reconstruction slice thickness, reconstruction kernel and radiation doses. Data from all axial, coronal and sagittal planes, available from the picture archiving and communication system (PACS), were utilized. In addition, for the DECT scanner, material specific pulmonary iodine maps, an alternative for depicting perfusion defects in PE, were also available for the expert reviewers.Table 1Scanner models, scan- and reconstruction parameters and radiation doses used for the two indications. Doses represent median values with 95% confidence intervals between brackets.Scanner modelNumber of casesTube voltage (kV)Single collimation width (mm)Reconstructed slice thickness (mm)Reconstruction methodReconstruction kernelCTDIvol (mGy)DLP (mGy.cm)Intracranial hemorrhage (ICH)500GE Revolution212 (42%)DECT0.6250.625DLIR-MStandard35.8 (35.5–37.2)724 (698–751)GE Discovery 750HD173 (35%)1200.6250.625ASiR 30%Soft38.4 (37.5–39.4)756 (725–787)Philips iCT109 (22%)1000.6250.8iDose3UB (standard)29.9 (29.3–30.6)598 (581–614)Siemens Somatom AS406 (1%)1200.60.6FBPH31s60.81006 (990–1023)Pulmonary embolism (PE)500GE Revolution282 (56%)DECT0.6250.625ASiR-V 70%Standard6.4 (5.9–6.8)229 (211–248)GE Discovery 750HD203 (41%)100–1200.6250.625ASiR 30%Detail11.8 (10.8–12.8)441 (405–478)Philips iCT15 (3%)100–1200.6250.9iDose3B (standard)4.1 (3.8–4.4)174 (163–183) Open table in a new tab Image data processing by AI toolA commercially available, FDA-and CE-cleared (European Medical Devices Directive 93/42/EEC M5) AI tool, based on convolutional neural networks (Aidoc version 1.3, Tel Aviv, Israel) was implemented in our radiological workflow. The algorithm was trained and tested by a dataset that included approximately 50,000 non-contrast head CT studies for the detection of ICH [[17]Ojeda P, Zawaideh M, Mossa-Basha M, et al. The utility of deep learning: evaluation of a convolutional neural network for detection of intracranial bleeds on non-contrast head computed tomography studies. SPIE Medical Imaging, 2019, Proceedings Volume 10949, Medical Imaging 2019: Image Processing; 109493J.Google Scholar] and 28,000 CTPA studies for the detection of PE [[18]Weikert T. Winkel D.J. Bremerich J. et al.Automated detection of pulmonary embolism in CT pulmonary angiograms using an AI-powered algorithm.Eur Radiol. 2020; 30: 6545-6553https://doi.org/10.1007/s00330-020-06998-0Crossref PubMed Scopus (27) Google Scholar], collected from 9 different sites and 17 different scanner models. According to the manufacturer's specifications, CT acquisition should be performed with a 64-slice scanner or higher and with a reconstructed slice thickness between 0.625 and 5.1 mm for ICH and 0.5–3.0 mm for PE. In addition, all technically inadequate scans should be excluded, such as scans with motion artifacts, severe metal artifacts, inadequate field of view and sub-optimal contrast bolus (PE).As soon as they are available in our PACS, CT images are automatically pseudo-anonymized and subsequently send for AI processing to a cloud server. The AI technology processed the non-enhanced head CT dataset to rule out ICH and CTPA data to diagnose PE. Afterwards, quantitative results and location specific annotated images are sent back into the PACS as additional dicom series. In case of intracranial bleeding or pulmonary embolism, these additional AI marked series contain images with arrows pointed where the pathology is situated. The AI report is seamlessly integrated into the clinical workflow, with the results being automatically added to the CT study. The typical time between the CT acquisition to the notification (AI results available in PACS) varies between 3 and 7 min for ICH studies and 5–9 min for PE studies.Diagnostic performance and discrepancy reviewFrom the 500 consecutive head CT exams and CTPA exams that were presented to the AI tool, we registered the number of studies that were sent back with an AI report. Secondly, we evaluated its diagnostic performance by comparison to expert reviewing. The original clinical radiology report after consensus review by 3 neuro-radiologists for ICH and 3 thorax radiologists PE, was considered as gold standard. Six board-certified radiologists participated in the consensus review with each 5 up to 15 year of experience in reading unenhanced head CT and CTPA studies. The reviewers had access to prior and future studies, and were able to see clinical history and reports to diagnose. AI results were classified into true positive, false positive, true negative and false negative cases. True-positive (TP) cases contained hemorrhage or embolism detected by the AI tool, and subsequently confirmed by the consensus reviewers. True-negatives (TN) consisted of exams without ICH nor PE according to the AI tool and reviewers. False-positives (FP) were defined as cases that were flagged positive by the AI tool but found out to be negative. False-negatives (FN) were defined as cases that were classified by the AI tool as negative but decided to be positive for ICH/PE by consensus review. We quantified the diagnostic performance of the AI algorithm in ICH and PE detection by calculating the sensitivity, specificity, negative predictive value (NPV), positive predictive value (PPV) and accuracy. The concordance between the AI tool and consensus review on each pathology, was calculated using percentage of agreement and Cohen's k statistic.In addition, the expert reviewers performed a detailed discrepancy analysis of the false-positive and false-negative cases in order to identify the reason for miss classification by AI.ResultsTable 2 summarizes the AI performance results for detecting ICH and PE. From the 500 presented cases, the AI algorithm could process 77.6% (388 of 500) for ICH evaluation during real-time radiology work stream. No ICH evaluation was performed for 112 studies. There was a difference in process rate between scanners models ranging from 84.9% (GE Discovery HD) to 53.8% (Philips iCT). All 6 Siemens cases were rejected for processing. From all 388 processed studies, the AI tool flagged 31 (7.9%) exams as having ICH. Expert review flagged 37 (9.5%) hemorrhages. Substantial agreement (kappa-value of 0.65) between AI and expert reading was observed. The performance for ICH showed 0.84 sensitivity and 0.94 specificity. The negative and positive predictive values were 0.98 and 0.61 respectively. The AI tool failed to label 1.7% (6 of 337) cases, that were agreed to have hemorrhage after review by three subspecialists (false negative by the AI tool). Those six cases summed in Table 3, conclude two discrete subarachnoid hemorrhages, two subdural hemorrhages and two parenchymal hemorrhages. Twenty positive results out of 37 flagged exams by the AI tool, were labelled as false-positive findings after consensus review, which the reviewers assigned to falcine or basal ganglia calcifications (9/20 cases), beam hardening artifacts (8/20 cases) and hyperdense dural sinuses (3/20), shown in Table 3.Table 2Number of processed studied and diagnostic performance of AI tool in detecting ICH and PE. Diagnostic accuracy values between brackets represent 95% confidence intervals.ICHPENumber of studies presented500500Studies with AI resultAll scanners77.6% (388/500)89.6% (448/500)GE Revolution84.4% (179/212)92.5% (261/282)GE Discovery 750HD84.9% (147/173)90.6% (184/203)Philips iCT56.8% (62/109) aSignificantly lower than both GE scanners (p < 0.05, Fischer Exact Probability test).13.3% (2/15) aSignificantly lower than both GE scanners (p < 0.05, Fischer Exact Probability test).Siemens Somatom AS400% (0/6) aSignificantly lower than both GE scanners (p < 0.05, Fischer Exact Probability test).N.a.Diagnostic PerformanceSensitivity0.84 (0.68–0.94)0.73 (0.62–0.82)Specificity0.94 (0.91–0.96)0.95 (0.93–0.97)NPV0.98 (0.96–0.99)0.94 (0.91–0.96)PPV0.61 (0.46–0.74)0.73 (0.62–0.82)Accuracy0.93 (0.90–0.96)0.98 (0.96–0.99)N.a. Not available.a Significantly lower than both GE scanners (p < 0.05, Fischer Exact Probability test). Open table in a new tab Table 3Detailed analysis of false negative and false positive ICH cases by AI.Detailed analysis of False Negative ICH cases by AISubarachnoid hemorrhages33% (2/6)Subdural hemorrhages33% (2/6)Parenchymal hemorrhages33% (2/6)Detailed analysis of False Positive ICH cases by AIFalcine and Basal ganglia calcifications45% (9/20)Beam hardening artefacts40% (8/20)Hyperdense dural sinuses15% (3/20) Open table in a new tab The AI technology created a report for 448 (89.6%) consecutive CTPA's. Similar to ICH, the process rate for the Philips iCT was lower (13.3%), compared to the GE scanners (90.6% and 92.5%). The sensitivity and specificity and accuracy were 0.73, 0.95 and 0.90 respectively. The kappa value for agreement between AI and expert reading was 0.78, indicating a substantial concordance. The expert readers detected 82 cases positive for PE. The AI system did not identify 19 of these 82 PE cases (false negative by AI). Nine of these patients, had chronic pulmonary embolisms. Six cases had masquerading artifacts. In three cases, an underlying pathology had concealed the present emboli. In 1 patient, a superimposing vein was the cause of the missed pulmonary embolism. 17 out of 19 misdiagnosed patients had subsegmental and segmental PE. Chronic known central emboli were missed in 1 patient. Lobar emboli were misdiagnosed by AI in 1 patient during delayed scan phase.24.4% of studies were found to be false-positive findings by the AI solution. According the consensus reviewers, six FP cases were due to contrast agent-related flow artifacts, beam hardening artifacts and breathing artifacts. Another six false positive cases were due to the masking effect of associated pathologies (such as infiltrate, metastasis, pleural effusion, atelectasis and fibrosis) or superposition anatomy (e.g. pulmonary vein, lymph node, hilar soft tissue, bronchus, azygos vein or pulmonary artery bifurcation). Withal, 7 out of 18 patients had a false positive diagnosis caused by a combination by the aforementioned factors.Examples of intracranial hemorrhages and pulmonary embolism with AI detection are shown in Fig. 1, Fig. 2, Fig. 3, Fig. 4.Fig. 2False negative ICH case by AI: subtle stroke hemorrhagic transformation not identified by the AI tool (white arrow).View Large Image Figure ViewerDownload Hi-res image Download (PPT)Fig. 3Left: false Positive ICH case by AI (yellow arrow) probably due to the presence of surrounded periventricular white matter hypoattenuation. Right: same image with different window/level settings. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)View Large Image Figure ViewerDownload Hi-res image Download (PPT)Fig. 4Top (A and B): False negative PE case by AI, lesion (white arrow) probably missed due to masquerading anatomy by hilar soft tissue. Bottom (A and B): False positive PE case by AI, a pulmonary vein (yellow arrow) is the cause of a false positive marked case by AI. (For interpretation of the references to colour in this figure legend, the reader is referred to the web version of this article.)View Large Image Figure ViewerDownload Hi-res image Download (PPT)DiscussionThis two-folded study assessed the diagnostic performance of an AI algorithm to automatically rule out ICH on non-contrast head CTs on one hand, and PE diagnose via CTPAs on the other hand, with real-time clinical work flow integration. In a diverse real-time clinical setting, 77.6% (388/500) of consecutive head CT exams and 89.6% (448/500) CTPA exams could be automatically evaluated by AI. However, AI process rates increase to 84.6% (326/385) for head CT exams and 91.7% (445/485) for CTPA if we only consider the two GE scanners who represent the bulk of the studies. We did not asses the cause of failure to process nor the follow up of these patients, as this was not part of the study protocol. Possible causes can be hospital-network related or can be attributed to inadequate radiological quality due to, for example, increased noise, the presence of motion artifacts or metal artifacts. The few Siemens cases were rejected for AI processing (0/6) because they were not compliant with the AI-tool requirements (<64 slice CT).With a specificity of 0.94 and a 0.98 negative predictive value, our ICH study is in line with prior work. Previous research with the same AI tool reported a specificity of 0.99 and a NPV of 0.98 for ICH detection. However, the sensitivity and PPV were 0.84 and 0.61 in our study, and remain moderate in comparison with previous studies, in which a sensitivity of 0.95 and 0.98 PPV was obtained [[19]Rao B. Zohrabian V. Cedeno P. Saha A. Pahade J. Davis M.A. Utility of artificial intelligence tool as a prospective radiology peer reviewer - detection of unreported intracranial hemorrhage [published online ahead of print, 2020 Feb 24].Acad Radiol. 2020; S1076–6332: 30084-30092https://doi.org/10.1016/j.acra.2020.01.035Abstract Full Text Full Text PDF Scopus (22) Google Scholar]. Weikert et al., stated that prior research with a sensitivity above 0.85 accepted more false positive findings, which increase the amount of false positive cases, thereby increasing the radiologist's workload and time to therapy initiation [[18]Weikert T. Winkel D.J. Bremerich J. et al.Automated detection of pulmonary embolism in CT pulmonary angiograms using an AI-powered algorithm.Eur Radiol. 2020; 30: 6545-6553https://doi.org/10.1007/s00330-020-06998-0Crossref PubMed Scopus (27) Google Scholar].False positive cases were more frequent (54% or 20/37) than false negatives, and mostly due to falcine or basal ganglia calcifications, hyperdense dural sinuses and streak artifacts, these results are in line with prior work assessed by Roa et al., effortlessly recognized without any difficulties by the original reporting radiologist [[19]Rao B. Zohrabian V. Cedeno P. Saha A. Pahade J. Davis M.A. Utility of artificial intelligence tool as a prospective radiology peer reviewer - detection of unreported intracranial hemorrhage [published online ahead of print, 2020 Feb 24].Acad Radiol. 2020; S1076–6332: 30084-30092https://doi.org/10.1016/j.acra.2020.01.035Abstract Full Text Full Text PDF Scopus (22) Google Scholar]. An example of a false positive ICH case by AI (yellow arrow) is shown in Fig. 3, probably due to the presence of surrounded periventricular white matter hypo-attenuation.False negatives ICH cases occurred in a limited number of cases (6/337) and were mainly seen in very small hemorrhages or follow-up exams within deteriorating patients.The most commonly false negative cases were sulcal subdural and subarachnoid hemorrhage, predominant in convex brain regions. However, we had one less subtle stroke hemorrhagic transformation (Fig. 2). Although, this missed finding could still be explained by very low HU densities within the lesion itself, luckily it was fast diagnosed by the original reporting radiologist. On one hand, the interpreting radiologist should scrutinize those critic brain regions very carefully. On the other hand, a missed subtle hemorrhage may be inconsequential, it needs to get your attention as well and may indicate further examination.With high specificity and negative predictive value, the AI tool shows the potential to rule out ICH.For our PE study we achieved a rather low sensitivity of 0.73 when compared to the study by Weikert et al., who reached a sensitivity level of 0.93 [[18]Weikert T. Winkel D.J. Bremerich J. et al.Automated detection of pulmonary embolism in CT pulmonary angiograms using an AI-powered algorithm.Eur Radiol. 2020; 30: 6545-6553https://doi.org/10.1007/s00330-020-06998-0Crossref PubMed Scopus (27) Google Scholar]. Firstly, this can be explained by a high prevalence of chronic emboli in our study population (9/19 cases). Secondly, 6 out of 19 patients had marked artifacts (movement, beam hardening and contrast agent-related flow artifacts). After consensus review, we also found 18 false positive cases flagged by the AI tool. According to the reviewers, also mainly due to artifacts, the other cases can be clarified by a combination of artifacts, masquerading pathologies (such as infiltrate, metastatis, pleural effusion, atelectasis, fibrosis) and superposing anatomy (e.g. pulmonary vein, lymph node, hilar soft tissue, bronchus, azygos vein, pulmonary artery bifurcation). Detailed analysis of false negative and false positive PE cases by AI are reported in Table 4. Examples of false negative and false positive PE cases by AI are shown in Fig. 4. Although false positive results can increase workload, these were again easily
Referência(s)