Portal de Periódicos da CAPES

Automatic Brand Identification of Orthopedic Implants from Radiographs: Ready for the Next Step?

2022; Radiological Society of North America; Volume: 4; Issue: 2 Linguagem: Inglês

10.1148/ryai.220008

2638-6100

Merel Huisman, Nikolas Leßmann,

Anatomy and Medical Technology

HomeRadiology: Artificial IntelligenceVol. 4, No. 2 PreviousNext CommentaryFree AccessAutomatic Brand Identification of Orthopedic Implants from Radiographs: Ready for the Next Step?Merel Huisman, Nikolas LessmannMerel Huisman, Nikolas LessmannAuthor AffiliationsFrom the Department of Radiology, University Medical Center Utrecht, Heidelberglaan 100, Utrecht 3508, the Netherlands (M.H.); and Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Nijmegen, the Netherlands (N.L.).Address correspondence to M.H.Merel HuismanNikolas LessmannPublished Online:Mar 2 2022https://doi.org/10.1148/ryai.220008MoreSectionsPDF ToolsImage ViewerAdd to favoritesCiteTrack CitationsPermissionsReprints ShareShare onFacebookTwitterLinked In See article by Dutt and Mendonca et al in this issue.Merel Huisman, MD, PhD, is a radiologist with subspecialty interest in cardiothoracic radiology and musculoskeletal radiology. As an epidemiologist by training, her passion is the intersection between data science and clinical epidemiology. She is an EuSoMII board member and working group member of several national and international initiatives concerning standardization of artificial intelligence in health care. In 2021, she won an AuntMinnie Europe Award in the Rising Star category for her contribution to the field.Download as PowerPointOpen in Image Viewer Nikolas Lessmann, PhD, is an assistant professor at Radboud University Medical Center. He studied biomedical engineering at the University of Lübeck and obtained his doctorate from Utrecht University. In 2019, he joined the Diagnostic Image Analysis group, where his research interests focus on applications of machine learning and artificial intelligence in musculoskeletal image analysis.Download as PowerPointOpen in Image Viewer In an aging population, the need for revision surgery of orthopedic implants will become more prevalent. To adequately perform revision surgery, orthopedic surgeons must know the brand and model of the hardware in situ, for the simple reason that the appropriate instruments for removal have to be present in the operating room. If the available tools are incompatible with the hardware, delays and suboptimal procedures are the consequence, leading to higher costs, potentially higher complication rates, and considerable staff inconvenience. Usually, the surgeon relies on the original surgical notes to determine brand and model of the implanted hardware. A problem arises when the patient underwent surgery years ago, possibly in another country, and information on the implemented hardware is unavailable. In such cases, the surgeon has to rely on their experience or network to identify the implant preoperatively, which is often frustrating, time-consuming, and error prone. Radiologists do not typically recognize and/or report the brand and model of the hardware present on a radiograph. Even though this would be theoretically possible, it could be argued the advantages do not outweigh the disadvantages, as only in selected cases this would serve the orthopedic surgeon.Cervical spine hardware differs from hip and knee implants when it comes to the challenge of recognizing the brand and model in radiographs. Hip and knee implant registries have existed for many years now, and relatively few brands of hardware are used (1). Unlike prosthetic implants of the large joints, which can remain in situ for 15–20 years, spinal fixation hardware has a shorter life span; hence, developments are quicker and result in a wider variety of devices used. Therefore, a service that uses a diagnostic prediction model based on deep learning to quickly and reliably identify the implanted spinal hardware from radiographs in selected cases would be beneficial to patients and orthopedic surgeons alike.In this issue of Radiology: Artificial Intelligence, Dutt and Mendonca et al (2) pave the way for such a solution with a first step: the very successful proof of principle of a complete, replicable, and expandable pipeline for deep learning–based automatic detection and brand identification of cervical spinal hardware.In this single-center retrospective study (n = 984), Dutt and Mendonca et al describe and evaluate an artificial intelligence system that recognizes 10 of the most common cervical spine implants on anteroposterior or lateral view radiographs. Their system achieves an overall F1 score of 95% (95% CI: 92%, 98%) for both anterior and posterior hardware, indicating excellent model accuracy. An F1 score is a commonly used and robust accuracy measure and represents a combination of positive predictive value (precision) and sensitivity (recall) that is reliable regardless of the prevalence of the outcome. The corresponding area under the receiver operating characteristic curve is approaching 1, indicating near-perfect discrimination.The deep learning approach taken in this study does not offer many surprises at first sight. The authors relied on two popular neural network architectures, Efficient-Det and DenseNet, and assembled them into a straight-forward pipeline. However, with the excellent results that this approach achieves, the study adds to a growing body of evidence that orthopedic hardware identification might not be all too big of a challenge for modern neural networks (3). Several recent studies demonstrated that similarly standard approaches yield strong results for other types of hardware as well, such as knee (4) and hip (5,6) implants. In a study by Patel et al (7), such an approach was even more accurate in identifying implants than experienced orthopedic surgeons. These studies paint the picture of a technology mature enough to move beyond mere proof-of-principle studies.A current limitation that still needs to be addressed in general is the low number of implants that these systems can distinguish. Dutt and Mendonca et al built a system that can identify 10 cervical spine hardware models; other publications describe systems for other types of hardware that are equally restricted. Clearly, the practical value of automatic implant identification would be much greater if a single system could distinguish a broad range of implants, including various types of implants as well as older and rarer models that are especially problematic when planning revision surgery. Therefore, a pivotal question for further research will be how to efficiently scale these systems to a larger number of implants.For scaling up artificial intelligence models to larger datasets and more classes (ie, implant models), the approach of Dutt and Mendonca et al offers an attractive alternative to simply training a single prediction model with potentially hundreds of classes, corresponding to all kinds of implants. Their proposal is to divide the task into two serial tasks: a neural network specialized in detection (EfficientDet) first draws boxes around implants visible on the radiograph. It also labels the detected implants according to their general type, in this case, either anterior or posterior cervical spine hardware. Depending on which type of implant this gatekeeper network has identified, another neural network (DenseNet) specialized in that particular hardware type identifies the specific brand and model of the implant. With this strategy, the implant identification system becomes a collection of neural networks for different types of implants, and the addition of another hardware type becomes a matter of adding another network to this collection.Training these networks requires a set of images for which not only the type and model of the implant are known, but also in which the locations of the implants are annotated. These are used to train the gatekeeper network to detect implants and to confine the implant identification networks to relevant parts of the image. Rather than manually drawing boxes around the implants in the entire dataset, the article proposes a weakly supervised approach. Weak supervision is a form of supervised machine learning where the data are not hand annotated by an expert, but where less reliable and therefore weaker annotations are used instead. These can be, for instance, annotations crowdsourced from laypeople or annotations obtained with another machine learning model. The approach of Dutt and Mendonca et al requires annotation of a limited number of images, uses these to train the gatekeeper network that detects implants in the image, and uses that network to annotate the implant locations in the rest of the images. The study also gives an indication of how weak these annotations actually are: approximately 10% of the images were annotated, and in another 1900 randomly selected images, the automatically annotated implant location was reviewed, revealing mistakes in only 53 of 1900 images (3%).The study unfortunately does not give an indication of the minimum number of images that need to be hand annotated. This question stands at the beginning of most machine learning projects and ties in with the question of the minimum level of performance that the machine learning model needs to reach to become useful, be it for use in clinical practice or for weakly supervised learning. Finding reasonable estimates of these numbers will be important for making such approaches more attractive, making it easier for scientists to adopt the best approach.Another practical matter is collecting a large training set, consisting of radiographs of patients who received an implant of which brand and model can be determined reliably. Although Dutt and Mendonca et al manually reviewed surgical notes for almost 1000 patients, there is also promising research into automatic text analysis of surgical notes (8). Potentially, such methods could become a second source of weak labels and might facilitate assembly of much larger training sets.As with all single-center studies, external validation needs to be done on data that differ from the source population sampled in this study. We applaud the authors for including baseline characteristics of the patient population, including self-reported race and body mass index, which is useful for other institutions to make an estimation of how the results might translate to their local patient population. The authors provide free access to the final hardware localization and brand classification model at GitHub (a commonly used code repository for collaboration and version control) for other institutions to use and expand on.Dutt and Mendonca et al show impressive model performance in their cohort, indicating modern neural networks might have added value in clinical practice by reliably identifying medical devices. In a few years from now, one could think of a pay-per-use model for fast identification of any depicted medical device, perhaps in the form of a secured website or even a mobile app.Disclosures of Conflicts of Interest: M.H. Radiology: Artificial Intelligence trainee editorial board member. N.L. No relevant relationships.AcknowledgmentWe would like to thank J.J. Verlaan, MD, PhD (spine surgeon, University Medical Center Utrecht, Utrecht, the Netherlands) for his contributions.Authors declared no funding for this work.References1. Malchau H, Garellick G, Berry D, et al. Arthroplasty implant registries over the past five decades: Development, current, and future impact. J Orthop Res 2018;36(9):2319–2330. Crossref, Medline, Google Scholar2. Dutt R, Mendonca D, Phen M, et al. Automatic localization and brand detection of cervical spine hardware on radiographs using weakly supervised machine learning. Radiol Artif Intell 2022;4(2):e210099. Link, Google Scholar3. Ren M, Yi PH. Artificial intelligence in orthopedic implant model classification: a systematic review. Skeletal Radiol 2022;51(2):407–416. Crossref, Medline, Google Scholar4. Yi PH, Wei J, Kim TK, et al. Automated detection & classification of knee arthroplasty using deep learning. Knee 2020;27(2):535–542. Crossref, Medline, Google Scholar5. Karnuta JM, Haeberle HS, Luu BC, et al. Artificial Intelligence to Identify Arthroplasty Implants From Radiographs of the Hip. J Arthroplasty 2021;36(7S):S290–S294.e1. Crossref, Medline, Google Scholar6. Borjali A, Chen AF, Bedair HS, et al. Comparing the performance of a deep convolutional neural network with orthopedic surgeons on the identification of total hip prosthesis design from plain radiographs. Med Phys 2021;48(5):2327–2336. Crossref, Medline, Google Scholar7. Patel R, Thong EHE, Batta V, Bharath AA, Francis D, Howard J. Automated identification of orthopedic implants on radiographs using deep learning. Radiol Artif Intell 2021;3(4):e200183. Link, Google Scholar8. Sagheb E, Ramazanian T, Tafti AP, et al. Use of natural language processing algorithms to identify common data elements in operative notes for knee arthroplasty. J Arthroplasty 2021;36(3):922–926. Crossref, Medline, Google ScholarArticle HistoryReceived: Jan 13 2022Revision requested: Jan 19 2022Revision received: Jan 19 2022Accepted: Jan 25 2022Published online: Mar 02 2022 FiguresReferencesRelatedDetailsAccompanying This ArticleAutomatic Localization and Brand Detection of Cervical Spine Hardware on Radiographs Using Weakly Supervised Machine LearningJan 19 2022Radiology: Artificial IntelligenceRecommended Articles Automated Identification of Orthopedic Implants on Radiographs Using Deep LearningRadiology: Artificial Intelligence2021Volume: 3Issue: 4Deep Learning Improves Predictions of the Need for Total Knee ReplacementRadiology2020Volume: 296Issue: 3pp. 594-595Automatic Localization and Brand Detection of Cervical Spine Hardware on Radiographs Using Weakly Supervised Machine LearningRadiology: Artificial Intelligence2022Volume: 4Issue: 2Adventures and Misadventures in Plastic Surgery and Soft-Tissue ImplantsRadioGraphics2017Volume: 37Issue: 7pp. 2145-2163Convolutional Neural Networks for Automated Fracture Detection and Localization on Wrist RadiographsRadiology: Artificial Intelligence2019Volume: 1Issue: 1See More RSNA Education Exhibits Beyond "Prosthesis in Situ" - A Radiological Review of Normal Appearances and Complications of Orthopaedic ImplantsDigital Posters2019Elusive Complications in Hip Arthroplasty - Dare to Spot It: Imaging Features of Uncommon Postoperative ComplicationsDigital Posters2019Keep It Moving: Spondylosis and Posterior Spinal Motion Preserving SurgeriesDigital Posters2019 RSNA Case Collection Hip Polyethylene Liner DissociationRSNA Case Collection2021Silicone implant ruptureRSNA Case Collection2020Dysostosis Multiplex (Hurler's Syndrome)RSNA Case Collection2021 Vol. 4, No. 2 Metrics Downloaded 82 times Altmetric Score PDF download