Assessing the Accuracy of an Online Chat-Based Artificial Intelligence Model in Providing Recommendations on Hypertension Management in Accordance With the 2017 American College of Cardiology/American Heart Association and 2018 European Society of Cardiology/European Society of Hypertension Guidelines
2023; Lippincott Williams & Wilkins; Volume: 80; Issue: 7 Linguagem: Inglês
10.1161/hypertensionaha.123.21183
ISSN1524-4563
AutoresJoseph Kassab, Joseph El Dahdah, Michel Chedid El Helou, Habib Layoun, Ashish Sarraju, Luke J. Laffin, Leslie Cho, Samir R. Kapadia, Patrick Collier, Serge C. Harb,
Tópico(s)Machine Learning in Healthcare
ResumoHomeHypertensionVol. 80, No. 7Assessing the Accuracy of an Online Chat-Based Artificial Intelligence Model in Providing Recommendations on Hypertension Management in Accordance With the 2017 American College of Cardiology/American Heart Association and 2018 European Society of Cardiology/European Society of Hypertension Guidelines Free AccessLetterPDF/EPUBAboutView PDFView EPUBSections ToolsAdd to favoritesDownload citationsTrack citationsPermissions ShareShare onFacebookTwitterLinked InMendeleyReddit Jump toFree AccessLetterPDF/EPUBAssessing the Accuracy of an Online Chat-Based Artificial Intelligence Model in Providing Recommendations on Hypertension Management in Accordance With the 2017 American College of Cardiology/American Heart Association and 2018 European Society of Cardiology/European Society of Hypertension Guidelines Joseph Kassab, Joseph El Dahdah, Michel Chedid El Helou, Habib Layoun, Ashish Sarraju, Luke J. Laffin, Leslie Cho, Samir R. Kapadia, Patrick Collier and Serge C. Harb Joseph KassabJoseph Kassab https://orcid.org/0000-0001-6761-7636 Department of Cardiovascular Medicine, Heart and Vascular Institute, Cleveland Clinic Foundation, OH. , Joseph El DahdahJoseph El Dahdah https://orcid.org/0000-0002-8480-3745 Department of Cardiovascular Medicine, Heart and Vascular Institute, Cleveland Clinic Foundation, OH. , Michel Chedid El HelouMichel Chedid El Helou https://orcid.org/0000-0002-8587-2881 Department of Cardiovascular Medicine, Heart and Vascular Institute, Cleveland Clinic Foundation, OH. , Habib LayounHabib Layoun Department of Cardiovascular Medicine, Heart and Vascular Institute, Cleveland Clinic Foundation, OH. , Ashish SarrajuAshish Sarraju https://orcid.org/0000-0003-1649-2110 Department of Cardiovascular Medicine, Heart and Vascular Institute, Cleveland Clinic Foundation, OH. , Luke J. LaffinLuke J. Laffin https://orcid.org/0000-0002-3262-792X Department of Cardiovascular Medicine, Heart and Vascular Institute, Cleveland Clinic Foundation, OH. , Leslie ChoLeslie Cho https://orcid.org/0000-0002-1190-1774 Department of Cardiovascular Medicine, Heart and Vascular Institute, Cleveland Clinic Foundation, OH. , Samir R. KapadiaSamir R. Kapadia https://orcid.org/0000-0002-0026-3391 Department of Cardiovascular Medicine, Heart and Vascular Institute, Cleveland Clinic Foundation, OH. , Patrick CollierPatrick Collier https://orcid.org/0000-0003-3941-9653 Department of Cardiovascular Medicine, Heart and Vascular Institute, Cleveland Clinic Foundation, OH. and Serge C. HarbSerge C. Harb Correspondence to: Serge C. Harb, Department of Cardiovascular Medicine, Cleveland Clinic Foundation, 9500 Euclid Ave, J2-3, Cleveland, OH 44195. Email E-mail Address: [email protected] https://orcid.org/0000-0002-7442-4928 Department of Cardiovascular Medicine, Heart and Vascular Institute, Cleveland Clinic Foundation, OH. Originally published16 May 2023https://doi.org/10.1161/HYPERTENSIONAHA.123.21183Hypertension. 2023;80:e125–e127Other version(s) of this articleYou are viewing the most recent version of this article. Previous versions: May 16, 2023: Ahead of Print May 16, 2023: Ahead of Print Despite available treatment options, hypertension control remains poor in many regions worldwide. Patients often seek additional information beyond what their physician provides and are increasingly turning to internet sources for guidance. However, the quality and accuracy of online resources are highly variable. Artificial intelligence (AI) models may improve information accuracy and more effectively educate patients about hypertension. In November 2022, a chat-based AI language model (ChatGPT) that generates text responses through natural language processing became available to the public and rapidly gained millions of daily users. While not intended for medical purposes, a recent study suggested that this technology may possess the ability to provide accurate recommendations for cardiovascular disease prevention.1 We aimed to evaluate its ability to provide accurate patient-facing recommendations related to arterial hypertension by comparing them to the latest 2017 American College of Cardiology/American Heart Association and 2018 European Society of Cardiology/European Society of Hypertension guidelines on managing high blood pressure in adults.2,3Three physicians with extensive clinical and patient care experience created a set of 35 questions addressing arterial hypertension in adults. Questions focused on risk factors, preventive measures, as well as diagnosis and management and were designed in alignment with guideline-based topics and the physicians' clinical experience. The study was conducted in January 2023 using ChatGPT default model (GPT-3.5). The questions were posed to the online AI model 3 times to assess response consistency. Each set of answers was assigned for review to 1 of the 3 physicians. Physicians were explicitly asked to reference the current 2017 American College of Cardiology/American Heart Association and 2018 European Society of Cardiology/European Society of Hypertension guidelines2,3 when assessing the answers and grade each answer as accurate or inaccurate. A physician considered a response accurate if it included appropriate information found in the American and European guidelines or inaccurate if it (1) included inappropriate information that did not adhere to either of the guidelines or (2) missed essential details that a patient would find valuable, based on the physician's clinical judgment. An independent reviewer then combined the physicians' answers. Ultimately, to limit measurement bias and subjectivity, for each question, an answer was considered accurate if all 3 physicians unanimously deemed it to be accurate and inaccurate if at least 1 physician deemed it inaccurate.The AI model's responses to 31 out of the 35 questions (88%) were graded as accurate. Four responses were graded as inaccurate (12%; Table). For example, when asked about basic and optional laboratory tests upon a diagnosis of primary hypertension, the AI model suggested that lipid panel and ECG are optional tests when they should be ordered systematically.2,3 All AI-generated recommendations that were deemed accurate included information from the 2017 American College of Cardiology/American Heart Association guidelines (31/31). However, information from the 2018 ESC/ESH guidelines was included in only 18 of 31 accurate answers (58%). In the 13 accurate answers that did not incorporate European guidelines, the model solely addressed the question according to the American guidelines. For example, when asked about the different stages/grades of hypertension, the AI model defined them according to American guidelines. When asked about the benefits of nonpharmacological interventions like the Dietary Approaches to Stop Hypertension diet, answer also provided evidence and recommendations found in the 2017 American College of Cardiology/American Heart Association guidelines (ie, average of 11 mm Hg reduction in systolic blood pressure with the Dietary Approaches to Stop Hypertension diet).Table. Assessment of Responses From a Chat-Based AI Model to Frequently Asked Questions on the Management of High Blood PressureQuestionAccurateInaccurateACC/AHAESC/ESHDiagnosis and definitionsWhat are the stages/grades of hypertension?✓How common is high blood pressure?✓How to confirm hypertension?✓✓What is ambulatory blood pressure monitoring?✓✓What is white-coat hypertension?✓✓What is masked hypertension?✓✓What is resistant hypertension?✓✓What is the prevalence of resistant hypertension?✓What are the environmental risk factors associated with hypertension?✓Evaluation and managementIs there a genetic predisposition for hypertension?✓✓What are the basic and optional laboratory tests for primary hypertension?✓What is a DASH diet?✓✓How much SBP reduction can be obtained with a DASH diet?✓What is the recommended amount of sodium consumption to lower blood pressure?✓How much SBP reduction can be obtained with sodium restriction?✓What are the recommendations for alcohol consumption in managing hypertension?✓How much SBP reduction can be obtained with weight loss?✓✓What is the effect of aerobic exercise on hypertension?✓What are the first-line medications for hypertension?✓✓When to start pharmacological treatment for stage 1/grade 1 hypertension?✓What is the treatment for stage 2/grade 2 hypertension?✓✓When to reassess patients with stage 1/grade 1 hypertension and no pharmacological treatment?✓When to reassess patients with stage 2/grade 2 hypertension and pharmacological treatment?✓What is the blood pressure target after treatment in patients >65 y old?✓What are the causes of secondary hypertension?✓✓ComplicationsWhat is hypertensive urgency?✓What is hypertensive emergency?✓✓What are the guidelines for lowering blood pressure in hypertensive emergency for adults without a compelling condition?*✓What are the guidelines for lowering blood pressure in hypertensive emergency due to aortic dissection?✓✓Which medications are used in hypertensive emergency?✓✓Special considerationsWhich antihypertensive medications are recommended in black adults?✓✓Is combination therapy recommended for black patients?✓✓Which antihypertensive medications are recommended in women who plan to become pregnant?✓✓What are the recommended antihypertensives in patients with diabetes?✓What is the first line recommended antihypertensive in patients with chronic kidney disease?✓✓ACC indicates American College of Cardiology; AHA, American heart Association; DASH, dietary approaches to stop hypertension; ESC, European Society of Cardiology; ESH, European Society of Hypertension; ICU, intensive care unit; MAP, mean arterial pressure; and SBP, systolic blood pressure.* The management of hypertensive emergency depends on the severity of the acute target organ damage and the presence of other medical conditions. The guidelines for the management of hypertensive emergency in adults without a compelling condition include Admission to an ICU or other monitored setting for continuous blood pressure monitoring and assessment of organ function. Rapid but gradual reduction of blood pressure over a period of minutes to hours, depending on the severity of the target organ damage. In general, the goal is to reduce MAP by no more than 25% within the first hour and to lower blood pressure to less than 160/100 mmHg within the next 2 to 6 hours. Intravenous medications, such as nitroprusside, nicardipine, or labetalol, are the preferred agents for blood pressure control in most cases. It is important to note that the management of hypertensive emergency requires careful evaluation and treatment by a health care provider experienced in the management of severe hypertension.We found that a chat-based AI model offered largely accurate recommendations on managing high blood pressure that closely aligned with the current published guidelines. The AI model also performed well in answering questions relative to blood pressure treatment differences across age, sex, and race. However, the study demonstrated that the currently available model predominantly provides responses based on the 2017 American recommendations rather than the 2018 European guidelines. Given that the model was designed in the United States, a possible explanation could be that the text data on which the model was trained primarily consists of American sources. Future studies should assess the conformity of this model to non-US guidelines by specifically asking it to provide answers based on specific guidelines.Studies demonstrate that lack of awareness and underuse of guideline-recommended medications are among the critical factors contributing to poor blood pressure control.4 The AI model's high accuracy rate (88%) suggests its potential for assisting in hypertension patient education and communication. However, enhancing accuracy to reduce risks from incorrect recommendations remains essential. Recent data show that 60% of Americans would be uncomfortable with providers relying on AI in their own health care.5 This further reinforces the importance of studies designed to assess this technology. Whether future, medically facing, validated generative AI models can augment clinical decision-support and guideline adherence should also be explored.This study has several limitations. The current AI model is not designed for medical use and AI models are susceptible to training data limitations and bias. Additionally, heterogeneity between the set of 3 AI responses and inter-reviewer agreement of accuracy were not assessed in detail. Future research should aim to develop a standardized, validated system for grading responses. The readability and actionability of AI model responses should be assessed in future studies.Article InformationData AvailabilityIn accordance with Cleveland Clinic policy guidelines, data sharing with external parties is not allowed. However, the use of ChatGPT is free and accessible to anyone at chat.openai.com.Sources of FundingNone.Disclosures None.Footnotes*J. El Dahdah and M. Chedid El Helou contributed equally.For Sources of Funding and Disclosures, see page e127.Correspondence to: Serge C. Harb, Department of Cardiovascular Medicine, Cleveland Clinic Foundation, 9500 Euclid Ave, J2-3, Cleveland, OH 44195. Email harbs@ccf.orgReferences1. Sarraju A, Bruemmer D, Van Iterson E, Cho L, Rodriguez F, Laffin L. Appropriateness of cardiovascular disease prevention recommendations obtained from a popular online chat-based artificial intelligence model.JAMA. 2023; 329:842–844. doi: 10.1001/jama.2023.1044CrossrefMedlineGoogle Scholar2. Whelton PK, Carey RM, Aronow WS, Casey DE, Collins KJ, Dennison Himmelfarb C, DePalma SM, Gidding S, Jamerson KA, Jones DW, et al. 2017 ACC/AHA/AAPA/ABC/ACPM/AGS/APhA/ASH/ASPC/NMA/PCNA Guideline for the prevention, detection, evaluation, and management of high blood pressure in adults: a report of the American College of Cardiology/American Heart Association Task Force on Clinical Practice Guidelines.Hypertension. 2018; 71:e13–e115. doi: 10.1161/HYP.0000000000000065LinkGoogle Scholar3. Williams B, Mancia G, Spiering W, Agabiti Rosei E, Azizi M, Burnier M, Clement DL, Coca A, de Simone G, Dominiczak A, et al; ESC Scientific Document Group. 2018 ESC/ESH Guidelines for the management of arterial hypertension: the Task Force for the management of arterial hypertension of the European Society of Cardiology (ESC) and the European Society of Hypertension (ESH).Eur Heart J. 2018; 39:3021–3104. doi: 10.1093/eurheartj/ehy339CrossrefMedlineGoogle Scholar4. Carey RM, Muntner P, Bosworth HB, Whelton PK. Prevention and control of hypertension: JACC health promotion series.J Am Coll Cardiol. 2018; 72:1278–1293. doi: 10.1016/j.jacc.2018.07.008CrossrefMedlineGoogle Scholar5. Tyson A, Pasquini G, Spencer A, Funk C. 60% of Americans Would Be Uncomfortable With Provider Relying on AI in Their Own Health Care.Pew Research Center Science & Society; 2023. https://www.pewresearch.org/science/2023/02/22/60-of-americans-would-be-uncomfortable-with-provider-relying-on-ai-in-their-own-health-care/Google Scholar eLetters(0) eLetters should relate to an article recently published in the journal and are not a forum for providing unpublished data. Comments are reviewed for appropriate use of tone and language. Comments are not peer-reviewed. Acceptable comments are posted to the journal website only. Comments are not published in an issue and are not indexed in PubMed. Comments should be no longer than 500 words and will only be posted online. References are limited to 10. Authors of the article cited in the comment will be invited to reply, as appropriate. Comments and feedback on AHA/ASA Scientific Statements and Guidelines should be directed to the AHA/ASA Manuscript Oversight Committee via its Correspondence page. Sign In to Submit a Response to This Article Previous Back to top Next FiguresReferencesRelatedDetailsCited ByKassab J, Kapadia V, Massad C, Sarraju A, Ramchand J, Kapadia S and Harb S (2023) Comparative Analysis of Chat‐Based Artificial Intelligence Models in Addressing Common and Challenging Valvular Heart Disease Clinical Scenarios, Journal of the American Heart Association, 12:22, Online publication date: 21-Nov-2023. Omiye J, Gui H, Daneshjou R, Cai Z and Muralidharan V (2023) Principles, applications, and future of artificial intelligence in dermatology, Frontiers in Medicine, 10.3389/fmed.2023.1278232, 10 Kassab J, Nasr L, Gebrael G, Chedid El Helou M, Saba L, Haroun E, Dahdah J and Nasr F (2023) AI-based online chat and the future of oncology care: a promising technology or a solution in search of a problem?, Frontiers in Oncology, 10.3389/fonc.2023.1176617, 13 July 2023Vol 80, Issue 7 Advertisement Article Information Metrics © 2023 American Heart Association, Inc.https://doi.org/10.1161/HYPERTENSIONAHA.123.21183PMID: 37190998 Originally publishedMay 16, 2023 Keywordsartificial intelligencehypertensionpatientphysiciantreatmentPDF download Advertisement Subjects High Blood Pressure Hypertension
Referência(s)