Factors Associated with Declining Residency Program Pass Rates on the ABIM Certification Examination
2016; Elsevier BV; Volume: 129; Issue: 7 Linguagem: Inglês
10.1016/j.amjmed.2016.03.020
ISSN1555-7162
AutoresLisa L. Willett, Andrew J. Halvorsen, Michael Adams, Karen M. Chacko, Saima Chaudhry, Furman S. McDonald, Amy S. Oxentenko, Sara Swenson, Aimee K. Zaas, Vineet M. Arora,
Tópico(s)Radiology practices and education
ResumoPerspectives Viewpoints•American Board of Internal Medicine 3-year rolling pass rates vary across geographic region, program size, and program director tenure.•Programs offer a variety of methods to prepare residents for the examination, and report making changes in response to failures.•Program directors attribute resident failures to low performance on prior standardized tests, and are giving greater consideration to US Medical Licensing Examination scores when ranking student applicants. •American Board of Internal Medicine 3-year rolling pass rates vary across geographic region, program size, and program director tenure.•Programs offer a variety of methods to prepare residents for the examination, and report making changes in response to failures.•Program directors attribute resident failures to low performance on prior standardized tests, and are giving greater consideration to US Medical Licensing Examination scores when ranking student applicants. The American Board of Internal Medicine (ABIM) establishes uniform standards for physicians in practice.1American Board of Internal Medicine (ABIM)Exam administration history.2015http://www.abim.org/about/default.aspxGoogle Scholar The ABIM certifies that internists demonstrate knowledge, skills, and attitudes essential for patient care. For initial certification, a physician must complete the requisite medical education and clinical training, and pass a high-stakes written examination.2American Board of Internal Medicine (ABIM)ABIM certification guide.2015http://www.abim.org/certification/Google Scholar A physician's board certification status is publicly reported, and considered a measure of physician competence.3Ramsey P.G. Carline J.D. Inui T.S. et al.Predictive validity of certification by the American Board of Internal Medicine.Ann Intern Med. 1989; 110: 719-726Crossref PubMed Scopus (156) Google Scholar, 4Sharp L.K. Bashook P.G. Lipsky M.S. et al.Specialty board certification and clinical outcomes: the missing link.Acad Med. 2002; 77: 534-542Crossref PubMed Scopus (141) Google Scholar The ABIM certification examination is also high stakes for residency programs. For accreditation in Internal Medicine (IM), a program must have at least 80% of its graduates pass the examination on the first attempt in the most recently defined 3-year period, called the “3-year rolling pass rate.”5Accreditation Council for Graduate Medical Education (ACGME)ACGME program requirements for graduate medical education in internal medicine.2015http://acgme.org/acgmeweb/Portals/0/PFAssets/2013-PR-FAQ-PIF/140_internal_medicine_07012013.pdfGoogle Scholar Over the past 6 years, the number of residency programs achieving an 80% 3-year program pass rate declined from 96% in 2008 to 75% in 2013, with 22% of programs falling below the accreditation requirement for the most recently reported interval (2012-2014) (Figure 1),6American Board of Internal Medicine (ABIM)Residency program pass rates.2015http://www.abim.org/pdf/passrates/residency-program-pass-rates.pdfGoogle Scholar despite individual pass rates remaining stable at 84%-91%.7American Board of Internal Medicine (ABIM)First-time taker pass rates initial certification.2015http://www.abim.org/pdf/pass-rates/cert.pdfGoogle Scholar Given the importance of the certification examination for residents, program accreditation, and patients, the decline in program pass rates is concerning and warrants study. Understanding factors that place trainees and programs at risk of failure may provide solutions to better prepare residents for success. Residency program directors (PDs) are uniquely aware of certification examination outcomes, at both the individual and program level. Thus, we conducted a national survey of IM PDs to identify program characteristics associated with program pass rates, explore reasons, and program response to the decline. The Association of Program Directors in Internal Medicine Survey Committee develops yearly questionnaires to track characteristics and issues facing IM residency programs. We used the 2013 survey to ask questions related to the declining ABIM program pass rates. E-mail notification with a program-specific hyperlink to a Web-based questionnaire was sent August-November 2013 to all 365 Association of Program Directors in Internal Medicine member programs, representing 93.4% of Accreditation Council for Graduate Medical Education (ACGME)-accredited IM residency programs. Prior to blinding program identity for analysis, survey responses were appended with data from publicly available sources, accessed December 2013. Programs were assigned to geographic regions by US Census Bureau definition.8US Census BureauCensus regions and divisions of the United States.2015http://www2.census.gov/geo/pdfs/maps-data/maps/reference/us_regdiv.pdfGoogle Scholar Program type was obtained from the American Medical Association Fellowship and Residency Electronic Interactive Database Access System Online.9American Medical Association (AMA)FREIDA online.2015http://www.ama-assn.org/ama/pub/education-careers/graduate-medical-education/freida-online.pageGoogle Scholar The 3-year program pass rates on the certification examination were obtained from ABIM.6American Board of Internal Medicine (ABIM)Residency program pass rates.2015http://www.abim.org/pdf/passrates/residency-program-pass-rates.pdfGoogle Scholar Data from ACGME10Public reporting page for the Accreditation Council for Graduate Medical Education (ACGME). 2015. Available at: https://apps.acgme.org/ads/public/. Accessed December 29, 2015.Google Scholar included accreditation cycle length, government affiliations, number of approved training positions, and PD appointment date. We included programs with >10 residents meeting all reported criteria. This study was exempt by the Mayo Clinic Institutional Review Board. To evaluate sample representativeness, we compared programs responding to nonresponders across the publicly available variables (program type, region, size, PD tenure, and ABIM program pass rates) using Fisher's exact tests or 2 sample t tests, as appropriate. We assessed differences in 3-year program pass rates (years 2010-2012)6American Board of Internal Medicine (ABIM)Residency program pass rates.2015http://www.abim.org/pdf/passrates/residency-program-pass-rates.pdfGoogle Scholar across program demographics (program type, region, size, and PD tenure), resident demographics (percentage of US medical graduates, women, and underrepresented minorities), and PD responses to survey items using multivariate analysis of variance models. PDs' report of correlates with residents who failed were described. Differences in PD agreement and program offerings included: 1) perceived reasons for program pass rate decline, 2) perceptions of why individual residents in their program did not pass, 3) methods to prepare their residents for the examination, and 4) response to the decline. We collapsed the original 5-point agreement scale (1 = “strongly disagree,” 2 = “disagree,” 3 = “neutral,” 4 = “agree,” 5 = “strongly agree”) to a binary response (1 = “strongly agree” or “agree,” 0 = “neutral,” “disagree,” or “strongly disagree”). Associations were assessed with Fisher's exact tests for 2-way tables, 2 sample t tests, or logistic regression, as appropriate. Continuous-valued numeric characteristics were trichotomized using sample tertiles to create similarly sized groups within demographics. The significance level for all analyses was set at 0.01 for multiple comparisons, and all tests were 2-sided. Analyses used SAS 9.3 (SAS Institute, Cary, NC). Surveys were returned by 265 PDs (73%). Responding programs did not differ from nonresponding (Table 1). Of ACGME-accredited programs, 370/391 (94.6%) had 3-year program pass rates (years 2010-2012) available from the ABIM, with a mean (95% confidence interval [CI]) pass rate of 85.0% (84.0-86.0); mean (95% CI) program pass rates from survey respondents was 85.7% (84.6-86.9).Table 1Association of Program Directors in Internal Medicine 2013 Survey Responders vs Nonresponders for Publicly Available Characteristics of 365 Internal Medicine Residency ProgramsRespondersn = 265Nonrespondersn = 100P-ValueCharacteristic Program type, n (%).61∗Fisher's exact test.Community-based, university-affiliated hospital136 (51.3%)56 (56.0%)University-based99 (37.4%)31 (31.0%)Community-based25 (9.4%)10 (10.0%)Military-based5 (1.9%)3 (3.0%) Region, n (%).70∗Fisher's exact test.Northeast94 (35.5%)31 (31.0%)South67 (25.3%)32 (32.0%)Midwest64 (24.2%)22 (22.0%)West38 (14.3%)15 (15.0%)Unincorporated territory2 (0.8%)0 (0.0%) Program size, Median (Q1-Q3) approved positions56 (38-93)52.5 (36.5-80.5).27†Welch's t test. Program director tenure, Median (Q1-Q3) years4 (1-9)5 (2-9.5).55†Welch's t test. ABIM program pass rate ('10-'12), Mean (95% CI) percent85.7 (84.6-86.9)84.7 (82.5-86.8).38†Welch's t test. Programs below 80% RRC Threshold, No. (%)54 (20.4%)25 (25.0%).31∗Fisher's exact test.ABIM = American Board of Internal Medicine; ACGME = Accreditation Council for Graduate Medical Education; ANOVA = analysis of variance; CI = confidence interval; Q1 = first quartile; Q3 = third quartile.∗ Fisher's exact test.† Welch's t test. Open table in a new tab ABIM = American Board of Internal Medicine; ACGME = Accreditation Council for Graduate Medical Education; ANOVA = analysis of variance; CI = confidence interval; Q1 = first quartile; Q3 = third quartile. Average 3-year program pass rates varied significantly across regions, program sizes, and PD tenures (all P ≤ .004) (Table 2). Military-based programs and programs in Puerto Rico had the lowest pass rates. Programs with >80% of positions filled with US medical graduates had the highest pass rates. Mean program pass rates did not differ according to programs' percentages of women or underrepresented minority graduates (Table 3).Table 2Publicly Available Program Characteristics vs ABIM Program Pass Rate ('10-'12) Multivariate ANOVA Summaries (N = 370)Program Characteristic2010-2012 ABIM Program Pass Raten (%)Mean (95% CI)P-ValueProgram type.20 Community-based, university affiliated hospital194 (52.4)84.5 (83.1-86.0) University-based132 (35.7)86.0 (84.6-87.5) Community-based36 (9.7)85.1 (81.0-89.2) Military-based8 (2.2)79.5 (70.2-88.8)Region.0007 Northeast131 (35.4)85.3 (83.6-87.0) South96 (25.9)83.9 (81.9-85.8) Midwest84 (22.7)86.5 (84.6-88.4) West51 (13.8)86.1 (83.3-88.9) Puerto Rico8 (2.2)71.8 (56.0-87.5)Program size.004 Large (≥76 approved positions)119 (32.2)87.0 (85.7-88.3) Medium (43-75 approved positions)119 (32.2)85.3 (83.9-86.7) Small (≤42 approved positions)132 (35.7)83.0 (80.7-85.2)Program director tenure.002 Long (>7 years)124 (33.5)87.4 (85.8-89.1) Medium (2-7 years)152 (41.1)84.2 (82.5-85.8) Short ( 80%)86 (32.5)88.6 (86.7-90.4) Medium (12%-80%)85 (32.1)83.2 (81.4-85.1) Low ( 47%)78 (29.4)86.1 (83.8-88.5) Medium (32%-47%)82 (30.9)87.2 (85.5-88.9) Low ( 12.5%)74 (27.9)86.2 (83.8-88.6) Medium (5.3%-12.5%)82 (30.9)85.6 (83.3-87.8) Low ( .07). Some provided funding for the Medical Knowledge Self-Assessment Program (61.1%), study materials (49.8%), commercial review courses (26.0%), ABIM-specific study materials (20.8%), and travel expenses for courses (14.3%). The majority administered the ITE to all postgraduate year (PGY) levels (PGY2 93.6%, PGY3 91.7%, PGY1 86.4%). Eighty-five percent used an ITE threshold score to identify residents at risk for failing the ABIM. The majority (81.8%) used a national percentile rank compared with peers, with a median (Q1-Q3) percentile of 30 (27.5-35). Programs with lower average pass rates (82.5% vs 87.4%) adopted a higher threshold, mean (95% CI) difference of 4.9% (2.1-7.8) (P = .001). We found no association between ITE thresholds used by PDs and 3-year program pass rates (P = .29). Sixty-four percent of PDs made changes to their board preparation methods in response to failures. Those who made changes had lower pass rates than those who had not (83.2% vs 92.7%), a mean (95% CI) difference of 9.6% (7.8-11.3) (P < .0001), and were more likely to fall below the ACGME requirement for ≥80% program pass rate (28.4% vs 1.4%; odds ratio 27.8; 95% CI, 3.8-205.4; P = .001). Sixty-two percent reported giving stronger consideration to US Medical Licensing Examination (USMLE)13US Medical Licensing Examination (USMLE)United States medical licensing examination.2015http://www.usmle.org/Google Scholar scores for ranking residency applicants than in years past. PDs who reported giving increased consideration of USMLE scores were more likely to have lower program pass rates than those who did not (83.7% vs 90.6%), a mean (95% CI) difference of 6.9% (4.9-8.9) (P < .0001), and were more likely to fall below the ACGME requirement for ≥80% pass rate (27.8% vs 5.3%; odds ratio 6.9; 95% CI, 2.4-20.1; P = .0004). Stable pass rates of individual first-time test takers in the face of declining 3-year program pass rates presents a paradox that is worth exploration. The stable pass rates of individuals suggest that the ABIM certification examination is not more difficult, nor are there intrinsic differences in the aggregate pool of medical trainees pursuing IM residencies. The reason for declining program pass rates is likely multifactorial, related to differences in the distribution of residents among certain programs who failed the ABIM examination. There may be a shift at the program level, such that residents who failed the examination were previously concentrated in a few programs and are now dispersed broadly across many programs. Conversely, residents most at risk of failing may now be concentrated within certain programs, with characteristics we identified. For example, it is possible that programs who place less emphasis on USMLE scores when ranking applicants, due to other highly desirable qualities, have lower pass rates. The shift may have disproportionately impacted programs with at-risk residents. We found examination pass rates to be lower for programs in Puerto Rico, smaller programs, those with shortest PD tenure and with more International Medical Graduates. Our findings are consistent with other studies of smaller programs and regional variance.14Falcone J.L. Middleton D.B. Pass rates on the American Board of Family Medicine certification exam by residency location and size.J Am Board Fam Med. 2013; 26: 453-459Crossref Scopus (15) Google Scholar, 15McKendry R.J. Dale P. Does the number of trainees in a postgraduate training program influence the pass rates on certifying examinations?.Clin Invest Med. 1995; 18: 73-79Google Scholar, 16Falcone J.L. Gonzalo J.D. Relationship between internal medicine program board examination pass rates, accreditation standards, and program size.Int J Med Educ. 2014; 5: 11-14Crossref PubMed Scopus (10) Google Scholar, 17Atsawarungruangkit A. Relationship of residency program characteristics with pass rate of the American Board of Internal Medicine certifying exam.Med Educ Online. 2015; 20: 28631Crossref Scopus (10) Google Scholar Reasons for these associations are not known, and may reflect differences in educational support, baseline knowledge of trainees, variations in clinical exposures, availability of resources for board preparation, or unstable programs. Further study to identify why certain programs are at risk is essential to ensure that programs fulfill accreditation requirements and that all trainees get the support they need to pass the examination. In our survey, PDs perceive the national decline in program pass rates to be due to residents spending less time reading and reflecting about patients, and having less clinical experience due to duty hours. Although prior studies show residents spend less time with patients and in educational conferences due to duty hour limitations,18Desai S.V. Feldman L. Brown L. et al.Effect of the 2011 vs 2003 duty hour regulation-compliant models on sleep duration, trainee education, and continuity of patient care among internal medicine house staff: a randomized trial.JAMA Intern Med. 2013; 173: 649-655Crossref PubMed Scopus (170) Google Scholar, 19Myers J.S. Bellini L.M. Morris J.B. et al.Internal medicine and general surgery residents' attitudes about the ACGME duty hours regulations: a multicenter study.Acad Med. 2006; 81: 1052-1058Crossref PubMed Scopus (77) Google Scholar, 20Shea J.A. Willett L.L. Borman K.R. et al.Anticipated consequences of the 2011 duty hours standards: views of internal medicine and surgery program directors.Acad Med. 2012; 87: 895-903Crossref Scopus (28) Google Scholar the 2003 duty hour limitations were not associated with changes in ABIM pass rates.21Silbert J.H. Romano P.S. Itani K.M. et al.Assessing the effects of the 2003 resident duty hour reform on Internal Medicine board scores.Acad Med. 2014; 89: 1-8Google Scholar Of note, the timing of the decline seems to coincide more closely with the 2009 ACGME IM requirements for increased ambulatory training and other curricular changes.22Willett L.W. Estrada C.A. Adams M. et al.Challenges with continuity clinic and core faculty accreditation requirements.Am J Med. 2013; 126: 550-556Abstract Full Text Full Text PDF PubMed Scopus (6) Google Scholar While there isn't a direct hypothesis for how these changes affected program pass rates, it is plausible that some programs diverted program resources away from board preparation to meet 2009 mandates. Interestingly, the perceived decreased time in the clinical environment reported by PDs occurred concurrently with an increased emphasis on competencies other than medical knowledge, including system-based practice and practice-based learning and improvement.23Cooke M. Irby D.M. O'Brien B.C. Educating Physicians: A Call for Reform of Medical School and Residency. vol. 16. John Wiley & Sons, Philadelphia2010Google Scholar, 24Chaudhry S.I. Lien C. Ehrlich J. et al.Curricular content of internal medicine residency programs: a nationwide report.Am J Med. 2014; 127: 1247-1254Abstract Full Text Full Text PDF Scopus (9) Google Scholar Changes in the work environment that align with these competencies, including use of electronic health records, complex coordination of care transitions, increased documentation requirements, and other system-level complexities may be distracting from residents' attainment of medical knowledge. PDs attributed most resident failures with low performance on prior tests of medical knowledge; few attributed failures to low performance on clinical evaluations in other competency domains. How to best prepare residents for modern practice, while ensuring the fundamental medical knowledge of a competent physician, ready for unsupervised practice, remains an important area of study. Because of declining program pass rates, the majority of PDs reported giving higher priority to USMLE scores to rank medical students on their match list than in years past, especially those with lower pass rates. This is important information for all medical students, faculty advisors, and admissions committees. Given the increasingly competitive match, due to slow growth of graduate medical education positions compared with increased numbers of medical school applicants,25National Resident Matching ProgramThe Match, 2015 Main Residency Match Advance Data Tables.2015http://www.nrmp.org/wp-content/uploads/2015/03/ADT2015_final.pdfGoogle Scholar students with lower USMLE scores may not only be in jeopardy of not matching at the IM residency program of their choice; they may not match at all. Although our study is based on a large, nationally representative survey of PDs, it does have limitations. We assessed PDs' perceptions, not proven reasons, for the national decline in program pass rates and individual resident failures. For example, PDs perceive that ITE scores correlated with individual resident failures, but we did not collect actual score data with our survey. In addition, given the limited length of the survey, we may not have captured all perceptions about pass rates. In this survey of IM residency PDs, certain program characteristics were associated with lower program pass rates on the ABIM certification examination. Programs offer a variety of methods to prepare their residents for the examination, and reported making changes in response to failures. The majority reported giving stronger consideration to USMLE scores than in years past when ranking students on their match list. Further study is needed to understand the paradox of declining program pass rates with stable individual pass rates. We are grateful to the Mayo Clinic Survey Research Center for assistance with the survey design and data collection.
Referência(s)