Portal de Periódicos da CAPES

The Emergency Medicine Milestones: A Validation Study

2013; Wiley; Volume: 20; Issue: 7 Linguagem: Inglês

10.1111/acem.12166

1553-2712

Robert C. Korte, Michael S. Beeson, Chad M. Russ, Wallace A. Carter, Earl J. Reisdorff,

Clinical Reasoning and Diagnostic Skills

The Accreditation Council for Graduate Medical Education (ACGME) and the American Board of Medical Specialties sought to define milestones for skill and knowledge acquisition during residency training. Milestones are significant objective observable events. The milestones are listed within a structure that is derived from the ACGME general competencies. Major groups of milestones are called "subcompetencies." The original 24 subcompetencies containing 255 milestones for emergency medicine (EM) were developed through a multiorganizational group representing most EM stakeholder groups. To assure that the milestones reflected EM resident progress throughout training, the EM Milestones Working Group (EM MWG) sought to validate the individual milestones. A computer-based survey was sent to all EM residency programs. The survey period began on April 30, 2012, and concluded on May 15, 2012. Respondents were asked to assign each milestone to a specific level of skill or knowledge acquisition. These levels ranged from a beginning resident to an accomplished clinician. There were two different forms that divided the milestones into two groups of 12 subcompetencies each. Surveys were randomly assigned to programs. There were five respondents (the program director and four key faculty) requested from each of the 159 residences. There were responses from 96 programs (60.4%). Of the 795 survey recipients, 28 were excluded due to prior exposure to the EM milestones. Of the remaining 767 potential respondents, 281 completed the survey (36.6%) within a 16-day period. Based on the survey results, the working group adjusted the milestones in the following ways: one entire subcompetency (teaching) was eliminated, six new milestones were created, 34 milestones were eliminated, 26 milestones were reassigned to a lower level score, and 20 were reassigned to a higher level. Nineteen milestones were edited to provide greater clarity. The final result was 227 discrete milestones among 23 subcompetencies. The EM milestones were validated through a milestone assignment process using a computer-based survey completed by program directors and key faculty. Milestones were revised in accordance with the results to better align assignment within each performance level. Los Objetivos por Áreas de la Medicina de Urgencias y Emergencias: Un Estudio de Validación El Accreditation Council for Graduate Medical Education (ACGME) y la American Board of Medical Specialties buscaron definir los objetivos por áreas para la adquisición de las habilidades y el conocimiento durante la formación de la residencia. Los objetivos por áreas son hechos observables objetivos y significativos. Los objetivos por áreas se enumeran en una estructura que se deriva de las competencias generales del ACGME. Los grupos mayores de los objetivos por áreas se denominan subcompetencias. Las 24 subcompetencias originales que contienen 255 objetivos por áreas para la medicina de urgencias y emergencias (MUE) se desarrollaron a través de un grupo de varias organizaciones que representan a la mayoría de los grupos interesados en la MUE. Para asegurar que los objetivos por áreas reflejaron el progreso del residente de MUE mediante la formación, el grupo de trabajo de los objetivos por áreas de la MUE buscó validar individualmente estos objetivos. Se envió una encuesta electrónica a todos los programas de residencia de MUE. El periodo de la encuesta comenzó el 30 de abril de 2012 y concluyó el 15 de mayo de 2012. Se solicitó a los respondedores que asignaran a cada uno de los objetivos por áreas un nivel específico de adquisición de habilidades o conocimiento. Estos niveles variaron desde el residente a su inicio hasta un clínico consumado. Hubo dos formularios diferentes que dividieron los objetivos por áreas en dos grupos de 12 subcompetencias cada uno. Las encuestas se asignaron de forma aleatoria a los programas. Se seleccionaron cinco respondedores (el director de programa y cuatro profesores representativos de la facultad) para cada una de los 159 programas de residencia. Hubo respuestas de 96 programas (60,4%). De las 795 encuestas recibidas, 28 se excluyeron debido a la experiencia previa en los objetivos por áreas de la MUE. De los restantes, 767 respondedores potenciales, 281 completaron la encuesta (36,6%) en un periodo de 16 días. En base a los resultados de la encuesta, el grupo de trabajo ajustó los objetivos por áreas de la siguiente manera: se eliminó una subcompetencia íntegra (enseñanza), se crearon 6 nuevos objetivos por áreas, se eliminaron 34 objetivos por áreas, se reasignaron 26 objetivos por áreas a un nivel menor y 20 a un nivel mayor. Se corrigieron 19 objetivos por áreas para proporcionar una mayor claridad. El resultado final fueron 227 objetivos por áreas distintos en 23 subcompetencias. Los objetivos por áreas de la MUE se validaron a través de un proceso de asignación de objetivos por áreas mediante una encuesta electrónica completada por los directores de los programas y profesores representativos de la facultad. Los objetivos por áreas se revisaron de acuerdo a los resultados para una mejor alineación de las asignaciones en cada nivel de realización. As part of the Next Accreditation System, the Accreditation Council for Graduate Medical Education (ACGME) has proposed a new method for tracking resident acquisition of knowledge, skills, abilities, attitudes, and experiences.1 The ACGME and the American Board of Medical Specialties sought to define objective milestones for skill and knowledge acquisition as a physician progresses in residency training. Categories that contained significant objective observable events ("milestones") that define competency acquisition are called "subcompetencies." In late 2011, the ACGME, through the Residency Review Committee for Emergency Medicine (RRC-EM), and the American Board of Emergency Medicine (ABEM) jointly convened the Emergency Medicine Milestones Working Group (EM MWG). This group was composed of representatives from major stakeholder organizations within the specialty. By using the newly developed ABEM certification standards and material developed by ACGME expert panel groups for common competencies, the EM MWG identified the original 24 subcompetency groups and developed the 255 milestones within these groups. Resident acquisition of milestones will be assessed for every resident every 6 months. Ideally, a progressive pattern of achieving increasingly more complex and sophisticated milestones will characterize a resident's experience in training. Performance on subcompetency milestone acquisition will be a data set that is annually reviewed by the RRC-EM for each residency program. Given the prominent role of milestone assessment for the individual resident, as well as its use as a residency measurement, it is important to have a valid set of milestones. This is achieved through expert consensus with subsequent validation by the academic EM community, especially program directors and faculty who will use the milestones as an evaluation device. ABEM undertook this study on behalf of the EM MWG. The goal of this activity was to validate the milestones and additionally provide information that could be used to modify the milestones. A computer-based survey was developed from the original draft of the milestones developed by the EM MWG. Surveys were sent to all 159 residency programs that were ACGME-accredited at the time of the survey. The two survey forms were randomly assigned to programs. Program directors and key faculty were asked to complete the survey. Once the program director received the survey, he or she selected four key faculty to also complete the survey. Beyond asking that the participating faculty be key faculty, no other directions were provided. It was anticipated that program directors would select faculty who had experience in evaluating residents. This study was considered exempt by the institutional research review board of Akron General Medical Center. Each subcompetency had a separate survey page. Markers were extracted from the level categories and listed alphabetically. The respondent was asked to designate into which of the five levels the marker should be placed. Given the size of the survey (24 subcompetencies and 255 milestones), the milestones were separated into two forms (Forms A and B), each of which had 12 subcompetencies to complete. Form A contained even numbered subcompetencies; Form B contained odd-numbered subcompetencies. In this way, the subcompetencies were nearly evenly distributed. The milestones validation survey used a computer-based platform Vovici EFM Continuum (Herndon, VA). Form A and Form B were randomly sent to all 159 EM residency programs on April 30, 2012. Randomization was achieved by using a Bernoulli trial assignment through SAS (version 9.2, SAS Institute, Cary, NC) using an equal probability of 0.5. Despite a delay of 2 days for one of the forms due to a programming issue, the survey was concluded on May 15, 2012 (16 days). There was no follow-up survey sent to garner additional responses. The short duration of the study was due to a desire to release the milestones at a national EM meeting. There was considerable interest on the part of the EM community to see the milestones, which had been embargoed prior to the completion of the survey. Given the study design, the milestones could not be shared prior to the completion of the survey. ABEM was assisted by the Council of Emergency Medicine Residency Directors in encouraging program directors and faculty to complete the survey. The survey results were also directly deposited in Vovici EFM Continuum. The frequency of milestone assignment into a specific level designation was the primary outcome. In addition, the cumulative responses for each level were determined. The number of residencies that had any respondents was noted, as well as the overall response rate of all potential respondents. The survey results were then used by the EM MWG to amend the assignment of milestones within a level using a set of predefined decision rules (see Data Supplement S1, available as supporting information in the online version of this paper). The final number of subcompetencies and milestones was determined. The number of milestones within a given level was also defined. Data analysis was performed using SAS (version 9.2, SAS Institute). Frequencies, cumulative frequencies, and chi-square analysis were used where appropriate. Significance for chi-square was predetermined to be α < 0.05. Of the 795 survey recipients, 28 were excluded due to prior exposure to the milestones. Program directors and faculty who were involved in the development of the milestones, as well as the development of the ABEM certification materials, were excluded. This was a prospective exclusion criterion. Respondents were asked to voluntarily exclude themselves in the survey instructions if this exclusion applied. Of the remaining 767 potential respondents, 281 completed the survey (36.6%). There were 390 copies (49.1%) of Form A sent and 405 copies (50.9%) of Form B sent. Of the 281 respondents, 126 (44.8%) completed Form A and 155 (55.2%) completed Form B. There was no significant difference in the overall response rate between the two forms (p = 0.08). There were responses from 96 programs (60.4%). Likewise, there was no significant difference in response rates between the two forms for residencies (p = 0.32). Of all responses, 36.7% came from program directors. Of the original 255 milestones, the survey results supported keeping 155 (60.8%) milestones at the same level and without any editing. An example of results for a milestone and how the milestone assignments were revised is shown in Tables 1, 2, and 3. The survey results for the teaching subcompetency showed a nearly random pattern of responses, suggesting that the assessment of the progressive acquisition of teaching skills within a residency was problematic. The final number of milestones that were selected was 227, which was an 11.0% reduction in milestones from the original milestone set.2 There were 20 of the 255 milestones (7.8%) that were moved to a higher level and 26 milestones (10.2%) that were moved to a lower level. There were 34 milestones (13.3%) that were eliminated. There were 19 (7.5%) milestones that required editing. Every level category for every subcompetency required a milestone. The result of moving and eliminated milestones was that some levels no longer contained a milestone. Thus, there were six milestones that were created to fill "empty" levels. In the final set of milestones, there were 38 (of 227) in Level 1 (16.7%), 50 in Level 2 (22.0%), 61 in Level 3 (26.9%), 47 in Level 4 (20.7%), and 31 (13.7%) in Level 5. Describes a primary assessment on a critically ill or injured patient Recognizes abnormal vital signs Recognizes when a patient is unstable requiring immediate intervention Prioritizes vital critical initial stabilization action in the resuscitation of a critically ill or injured patient Performs a primary assessment on a critically ill or injured patient Discerns relevant data to formulate a diagnostic impression and plan Reassesses after implementing a stabilizing intervention Manages and prioritizes critically ill or injured patients Recognizes in a timely fashion when further clinical intervention is futile Evaluates the validity of a DNAR order Integrates hospital support services into a management strategy for a problematic stabilization situation Recognizes when a patient is unstable requiring immediate intervention Performs a primary assessment on a critically ill or injured patient Discerns relevant data to formulate a diagnostic impression and plan (2) Manages and prioritizes critically ill or injured patients (3) Prioritizes vital critical initial stabilization action in the resuscitation of a critically ill or injured patient (4) Reassesses after implementing a stabilizing intervention Evaluates the validity of a DNAR order (5) Recognizes in a timely fashion when further clinical intervention is futile Integrates hospital support services into a management strategy for a problematic stabilization situation Pursuant to the ACGME mandate, every medical specialty must develop a unique set of milestones.1 These milestones will be used as an evaluation platform for every resident every 6 months. There are three tiers to each milestone set. The highest tier is the general competency assignment. The second tier is the subcompetency within that general competency. Each subcompetency contains progressively complex and sophisticated observable objective milestones. The ability range of these milestones spans from a recently graduated physician from medical school (Level 1), to an experienced attending physician (Level 5). As the resident acquires greater skill, the performance of various milestones should reflect a higher-level rating within that subcompetency. Ideally, all residents should acquire all markers in the Level 4 category by the time of residency graduation. The development of the milestones used a substantial foundation of data in their construction. The milestones were developed from three key sources: ABEM initial certification standards (unpublished, internal document); the Model of the Clinical Practice of Emergency Medicine (EM Model)3; and ACGME common competency expert panels (unpublished, internal document). The ABEM initial certification standards were developed through multiple advisory panels, an ABEM internal task force, and a national survey of 26,000 ABEM-certified physicians. This input was reviewed by the ABEM board of directors and refined into a final set of certification standards and items that reflected essential knowledge, skills, and abilities (KSAs) for emergency physicians. These were largely anchored on physician tasks listed in the EM Model.3 The EM Model further provided a list of clinical procedures germane to practicing emergency physicians. Finally, the ACGME expert panels created milestones for the common competencies: professionalism, interpersonal and communication skills, systems-based practice, and practice-based learning and improvement. The common competency materials were amended to be more directly applicable to EM. The resulting draft EM Milestones had 24 subcompetencies with 255 associated milestones that were placed in five different performance levels. Level 1 was to represent the competency of a graduating medical student, Level 2 was an early intermediate level, Level 3 was an advanced intermediate level, Level 4 was a graduating resident level, and Level 5 was an experienced attending physician level. By placing the ABEM initial certification standards (and associated KSAs) in the Level 4 category, there would be alignment of graduating resident competency and the ABEM initial certifying standards. There were nine patient care subcompetencies built on the physician tasks found in the EM Model. Of these, eight used the ABEM certification standards to define the desired milestone performance for residency completion. There were five additional patient care subcompetencies that were procedure-based. For procedures-based subcompetencies, there were no available ABEM initial certification standards. The procedure-based patient care subcompetencies and milestones were developed through expert panel deliberation and consensus by the EM MWG. The level assignments of the procedure-based milestones by the EM MWG were closely aligned with the survey results. There was a subcompetency for medical knowledge that also did not incorporate an ABEM certification standard and was developed by the EM MWG. Of the remaining eight subcompetencies that were based on the common competencies (e.g., systems-based practice), three used ABEM-developed certification standards. In general, the subcompetencies and milestones related to the common competencies relied substantially on the ACGME expert panel material, yet integrated ABEM data as well as EM MWG expert opinion. To our knowledge, no other medical specialty has surveyed all residencies to validate the milestones. Moreover, no other specialty started with such robust information and certifying board input as the EM Milestones. The study reported here provided data that resulted in refining the original milestones. These refinements used a set of heuristics that were approved by the EM MWG prior to adjusting the milestones (Data Supplement S1). One such decision rule was that for milestones where 50% or more of survey respondents chose a level different from the EM MWG assignment, the respondents' results should be considered, particularly if it is at a higher level than the original assignment. The manner in which milestones were modified can be understood by examining the modification process for the subcompetency of emergency stabilization. The original scoring matrix had 12 milestones distributed across the five levels (Table 1). Of note, the description of emergency stabilization ("prioritizes critical initial stabilization actions and mobilizes hospital support services in the resuscitation of a critically ill or injured patient and reassesses after stabilization") is the actual ABEM testing standard for initial certification. The study results contained the frequency of level assignment as well as the cumulative frequencies for each milestone (Table 2). Applying the aforementioned heuristic rules, the subcompetency for emergency stabilization was revised to a final form (Table 3). For this subcompetency, one milestone was eliminated, one milestone was moved to a higher level, and three milestones were moved to lower levels. The remaining 22 subcompetencies were also adjusted by the EM MWG in a similar fashion. Once the revisions were completed, the final version was submitted to the RRC-EM for approval. On September 7, 2012, the RRC-EM approved the validated EM Milestones,3 making EM the first medical specialty to approve a final set of milestones. The respondents were asked to assign each milestone to a particular time or sequence in resident training. They were blinded to the original placement by the MWG. They did not know the content of the discussions or the rationale of the MWG members. This might have influenced marker assignment. Ideally, 100% of respondents would have participated. A greater response rate could have altered results, but it is unknown if this would have been a significant change. Still, acquiring responses from 60% of EM residencies is considerable, given the 16-day survey window without a subsequent resending of the survey. The overall response rate of 37% was less robust. Obviously, the strength and confidence in validity would be greater with increased numbers of respondents. The intention of this study was to assure that the milestones were assigned to the correct levels. It did not ask if the subcompetencies themselves were valid categories, nor did it have respondents comment on the milestones as measurements. Given the broad input into the EM Model that is revised every 2 years, as well as the survey of over 26,000 emergency physicians in the foundational ABEM work,2 there is substantial support for the milestone subcompetency categories. In addition, the link of the common competencies to the ACGME general competencies and the use of expert panels strengthen the support for the selected milestone subcompetency categories. The assignment of a milestone into a specific level could vary among residents of the same year level. Where a milestone was assigned could be determined by when a resident is assigned to a certain clinical rotation. For example, when an anesthesia rotation occurred might influence when a particular airway-related milestone was achieved. This would not be a function of competency or program quality, but merely a scheduling artifact. There were six new milestones created that did not undergo the same validation process. Likewise, those items that received editing did not undergo resurvey. The Emergency Medicine Milestones were developed with previously developed material from American Board of Emergency Medicine that closely aligns the Level 4 milestone expectations with the American Board of Emergency Medicine certifying standards. Using an initial set of emergency medicine milestones developed by the EM Milestone Working Group, the American Board of Emergency Medicine conducted a study whereby program directors and key faculty assigned objective, observable actions into progressive levels of competency acquisition. The study was used to affirm or alter the milestones or their assignments to levels. In less than 2 years, a final set of EM milestones was produced, validated, and approved by the Accreditation Council for Graduate Medical Education. Please note: The publisher is not responsible for the content or functionality of any supporting information supplied by the authors. Any queries (other than missing content) should be directed to the corresponding author for the article.