Portal de Periódicos da CAPES

Assessment of performance in orthopaedic training

2005; British Editorial Society of Bone & Joint Surgery; Volume: 87-B; Issue: 9 Linguagem: Inglês

10.1302/0301-620x.87b9.16434

2044-5377

David W. Pitts, D.I. Rowley, J. L. Sher,

Medical Education and Admissions

The Journal of Bone and Joint Surgery. British volumeVol. 87-B, No. 9 AnnotationFree AccessAssessment of performance in orthopaedic trainingD. Pitts, D. I. Rowley, J. L. SherD. PittsSurgical Education Development AdvisorUniversity Department of Orthopaedics & Trauma Surgery, TORT Centre, Ninewells Hospital and Medical School, Dundee DD1 9SY, UK.Search for more papers by this author, D. I. RowleyUniversity Department of Orthopaedics & Trauma Surgery, TORT Centre, Ninewells Hospital and Medical School, Dundee DD1 9SY, UK.Search for more papers by this author, J. L. SherWansbeck General Hospital, Woodhorn Lane, Ashington, Northumberland NE63 9JJ, UK.Search for more papers by this authorPublished Online:1 Sep 2005https://doi.org/10.1302/0301-620X.87B9.16434AboutSectionsPDF/EPUB ToolsDownload CitationsTrack CitationsPermissionsAdd to Favourites ShareShare onFacebookTwitterLinked InRedditEmail The reliable measurement of performance is a problem faced by training authorities worldwide. At a recent international orthopaedic conference, none of a group of experienced orthopaedic educators could report significant progress on this issue.The current Record of In Training Assessment (RITA) process which relies on simple assessment forms filled in by trainers and trainees is regarded widely as inadequate and lacking in objectivity. More robust evidence is needed.The Orthopaedic Competence Assessment Project (OCAP), a combined initiative of the Education Committee of the British Orthopaedic Association and the Specialist Advisory Committee (SAC) in Trauma & Orthopaedics was given the remit to improve training through a competency-based portfolio of coaching and assessment tools. These instruments had to be easily understood by a range of trainers and trainees, not time-consuming and simple to apply.OCAP recognises the need for a battery of tools to address the different aspects of performance. These have been described by Miller1 who refers to different levels of assessment which are best tested in different ways (Fig. 1). In Miller’s pyramid, ‘knowing’ and ‘knowing how’ are reliably tested in well-defined examination formats and set-piece assessments, such as clinical examinations and orals can monitor ‘shows how’. However, ‘does’ requires real-time assessment and it is to this need that OCAP has responded.Any assessment needs to be valid and reliable. Validity is well served by real clinical situations. Reliability, which is equally important, requires a tool which is practical, consistent and easy to repeat.The instruments described below follow the principles of Miller1 and Norman.2 In this article, we describe the instruments, their validation and how reliability might be ensured.OCAP instruments (Fig. 2)The package consists of two types of instrument: agenda setting/coaching, and assessment.The agenda-setting instruments of the trainer profile have been piloted and found to be acceptable in two large training programmes in the United Kingdom. The assessment instruments are also complete in terms of validity, but reliability studies are ongoing. The personal effectiveness instruments which coach and test generic skills are at an early stage of development.How OCAP works.In advance of an attachment, trainers and trainees exchange information about themselves. A trainee’s portfolio is shared with the trainer who provides a personal profile for the trainee. This profile is easily formatted from a database of ‘knowledge’ and ‘procedures’ and outlines the surgical training available in an attachment. The profile describes the procedures the trainer performs regularly and the knowledge base to underpin this activity. Both parties then meet at the start of the attachment and draw up a learning agreement of which the key objectives are: to identify the trainee’s level of knowledge and experience; to set realistic goals and expectations for the attachment; to agree the knowledge which should be acquired during the attachment and to define those aspects of the trainee’s performance to be assessed at the end of the attachment.The agreement needs regular review by both parties, including a formal review at mid-attachment. However, plans may have to change according to circumstances. For example, a target may become unrealistic due to unforeseen circumstances such as ill health or prove to be overly optimistic. At the end of the attachment, the goals need to be compared with achievement. This comparison provides evidence for the summative RITA.Figure 3 is an extract from a Procedures Profile showing a trainee’s estimate (A) of their position at start of attachment and the goal for the end of the job (G).Figure 4 is an extract from a Knowledge Map which similarly compares a trainee’s estimate of their position at the start and finish of an attachment.Performance-based assessmentsA cornerstone of OCAP is the collection of performance-based assessments. These are formal, structured assessments of clinical activity (including surgical performance). They take a holistic approach, including understanding a problem, communicating, planning, operating and ensuring clear post-operative instructions.Performance-based assessments identify and capture relevant activity, which is performed in sufficient numbers often enough to allow reliable measurement. They allow data capture through observations of a series of similar events. This is akin to the mini clinical examination and history taking skills (CEX) and simple practical procedures such as establishing an intravenous line which are based on work by Norcini et al.3 They found that snapshots of clinical activity can be used reliably provided about 15 similar but unrelated events are assessed by properly trained observers. The Royal College of Physicians expects about 40 CEX’s to be provided over the five years of Higher Medical Training (personal communication).OCAP uses a Delphi method (consensus of an expert group) of validating performance-based assessment. We identified 18 activities which were felt by a nominated group of experts to be representative of trauma and orthopaedic surgery. These include work in the wards and clinics, and commonly performed operations (Table I). Each operation is split into domains, including consent, common to pre-operative planning, preparation and surgical technique. Each domain has further components, the majority being all the performance-based assessments and all can be related to the outcomes defined in ‘Good Clinical Practice’.4 In each domain the trainee is required to demonstrate evidence of knowledge and skills (Table II).In order to validate the performance-based assessment and its domains, meetings of trainers and trainees took place in two regions of the United Kingdom. These were followed up by postal questionnaires. The overwhelming response was that the instruments are relevant, easily usable and supported the impression of the Delphic groups, confirming that the identified performance-based assessments are realistic.More recently we correlated the current performance-based assessment with the frequency of activity reported in the National Trauma and Orthopaedic Log Book.5Table III shows that in general the Delphi method of identifying commonly performed procedures is confirmed. However, there are problems ensuring that the procedures are performed often enough to permit reliable interpretation of data. For example, there was strong consensus that the application of an external fixator contains unique competencies. However, trainees seldom encounter this procedure outside a specialist attachment. As Sher et al6 point out, unless all training opportunities are used it would be difficult adequately to assess competence in many procedures.The strengths of OCAP include a consolidation sheet which not only lists procedures but also maps the domains of competency (Fig. 5). This permits a training director to record experience of core procedures and identify progress in the domains. For example, a trainee may be performing well technically but be consistently identified as a poor communicator. Such observations, especially when made by different observers, provide powerful information for appraisal.The current status of OCAPOCAP instruments have been reviewed and triangulation studies indicate they are valid. Early work shows they are reliable and further inter-observer studies are under way. Trainers and trainees have been provided with a structure based on conventional educational instruments and given a means of mapping back to core competencies. The evolving synergy with the orthopaedic log book gives strong supporting evidence of training activity.We now have a curriculum which meets the standards prescribed by national validating bodies and uses practicable language and taxonomy for trainers and trainees.Table I. List of performance-based assessments1Clinic activity in trauma and orthopaedics2Debridement of a wound3Carpal tunnel decompression4Digital and palmar fasciectomy5Diagnostic arthroscopy and simple arthroscopic procedures6Total knee replacement7First ray surgery to the foot8Total hip replacement9Lumbar discectomy10Compression hip screw for intertrochanteric fracture neck of femur11Hemiarthroplasty intracapsular fracture neck of femur12Application of limb external fixator13Operative fixation of Weber B fracture of ankle14Fixation of patella fractures by tension band wiring15Fixation olecranon fractures by tension band wiring16Intramedullary nailing for femoral or tibial shaft fractures17Tendon repair18Closed management of fracturesTable II. The core content of performance-based assessmentsIConsent C1Demonstrates sound knowledge of (contra) indications C2Demonstrates sound knowledge of complications of surgery C3Demonstrates awareness of specific problems at surgery generated by the disease being treated C4Explains the peri-operative process to the patient and/or relatives and checks understanding C5Explains likely outcome and time to recovery and checks understandingIIPre-operative planning PL1Demonstrates recognition of anatomical and pathological abnormalities and operative strategy to deal with these PL2Ability to make reasoned choice of appropriate device (if any) using appropriate investigations e.g. x-rays PL3Checks equipment and device requirements with operating room staff PL4Where applicable ensures the limb is markedIIIPre-operative preparation PR1Ensures proper and safe positioning of the patient on the operating table PR2Ensures devices e.g. diathermy and tourniquet are deployed safely PR3Arranges for and deploys supporting equipment e.g. imaging intensifiers effectively PR4Adequately prepares a sterile operating fieldIVExposure and closure E1Demonstrates knowledge of optimum skin incision E2Demonstrates respect for soft tissues including skin E3Achieves an adequate exposure and identifies all structures correctly E4Completes a sound reconstruction E5Protects the wound properly with dressings and splintsVIntra-operative technique IT1Follows a logical sequence or protocol for the procedure IT2Adheres to hospital protocols and policies IT3Anticipates and responds appropriately to variation IT4+See individual performance-based assessments for detailVIClosing the loop CL1Ensures the patient is transferred from operating table to bed CL2Construct a clear operation note, retaining the equipment documentation and checking x-rays where appropriate CL3Gives documented and sensible post-operative instructions CL4+See individual performance-based assessments for detailTable III. An example of a logbook correlationASTSSTUPTTotal exposure in years 1 to 6Total actually performed in years 1 to 6* ORIF, open reduction and internal fixation† MUA, manipulation under anaesthesiaA, assisted; STS, surgeon with trainer scrubbed; STU, surgeon with trainer present but unscrubbed; P, surgeon with trainer available but not in the operating room; T, teaching junior surgeonAll elective in Deanery X35331413719071001641All elective in Deanery X39122384247209655541Clinic activity in trauma and orthopaedics2Debridement of a wound3Hand – carpal tunnel decompression46440660504Hand Dupuytren surgery9714021125Knee arthroscopy with procedure242123381107825Knee arthroscopy9101117148385Shoulder and elbow – shoulder arthroscopy with procedure1071101995Shoulder and elbow – shoulder arthroscopy22110646Joint replacement knee51395121108567Foot and ankle – hallux valgus surgery131017132187Foot and ankle – lesser toe surgery5626019147Foot and ankle – midfoot and hindfoot surgery1351302298Joint replacement hip62384121117549Spine – discectomy72000929Spine – decompression620008210Trauma – hip fractures (intracapsular)8137598957911Trauma – hip fractures (extracapsular)3674710736012Trauma – application of external fixator55151171113Trauma – ORIF* ankle5104334564714Trauma – ORIF patella110204315Trauma – ORIF olecranon1215110816Trauma – femoral nails69381272016Trauma – tibial nails36161171317Hand – tendon repair331131211718Trauma – MUA† + Kirschner wiring3485988271Fig. 1 Miller’s pyramid.Fig. 2 Diagram of the Orthopaedic Competence Assessment Project system and tools.Fig. 3 An extract from Procedures Profile showing trainee’s estimate of their position at start (A) and goal (G) for end of attachment (1, know about; 2, seen; 3, managed with help; 4, managed; 5, confident to manage).Fig. 4 An extract from Knowledge Map showing trainee’s estimate (A) of their position at start and agreed goal (G) for end of attachment (1, knows of; 2, knows basic concepts; 3, knows generally; 4, knows specifically and broadly).Fig. 5 An example of the performance-based assessment consolidation sheetThe authors acknowledge the contribution of the members of the OCAP steering group (Tony Banks, Peter Briggs, Nick Clarke, Chris Howell, John Edge, Alan Norrish, Phil Turner, Keith Willett) and numerous trainers and trainees who continue to make invaluable contributions to the project.References1 Miller GE. The assessment of clinical skills/competence/performance. Acad Med 1990;65(9 Suppl):63–7. Crossref, Medline, ISI, Google Scholar2 Norman G. Examining the examination: Canadian versus US certification exam. Can Assoc Radiol J 2000;51:208–9. Medline, ISI, Google Scholar3 Norcini JJ, Blank LL, Duffy FD, Fortna Gregory S. The mini-CEX: a method for assessing clinical skills. Ann Intern Med 2003;138:476–81. Crossref, Medline, ISI, Google Scholar4 Good medical practice. Third ed, May 2001. http://www.gc-uk.org (accessed 16.6.05). Google Scholar5 National Orthopaedic and Trauma Log Book. Faculty of Health Informatics, Royal College of Surgeons of Edinburgh. Google Scholar6 Sher L, Reed M, Calvert P, Wallace A, Lamb A. Influencing the national training agent: the UK & Ireland orthopaedic eLogbook. J Bone Joint Surg [Br] 2005;87-B: 1182–6. Link, Google ScholarFiguresReferencesRelatedDetailsCited byAvaliação do desempenho dos residentes em ortopedia na prática clínica20 April 2022 | Revista Brasileira de Ortopedia, Vol. 57, No. 06Surgical portfolios: A systematic scoping reviewSurgery in Practice and Science, Vol. 10Assessing Operative Skill in the Competency-based Education Era8 October 2021 | Annals of Surgery, Vol. 275, No. 4Proficiency and Competency Assessment in Surgical Training6 May 2022The usefulness of work-based assessments in higher surgical training: A systematic reviewInternational Journal of Surgery, Vol. 94Challenges faced by orthopaedic trainees during the Covid-19 pandemic – An Irish perspectiveThe Surgeon, Vol. 19, No. 5Procedure-based assessments: the past, the present and the future26 July 2019 | Postgraduate Medical Journal, Vol. 96, No. 1131Analysis of Tools Used in Assessing Technical Skills and Operative Competence in Trauma and Orthopaedic Surgical Training1 June 2020 | JBJS Reviews, Vol. 8, No. 6Video-Based Assessment in Surgical Education: A Scoping ReviewJournal of Surgical Education, Vol. 76, No. 6Development and Evaluation of a New Formative Assessment of Surgical PerformanceJournal of Surgical Education, Vol. 75, No. 5Orthopaedic Education in Other Countries26 July 2017Orthopaedic Resident Assessment: Measuring Skills in Both Knowledge and Technical Skills26 July 2017Desarrollo de plan de entrenamiento con simulación de artroscopía para residentes de ortopediaActa Ortopédica Mexicana, Vol. 32, No. 5Existing and Future Educational Needs in Graduate and Postgraduate EducationThe Journal of Sexual Medicine, Vol. 14, No. 4Resident education in orthopaedic trauma the future role of competency-based medical educationM. T. Nousiainen, S. A. McQueen, J. Hall, W. Kraemer, P. Ferguson, J. L. Marsh, R. R. Reznick, M. R. Reed, R. Sonnadara1 October 2016 | The Bone & Joint Journal, Vol. 98-B, No. 10Operative Experience vs. Competence: A Curriculum Concordance and Learning Curve AnalysisJournal of Surgical Education, Vol. 73, No. 4Learning Curves in Arthroplasty in Orthopedic TraineesJournal of Surgical Education, Vol. 73, No. 4A Comprehensive, High-Quality Orthopedic Intern Surgical Skills ProgramJournal of Surgical Education, Vol. 73, No. 4Examining the barriers to meaningful assessment and feedback in medical trainingThe American Journal of Surgery, Vol. 211, No. 2Assessing Arthroscopic Skills Using Wireless Elbow-Worn Motion SensorsJournal of Bone and Joint Surgery, Vol. 97, No. 13Procedure-based assessments in trauma and orthopaedic training – The trainees’ perspective4 September 2014 | Medical Teacher, Vol. 37, No. 5The validity and reliability of workplacebased assessments in surgical trainingThe Bulletin of the Royal College of Surgeons of England, Vol. 97, No. 3Advances in arthroscopy—indications and therapeutic applications28 October 2014 | Nature Reviews Rheumatology, Vol. 11, No. 2Accuracy and awareness of lag screw placement when using the dynamic hip screw for fracture fixation1 September 2014 | Trauma, Vol. 17, No. 1Opinions of UK Orthopaedic Trainees Regarding Training, Assessment and the Trainee FeeThe Bulletin of the Royal College of Surgeons of England, Vol. 96, No. 10Orthopaedic Education in the United KingdomClinical Orthopaedics & Related Research, Vol. 472, No. 6Role of Simulation in Arthroscopy TrainingSimulation in Healthcare: The Journal of the Society for Simulation in Healthcare, Vol. 9, No. 2A Novel Approach to Assessing Technical Competence of Colorectal Surgery ResidentsAnnals of Surgery, Vol. 258, No. 6Reflections on current methods for evaluating skills during joint replacement surgery A scoping reviewR. Sonnadara, S. McQueen, P. Mironova, O. Safir, M. Nousiainen, P. Ferguson, B. Alman, W. Kraemer, R. Reznick1 November 2013 | The Bone & Joint Journal, Vol. 95-B, No. 11Workplace-based assessment in surgical training: experiences from the Intercollegiate Surgical Curriculum Programme8 May 2013 | ANZ Journal of Surgery, Vol. 83, No. 6The United Kingdom and Ireland Trauma & Orthopaedic eLogbook—An evidence base for enhancing trainingThe Surgeon, Vol. 10, No. 5Making sense of work-based assessment: ask the right questions, in the right way, about the right things, of the right people13 December 2011 | Medical Education, Vol. 46, No. 1Evaluation of procedure-based assessment for assessing trainees' skills in the operating theatre24 November 2010 | British Journal of Surgery, Vol. 98, No. 3Assessing the Surgeonʼs Technical Skills: Analysis of the Available ToolsAcademic Medicine, Vol. 85, No. 5Orthopaedic Surgery: Current Trends and Recent InnovationsCompetencies for a Canadian orthopaedic surgery core curriculumV. M. R. Wadey, P. Dev, R. Buckley, D. Walker, D. Hedden1 December 2009 | The Journal of Bone and Joint Surgery. British volume, Vol. 91-B, No. 12Orthopaedic training experience in the UK and Ireland: An analysis using the elogbookThe Surgeon, Vol. 7, No. 4Workplace-based assessment: assessing technical skill throughout the continuum of surgical trainingANZ Journal of Surgery, Vol. 79, No. 3Trauma experience in the UK and Ireland: Analysis of orthopaedic training using the FHI eLogbookInjury, Vol. 39, No. 8Validation of orthopaedic bench models for trauma surgeryJ. J. H. Leong, D. R. Leff, A. Das, R. Aggarwal, P. Reilly, H. D. E. Atkinson, R. J. Emery, A. W. Darzi1 July 2008 | The Journal of Bone and Joint Surgery. British volume, Vol. 90-B, No. 7Training a Competent SurgeonClinics in Podiatric Medicine and Surgery, Vol. 24, No. 1 Vol. 87-B, No. 9 Metrics History Published online 1 September 2005 Published in print 1 September 2005 InformationCopyright © 2005, The British Editorial Society of Bone and Joint Surgery: All rights reservedPDF download