Portal de Periódicos da CAPES

Comparing Patient-management Skills of Referred Physicians and Non-referred Physicians on a Computer-based Case-simulation Examination

2001; Lippincott Williams & Wilkins; Volume: 76; Issue: Supplement Linguagem: Inglês

10.1097/00001888-200110001-00009

1938-808X

Laurel Sample, Tony LaDuca, Cynthia Leung, Richard Hawkins, Margaret MacKrell Gaglione, William Liston, André F. De Champlain, Jeanne Guernsey, ANDREA CICCONE, MARTHA ILLIGE, Elizabeth J. Korinek,

Radiology practices and education

Diagnostic assessment of practicing physicians is modest in scale, and normative data for practicing physicians are virtually nonexistent in the U.S. It is the responsibility of state licensing boards to ensure the competency of their licensed physicians. Increasingly, state licensing boards are responding to the concerns of patients, communities, and hospitals regarding the continued clinical competence of physicians. 1 Precipitating events may involve deficiencies in prescription of pharmacotherapy, patient management, and interpersonal skills. The Assessment Center Program (ACP) is a joint activity of the National Board of Medical Examiners (NBME) and the Federation of State Medical Boards (FSMB). It is a program of standardized tests intended to assist state medical boards (SMBs) in their diagnostic assessment of physicians whose clinical competence has been questioned. This type of personalized assessment of physicians has increased since the 1980s, with a sizeable proportion of the work being done in Canada. 2,3 The target candidate of the ACP is a physician referred by a state medical board because of concerns about his or her clinical competence. In a study of doctors in California 4 it was found that disciplined doctors were less likely to be board certified than were a matched group of non-disciplined physicians. Age has also been reported to be a predictor of physicians' being identified for deficiencies in clinical competence. 5 Under a special arrangement with Colorado Personalized Education for Physicians (CPEP), referred doctors are tested at the CPEP facility, located near Denver, Colorado. The full CPEP assessment for generalist doctors requires three days of multi-method testing. For the last three and a half years, ACP has been administering computer-based case simulations (CCSs) to referred doctors at CPEP and separately to a comparison group of physicians. 6–8 The CCSs are a Windows-based simulation of the health care environment designed to assess a physician's skills in patient management. A presentation screen displayed at the start of a case describes the patient, the initial history and vitals, and the location of the clinical encounter (emergency room or outpatient office). From this point, the case is prompt-free and therefore examinee-driven. As one of its goals, the ACP has been exploring the utility of CCSs in the diagnostic assessment of referred doctors. It is anticipated that the case means for the comparison group will be higher than will be those of the referred group. In addition, we have been examining the relationships between selected demographic variables and CCS performances of referred physicians. Method Referred group. At the time of writing, 42 referred group physicians had taken the Windows-based CCS exam since August 1999 as part of the CPEP assessment program. Physicians are referred for a number of reasons, but most referrals involve some concern regarding clinical competence. The mean age of this referred group was 53, and the range was 30 to 76 years. There were eight women in the referred group. All referred physicians except two reported internal medicine, family practice, or general practice as their specialties. Two physicians were residents and therefore not board certified. Of the 40 remaining referred licensed physicians, 23 (58%) were not board certified. Comparison group. In the fall of 1998, a convenience sample of 32 Philadelphia-area, board-certified physicians completed the same eight-case CCS exam. All testing took place at the NBME offices, and the physicians were compensated. Score reports were not provided. Additionally, between July 1999 and October 2000, 49 staff physicians and residents from the emergency medicine, internal medicine, and general surgery departments at the Naval Medical Center in Portsmouth, Virginia (NMCP), volunteered to take the same eight-case CCS exam. Compensation was not provided, and individual score reports were not provided. The total number of physicians in the comparison group was 81. The mean age of the doctors in the comparison group was 38 years, and the range was 27 to 63 years. The comparison group was approximately 15 years younger, on average, than was the referred group. Specialties consisted of internal medicine (48), family practice (20), emergency medicine (8), and general surgery (5). All physicians in the comparison group were board certified with the exception of 14 residents from NMCP. The assessment. In CCSs, the physician is responsible for addressing the patient's concern, diagnosing the problem, and providing treatment. The physician is instructed to assume complete responsibility as the simulated patient's primary care physician. The physician types all orders on a simulated order sheet. The patient's condition evolves as a function of the physician's management choices. At the end of each case the physician is prompted to enter the primary diagnosis. Each action is recorded in a text file called a transaction list, which becomes the source for scoring. All actions are classified as one of three levels of benefit (i.e., most, more, least) or as one of three levels of risk. Raw scores are computed and then weighted to reflect expert ratings. 9,10 Scores are reported to CPEP on a scale of 1 to 9, with higher scores indicating better patient management skills. CCSs have recently been incorporated within the United States Medical Licensing Examination (USMLE) Step 3. Both referred and comparison-group-physicians took the same eight-case CCS exam, which was designed to reflect generalist practice. Specifically, the exam contained six medicine cases, one pediatric case, and one ob—gyn case. We are unable to describe further the content of these cases because of security concerns. All physicians took three orientation cases prior to completing the scored cases. A proctor remained in the testing room to answer technical questions regarding navigating the simulation. Answers to questions related to medical content were not provided. For referred doctors who requested assistance (because of unfamiliarity with computers), the CPEP proctor served as a scribe at the keyboard. Cases were presented in the same order for all examinees. Referred physicians completed a post—CCS interview with a medical director trained in the use of CCS. The medical director conducted a clinical interview to evaluate the decision-making process the referred physician used to manage the simulated patients. Results Case scores. Table 1 displays the descriptive statistics for referred and comparison groups. The case means of the referred group were lower than those of the comparison group for all eight cases. The comparison group's case means ranged from 4.40 for the pediatric case to 6.54 for a medicine case (case 4). The referred group's case means ranged from 3.41, also for the pediatric case, to 5.20 for a medicine case (case 3). To test whether case means differed across the groups, t-tests were computed for independent groups. Mean differences were statistically significant at the Bonferroni-adjusted alpha value of .006 (.05/8 cases) for five cases: cases 1, 2, 4, 6, and 8. All are “medicine-focused” cases except case 2, the one pediatric case. The largest mean difference (2.3) was seen for case 4. The mean difference for the grand mean (across all eight test cases) was also significant (t (121) = −5.970, p = .000).TABLE 1: CCS Performances of Physicians in the Referred and Comparison Groups, Cases 1–8Timing. Except for the pediatric case, the referred group took more time (real time spent on the case), on average, to complete each test case than did the comparison group. Independent t-tests were computed to test whether the case mean times were significantly different from those of the comparison group. Six mean differences were significant at a Bonferroni-adjusted alpha value of .006, specifically those for cases 1, 3, 4, 5, 7, and 8. It is possible that the referred group spent more time on the cases because of the context of their performance; that is, the referred physician was aware that his or her CCS performance was important to the overall evaluation by CPEP. The Pearson correlation between the mean time in minutes and the CCS grand mean for the referred group was significant at −.33 (p = .032). The Pearson correlation between the mean time and the CCS grand mean for the comparison group was .11. CCS scores and demographics of referred doctors. It has been reported that doctors who are older and not certified in a specialty are at increased risk of being identified as having deficiencies in clinical competence. 4,5 We examined this relationship by comparing the performance of the board-certified referred doctors (n = 17) on the CCSs with the scores of the non-certified referred doctors (n = 25). The mean age of the certified referred doctors was 48.3 years and that of the non-certified referred doctors was 55.8 years. The Pearson correlation between age and mean CCS score was significant at −.40 (p = .011). Older referred physicians performed less well on CCSs than did younger referred physicians. This is in line with the fact that many of the older referred physicians are not board certified. In fact, the correlation between age and certification (yes/no) was .37. The CCS mean for board-certified referred physicians was 5.02, and the CCS mean for non—board-certified referred physicians was 4.17. An independent-samples t-test was computed to test whether the mean difference in CCS mean scores (over all eight cases) between board-certified and non—board-certified physicians was significant. The mean difference was significant (t (40) = −2.96, p = .005), with board-certified physicians outperforming their counter-parts. Because age and certification are correlated, we then performed a univariate analysis of variance with certification as the fixed factor, mean CCS score as the dependent variable, and age as the covariate. Certification status was significant in this model. Discussion We have reported on the performances of referred and comparison-group physicians on an eight-case computer-based case simulation (CCS) examination of patient management skills. The purpose of this study was to examine the usefulness of CCSs in assessing patient management skills of referred physicians. Descriptive statistics of CCS performance show that the referred physicians performed less well than did the comparison-group physicians on all eight cases. This was the anticipated outcome, given that the referred physicians' competence had been called into question. The referred physicians scored significantly lower than did the comparison-group physicians on five cases. There was a significant difference in mean CCS scores (over all cases) between board-certified and non—board-certified referred physicians. Other researchers 4 have observed this relationship. Overall, physicians in the referred group took more time than comparison-group physicians took to complete each case. This may be related to the referred group's being older than the comparison group and less comfortable with Windows-based computer programs. In fact, CPEP reports that many referred physicians stated they had no computer experience and requested the proctor to enter orders during the examination. The results of this study provide evidence that CCS is useful in the assessment of the patient management skills of referred physicians. The results show a pattern of comparison-group physicians' performing better, on average, then referred group physicians do. Board-certified physicians perform better on CCSs than do non—board-certified physicians, and there was a significant, negative correlation between mean CCS score and age of the referred physician. It is the goal of this program to continue to collect comparison-group data on CCSs. Currently, the comparison group consists of two combined convenience samples. We recognize that this convenience sample of 81 comparison physicians is not adequate for generalized statements of performance. However, for those organizations involved in assessing physicians' performances, this study provides formative data regarding the patient-management performances of competent, practicing physicians as assessed by CCSs. A database of such information will provide a standard with which to compare referred physicians.