Portal de Periódicos da CAPES

Sidestepping Superstitious Learning, Ambiguity, and Other Roadblocks: A Feedback Model of Diagnostic Problem Solving

2008; Elsevier BV; Volume: 121; Issue: 5 Linguagem: Inglês

10.1016/j.amjmed.2008.02.003

1555-7162

Jenny W. Rudolph, J. Bradley Morrison,

Innovations in Medical Education

A central argument of Drs. Eta S. Berner and Mark L. Graber's review1Berner E. Graber M.L. Overconfidence as a cause of diagnostic error in medicine.Am J Med. 2008; 121: S2-S23Abstract Full Text Full Text PDF PubMed Scopus (606) Google Scholar is that feedback processes are crucial to enhancing or inhibiting the quality of diagnostic problem solving over time. Our goal is to enrich the conversation about diagnostic problem solving by presenting an explicit model of the feedback processes inherent in improving diagnostic problem solving. We present a simple, generic model of the fundamental feedback processes at play in calibrating or improving diagnostic problem-solving skill over time. To amplify these key processes, this commentary draws on a 50-year evidence and theory base from the discipline of system dynamics.2Forrester J.W. Industrial Dynamics. Productivity Press, Portland, OR1994Google Scholar, 3Sterman J. Business Dynamics: Systems Thinking and Modeling for a Complex World. Irwin/McGraw Hill, Homewood, IL2000Google ScholarUsing Berner and Graber's analysis1Berner E. Graber M.L. Overconfidence as a cause of diagnostic error in medicine.Am J Med. 2008; 121: S2-S23Abstract Full Text Full Text PDF PubMed Scopus (606) Google Scholar of the challenges of feedback and calibration as a starting point, we depict how feedback loops can operate in a robust or benign manner to support and improve immediate and long-term diagnostic problem solving. Drawing on insights from research on how people manage problem solving that involves dynamic feedback, we then describe how this process is likely to break down. Finally, leverage points for improving diagnostic problem solving and avoiding error are provided.To improve diagnostic problem solving, practitioners and researchers need to move away from viewing diagnosis as a "one-shot deal." When diagnosis is perceived as a stand-alone, discrete episode of judgment, the solutions suggested to resolve error focus on reducing cognitive biases and increasing expertise and vigilance at the individual clinician level. It is not that such recommendations have no merit, but simply that they are only a small piece of a much larger repertoire of possible solutions that come into sight when we regard diagnostic problem solving as a recursive, feedback-driven process. Put differently, rather than viewing diagnosis as an event or episode, we suggest emphasizing it as an active, ongoing practice in which clinicians revise and redraft their conclusions over time.4Hogarth R.M. Beyond discrete biases: functional and dysfunctional aspects of judgmental heuristics.Psychol Bull. 1981; 90: 197-217Crossref Scopus (405) Google Scholar, 5Kleinmuntz D.N. Cognitive heuristics and feedback in a dynamic decision environment.Manage Sci. 1985; 31: 680-702Crossref Scopus (106) Google Scholar, 6Weick K.E. Sutcliffe K. Obstfeld D. Organizing and the process of sensemaking.Organ Sci. 2005; 16: 409-421Crossref Scopus (3621) Google ScholarWhen calibration works: an optimal feedback processFrom the moment a clinician begins a patient encounter, he/she is selecting, labeling, and processing information (e.g., symptoms, results from studies, and other data) from the client or his record. The practitioner shapes this information into a diagnosis that, in turn, influences his/her view and collection of subsequent information. Discrete decisions made without feedback have been likened to hitting a target from a distance in one try; in contrast, diagnostic problem solving is analogous to a situation where one can monitor and correct the trajectory based on feedback.4Hogarth R.M. Beyond discrete biases: functional and dysfunctional aspects of judgmental heuristics.Psychol Bull. 1981; 90: 197-217Crossref Scopus (405) Google Scholar, 5Kleinmuntz D.N. Cognitive heuristics and feedback in a dynamic decision environment.Manage Sci. 1985; 31: 680-702Crossref Scopus (106) Google ScholarPatient care is a feedback process in which the clinician makes judgments and takes actions with the intended rationale of bringing the patient closer to the desired, presumably healthier, status. This process of observing/diagnosing/treating/observing describes a balancing or goal-seeking feedback loop, in which feedback about the patient's status allows a clinician to calibrate therapy over the very short term. Although physicians may be able to adjust a diagnosis and treatment based on conversation and examination during a specific patient encounter, Berner and Graber1Berner E. Graber M.L. Overconfidence as a cause of diagnostic error in medicine.Am J Med. 2008; 121: S2-S23Abstract Full Text Full Text PDF PubMed Scopus (606) Google Scholar argue that lack of timely or consistent feedback on the accuracy and quality of diagnoses over the long term makes it difficult for them to improve their diagnostic problem-solving skills over time. Once out of medical school and residency, most physicians operate in a "no news is good news" mode, believing that unless they hear about problems, the diagnoses they have made are correct. Berner and Graber invoke a well-established fact of learning theory, namely, that improvement is nearly impossible without accurate and timely feedback. Improving one's diagnostic problem-solving skill, they argue, requires an ability to calibrate the match between the diagnosis made and the patient's actual long-term status.The generic feedback process that would allow a clinician to calibrate and improve a key element of long-term diagnostic skill, the quality of his/her "diagnostic schemas," is depicted in Figure 1. A diagnosis is the result of applying a diagnostic schema to information about the patient as the clinician perceives it. Schema is a term from cognitive science referring to a person's mental model, or internal image of a given professional domain or area.7Gentner D. Stevens A.L. Mental Models. Lawrence Erlbaum Associates, Hillsdale, NJ1983Google Scholar Schemas form the basis of processes such as "recognition-primed decision making" that allow clinicians to match a library of images of past experiences with the present constellation of signs and symptoms to formulate a diagnosis.8Klein G. Sources of Power: How People Make Decisions. MIT Press, Cambridge, MA1998Google ScholarThe long-term feedback process in diagnosing and treating an individual patient depicted in Figure 1, like the short-term feedback process, is a balancing or adaptive process. It is a longer-term process of learning from experience, in which the clinician adjusts the diagnostic schema for the patient by comparing expected outcomes with observed actual outcomes. To illustrate how this loop operates, we start with Diagnosis. In making a Diagnosis, the clinician employs the current Diagnostic Schema, developed through training and experience, to interpret patient information and recommend a specific course of Therapy. Based on the the therapy recommended, the clinician expects the patient's condition will evolve in a certain way to yield Expected Patient Outcomes. Ideally, after some time has elapsed for the therapy to take effect, the clinician sees the actual Observed Patient Outcomes. Comparing the Observed Patient Outcomes with Expected Patient Outcomes (this comparison is often tacit or unconscious), the clinician then identifies the Patient Outcome Gap, which stimulates Updating or revising of the existing Diagnostic Schema. In optimal settings, this schema accounts well for the patient's history, constellation of signs and symptoms, and treatment results. To the extent that the diagnostic schema improves, the quality of the clinican's diagnoses at later patient encounters also improves.Barriers to improving diagnostic problem-solving skill over timeBerner and Graber1Berner E. Graber M.L. Overconfidence as a cause of diagnostic error in medicine.Am J Med. 2008; 121: S2-S23Abstract Full Text Full Text PDF PubMed Scopus (606) Google Scholar and other contributors to this supplement note that a simple but significant barrier to enhancing diagnostic problem-solving skill over time is that the link between therapy and observed patient outcomes often is nonexistent. In the absence of significant information provided by autopsy, data from downstream clinicians, or tailored quality measures, clinicians are unable to update their diagnostic schema. Several decades of research on how people manage information in the face of dynamic feedback reveal other challenges as well. We highlight 3 significant barriers to updating diagnostic schema in a sound way: delays, ambiguous feedback, and superstitious learning.2Forrester J.W. Industrial Dynamics. Productivity Press, Portland, OR1994Google Scholar, 9Sterman J.D. Learning from evidence in a complex world.Am J Public Health. 2006; 96: 505-514Crossref PubMed Scopus (571) Google Scholar, 10Repenning N.P. Sterman J.D. Capability traps and self-confirming attribution errors in the dynamics of process improvement.Adm Sci Q. 2002; 47: 265-295Crossref Scopus (304) Google ScholarDelaysFor both an immediate patient encounter and the long-term process of improving and updating one's diagnostic schema, delays in feedback can cause problems. Delays slow the accumulation of evidence and create fluctuations in evidence that make it difficult to draw sound conclusions.9Sterman J.D. Learning from evidence in a complex world.Am J Public Health. 2006; 96: 505-514Crossref PubMed Scopus (571) Google Scholar Obviously, as the length of time between therapy and its impact increases, the likelihood that the physician will observe the outcome decreases. Examples of this include patients who do not experience the full consequences of the therapy or physicians who do not see the patient again, thereby rendering outcome feedback unavailable. Time delays, thus, partially explain why the link from therapy to observed patient outcomes may be so weak, as Berner and Graber1Berner E. Graber M.L. Overconfidence as a cause of diagnostic error in medicine.Am J Med. 2008; 121: S2-S23Abstract Full Text Full Text PDF PubMed Scopus (606) Google Scholar suggest.Delays compromise learning even when outcome feedback is available. Delays between cause and effect make inferences about causality far more difficult because they give rise to a characteristic of feedback systems known as dynamic complexity.2Forrester J.W. Industrial Dynamics. Productivity Press, Portland, OR1994Google Scholar, 9Sterman J.D. Learning from evidence in a complex world.Am J Public Health. 2006; 96: 505-514Crossref PubMed Scopus (571) Google Scholar In diagnostic problem solving, dynamic complexity can take the form of unexpected oscillations between desired and undesired therapeutic outcomes, amplification of certainty on the part of the clinician (e.g., fixation), and excessive or diminished commitment to particular treatments.11Rudolph J.W. Morrison J.B. Confidence, error and ingenuity in diagnostic problem solving: clarifying the role of exploration and exploitation.2007Google Scholar For example, if effects from therapy occur after the physician's felt need to move forward with patient care, he/she may pursue contraindicated interventions or drop indicated ones—continuing to intervene although curative measures have been taken or failing to intervene although treatment has been inadequate. Research repeatedly has demonstrated the failure to learn in situations with even modest amounts of dynamic complexity.9Sterman J.D. Learning from evidence in a complex world.Am J Public Health. 2006; 96: 505-514Crossref PubMed Scopus (571) Google Scholar Finally, time delays quite simply slow down the completion of the feedback loop; longer delays mean fewer learning cycles in any time period.Ambigious FeedbackAlthough a clinican may receive feedback about how his/her diagnosis and therapy has influenced the patient, effectiveness can be compromised because such feedback often is ambiguous. The primary problem is that changes in the patient's observed status caused by the physician's actions are influenced by a range of other clinical and lifestyle variables both inside and outside the clinician's control. Confusingly, data about their patients can equally support a wide variety of clinical conclusions, making it difficult for physicians to assess what actions actually work best. Controlled experimentation is almost never possible in real clinical settings. Ambiguous information invites subjective interpretation, and, like many people, physicians tend to make self-fulfilling interpretations (e.g., "The diagnosis was correct") in the face of such ambiguity, perhaps missing the opportunity to update flawed diagnostic schema.Superstitious LearningIn the face of time delays and ambiguity, superstitious learning thrives. Sterman9Sterman J.D. Learning from evidence in a complex world.Am J Public Health. 2006; 96: 505-514Crossref PubMed Scopus (571) Google Scholar relates the case of Baseball Hall of Fame hitter Wade Boggs, who ate chicken every game day for years because he had played well once following a dinner of lemon chicken. While this might seem laughable, ambiguous or weak feedback supports "strong but wrong" self-confirming attributions about what works.12Sterman J. Repenning N. Kofman F. Unanticipated side effects of successful quality programs: exploring a paradox of organizational improvement.Manage Sci. 1997; 43: 503-521Crossref Scopus (232) Google Scholar, 13Reason J. Human Error. Cambridge University Press, New York1990Crossref Google Scholar During the time gap between therapy and observed outcome, much transpires that the clinician does not directly observe. Physicians, like other people, fill in the blanks with their own superstitious explanations—conclusions that fit the data but are based on weak or spurious correlations (e.g., eating chicken improves baseball performance).The lessons of superstitious learning persist because satisfactory explanations (e.g., scurvy is an unavoidable result of lengthy sea voyages) suppress the search for better answers (e.g., scurvy results from vitamin C deficiency). Recent studies show that only about 15% of physicians' decisions are evidence based; weak or ambiguous feedback contributes to this situation by preventing physicians from learning when their self-confirming routines are inappropriate, inaccurate, or dangerous.How confidence can disrupt learningHow does such pseudolearning persist? Berner and Graber1Berner E. Graber M.L. Overconfidence as a cause of diagnostic error in medicine.Am J Med. 2008; 121: S2-S23Abstract Full Text Full Text PDF PubMed Scopus (606) Google Scholar argue that confidence or overconfidence plays a role. The feedback process we have described (Figure 1) is a balancing loop that attempts to close the gap between expected and observed patient outcomes. When that gap does not close, clinicians should seek additional or alternative data. But Berner and Graber show that often does not happen. To understand why, we introduce another feedback loop in Figure 2.Figure 2How confidence impedes calibration.Show full captionThe "R" labeled "self-confirming bias" signifies a reinforcing loop that amplifies clinicians' confidence in their current diagnostic problem-solving skill. Arrows indicate the direction of causality.View Large Image Figure ViewerDownload (PPT)To understand the impact of the self-confirming bias loop (Figure 2), the contrast between the process by which physicians ideally update their diagnostic schema and the actual one described by Berner and Graber1Berner E. Graber M.L. Overconfidence as a cause of diagnostic error in medicine.Am J Med. 2008; 121: S2-S23Abstract Full Text Full Text PDF PubMed Scopus (606) Google Scholar should be kept in mind: In the adaptive scenario, where learning occurs when Therapy influences the Observed Patient Outcomes, the physician observes these outcomes and is informed by the Patient Outcome Gap. In situations where the link between Therapy and Observed Patient Outcomes is nonexistent or weak, the Patient Outcome Gap is either unknown or unclear.Berner and Graber1Berner E. Graber M.L. Overconfidence as a cause of diagnostic error in medicine.Am J Med. 2008; 121: S2-S23Abstract Full Text Full Text PDF PubMed Scopus (606) Google Scholar argue that in the absence of such clear feedback, physicians feel little need to update their current Diagnostic Schema. Thus, a felt need for Updating declines and Confidence increases. As Confidence increases, the felt need for Updating decreases further in a reinforcing cycle. While calibrating or improving one's diagnostic problem solving already faces the significant challenges posed by missing or ambiguous feedback, lack of feedback also triggers a vicious reinforcing cycle that erroneously amplifies confidence. It is this reinforcing confidence cycle that is the nail in the coffin of robust learning that would allow clinicians to improve diagnostic problem solving over time.In conclusion, we ask, "Does a doctor who has practiced for 30 years have a lower rate of diagnostic error than a doctor who has practiced for 5 years?" If the feedback processes we have described were functioning optimally, the answer should be a resounding "Yes!" Based on the review by Berner and Graber,1Berner E. Graber M.L. Overconfidence as a cause of diagnostic error in medicine.Am J Med. 2008; 121: S2-S23Abstract Full Text Full Text PDF PubMed Scopus (606) Google Scholar however, the answer is unclear. To contribute to policies that reduce the rate of diagnostic errors, we have highlighted 2 faces of the balancing feedback processes that drive diagnostic problem solving. These processes can function adaptively, improving diagnostic schema over time and problem solving during a patient encounter. If physicians in practice for 30 years had a notably lower rate of diagnostic error than their rookie counterparts, it would indicate these loops were functioning well. But these processes break down when crucial links are weakened or do not function at all. When this happens, adaptive learning processes are further hobbled by a vicious reinforcing cycle that maintains or amplifies a misplaced sense of confidence.If, as scholars of human judgment have argued, overconfidence is a highly ingrained human trait, trying to reduce it is a Sisyphean task.14Bazerman M.H. Judgment in Managerial Decision Making. John Wiley and Sons, New York1986Google Scholar The leverage points for this uphill task lie, as our colleagues in this supplement have argued, in systematically assuring that downstream feedback is (1) available and (2) as unambiguous as possible so that physicians experience a felt need to update their diagnostic schema. It is this pressure to update that can weaken the reinforcing confidence loop.Author disclosuresThe authors report the following conflicts of interest with the sponsor of this supplement article or products discussed in this article.Jenny W. Rudolph, PhD, has no financial arrangement or affiliation with a corporate organization or a manufacturer of a product discussed in this article.J. Bradley Morrison, PhD, has no financial arrangement or affiliation with a corporate organization or a manufacturer of a product discussed in this article. A central argument of Drs. Eta S. Berner and Mark L. Graber's review1Berner E. Graber M.L. Overconfidence as a cause of diagnostic error in medicine.Am J Med. 2008; 121: S2-S23Abstract Full Text Full Text PDF PubMed Scopus (606) Google Scholar is that feedback processes are crucial to enhancing or inhibiting the quality of diagnostic problem solving over time. Our goal is to enrich the conversation about diagnostic problem solving by presenting an explicit model of the feedback processes inherent in improving diagnostic problem solving. We present a simple, generic model of the fundamental feedback processes at play in calibrating or improving diagnostic problem-solving skill over time. To amplify these key processes, this commentary draws on a 50-year evidence and theory base from the discipline of system dynamics.2Forrester J.W. Industrial Dynamics. Productivity Press, Portland, OR1994Google Scholar, 3Sterman J. Business Dynamics: Systems Thinking and Modeling for a Complex World. Irwin/McGraw Hill, Homewood, IL2000Google Scholar Using Berner and Graber's analysis1Berner E. Graber M.L. Overconfidence as a cause of diagnostic error in medicine.Am J Med. 2008; 121: S2-S23Abstract Full Text Full Text PDF PubMed Scopus (606) Google Scholar of the challenges of feedback and calibration as a starting point, we depict how feedback loops can operate in a robust or benign manner to support and improve immediate and long-term diagnostic problem solving. Drawing on insights from research on how people manage problem solving that involves dynamic feedback, we then describe how this process is likely to break down. Finally, leverage points for improving diagnostic problem solving and avoiding error are provided. To improve diagnostic problem solving, practitioners and researchers need to move away from viewing diagnosis as a "one-shot deal." When diagnosis is perceived as a stand-alone, discrete episode of judgment, the solutions suggested to resolve error focus on reducing cognitive biases and increasing expertise and vigilance at the individual clinician level. It is not that such recommendations have no merit, but simply that they are only a small piece of a much larger repertoire of possible solutions that come into sight when we regard diagnostic problem solving as a recursive, feedback-driven process. Put differently, rather than viewing diagnosis as an event or episode, we suggest emphasizing it as an active, ongoing practice in which clinicians revise and redraft their conclusions over time.4Hogarth R.M. Beyond discrete biases: functional and dysfunctional aspects of judgmental heuristics.Psychol Bull. 1981; 90: 197-217Crossref Scopus (405) Google Scholar, 5Kleinmuntz D.N. Cognitive heuristics and feedback in a dynamic decision environment.Manage Sci. 1985; 31: 680-702Crossref Scopus (106) Google Scholar, 6Weick K.E. Sutcliffe K. Obstfeld D. Organizing and the process of sensemaking.Organ Sci. 2005; 16: 409-421Crossref Scopus (3621) Google Scholar When calibration works: an optimal feedback processFrom the moment a clinician begins a patient encounter, he/she is selecting, labeling, and processing information (e.g., symptoms, results from studies, and other data) from the client or his record. The practitioner shapes this information into a diagnosis that, in turn, influences his/her view and collection of subsequent information. Discrete decisions made without feedback have been likened to hitting a target from a distance in one try; in contrast, diagnostic problem solving is analogous to a situation where one can monitor and correct the trajectory based on feedback.4Hogarth R.M. Beyond discrete biases: functional and dysfunctional aspects of judgmental heuristics.Psychol Bull. 1981; 90: 197-217Crossref Scopus (405) Google Scholar, 5Kleinmuntz D.N. Cognitive heuristics and feedback in a dynamic decision environment.Manage Sci. 1985; 31: 680-702Crossref Scopus (106) Google ScholarPatient care is a feedback process in which the clinician makes judgments and takes actions with the intended rationale of bringing the patient closer to the desired, presumably healthier, status. This process of observing/diagnosing/treating/observing describes a balancing or goal-seeking feedback loop, in which feedback about the patient's status allows a clinician to calibrate therapy over the very short term. Although physicians may be able to adjust a diagnosis and treatment based on conversation and examination during a specific patient encounter, Berner and Graber1Berner E. Graber M.L. Overconfidence as a cause of diagnostic error in medicine.Am J Med. 2008; 121: S2-S23Abstract Full Text Full Text PDF PubMed Scopus (606) Google Scholar argue that lack of timely or consistent feedback on the accuracy and quality of diagnoses over the long term makes it difficult for them to improve their diagnostic problem-solving skills over time. Once out of medical school and residency, most physicians operate in a "no news is good news" mode, believing that unless they hear about problems, the diagnoses they have made are correct. Berner and Graber invoke a well-established fact of learning theory, namely, that improvement is nearly impossible without accurate and timely feedback. Improving one's diagnostic problem-solving skill, they argue, requires an ability to calibrate the match between the diagnosis made and the patient's actual long-term status.The generic feedback process that would allow a clinician to calibrate and improve a key element of long-term diagnostic skill, the quality of his/her "diagnostic schemas," is depicted in Figure 1. A diagnosis is the result of applying a diagnostic schema to information about the patient as the clinician perceives it. Schema is a term from cognitive science referring to a person's mental model, or internal image of a given professional domain or area.7Gentner D. Stevens A.L. Mental Models. Lawrence Erlbaum Associates, Hillsdale, NJ1983Google Scholar Schemas form the basis of processes such as "recognition-primed decision making" that allow clinicians to match a library of images of past experiences with the present constellation of signs and symptoms to formulate a diagnosis.8Klein G. Sources of Power: How People Make Decisions. MIT Press, Cambridge, MA1998Google ScholarThe long-term feedback process in diagnosing and treating an individual patient depicted in Figure 1, like the short-term feedback process, is a balancing or adaptive process. It is a longer-term process of learning from experience, in which the clinician adjusts the diagnostic schema for the patient by comparing expected outcomes with observed actual outcomes. To illustrate how this loop operates, we start with Diagnosis. In making a Diagnosis, the clinician employs the current Diagnostic Schema, developed through training and experience, to interpret patient information and recommend a specific course of Therapy. Based on the the therapy recommended, the clinician expects the patient's condition will evolve in a certain way to yield Expected Patient Outcomes. Ideally, after some time has elapsed for the therapy to take effect, the clinician sees the actual Observed Patient Outcomes. Comparing the Observed Patient Outcomes with Expected Patient Outcomes (this comparison is often tacit or unconscious), the clinician then identifies the Patient Outcome Gap, which stimulates Updating or revising of the existing Diagnostic Schema. In optimal settings, this schema accounts well for the patient's history, constellation of signs and symptoms, and treatment results. To the extent that the diagnostic schema improves, the quality of the clinican's diagnoses at later patient encounters also improves. From the moment a clinician begins a patient encounter, he/she is selecting, labeling, and processing information (e.g., symptoms, results from studies, and other data) from the client or his record. The practitioner shapes this information into a diagnosis that, in turn, influences his/her view and collection of subsequent information. Discrete decisions made without feedback have been likened to hitting a target from a distance in one try; in contrast, diagnostic problem solving is analogous to a situation where one can monitor and correct the trajectory based on feedback.4Hogarth R.M. Beyond discrete biases: functional and dysfunctional aspects of judgmental heuristics.Psychol Bull. 1981; 90: 197-217Crossref Scopus (405) Google Scholar, 5Kleinmuntz D.N. Cognitive heuristics and feedback in a dynamic decision environment.Manage Sci. 1985; 31: 680-702Crossref Scopus (106) Google Scholar Patient care is a feedback process in which the clinician makes judgments and takes actions with the intended rationale of bringing the patient closer to the desired, presumably healthier, status. This process of observing/diagnosing/treating/observing describes a balancing or goal-seeking feedback loop, in which feedback about the patient's status allows a clinician to calibrate therapy over the very short term. Although physicians may be able to adjust a diagnosis and treatment based on conversation and examination during a specific patient encounter, Berner and Graber1Berner E. Graber M.L. Overconfidence as a cause of diagnostic error in medicine.Am J Med. 2008; 121: S2-S23Abstract Full Text Full Text PDF PubMed Scopus (606) Google Scholar argue that lack of timely or consistent feedback on the accuracy and quality of diagnoses over the long term makes it difficult for them to improve their diagnostic problem-solving skills over time. Once out of medical school and residency, most physicians operate in a "no news is good news" mode, believing that unless they hear about problems, the diagnoses they have made are correct. Berner and Graber invoke a well-established fact of learning theory, namely, that improvement is nearly impossible without accurate and timely feedback. Improving one's diagnostic problem-solving skill, they argue, requires an ability to calibrate the match between the diagnosis made and the patient's actual long-term status. The generic feedback process that would allow a clinician to calibrate and improve a key element of long-term diagnostic skill, the quality of his/her "diagnostic schemas," is depicted in Figure 1. A diagnosis is the result of applying a diagnostic schema to information about the patient as the clinician perceives it. Schema is a term from cognitive science referring to a person's mental model, or internal image of a give