Trial by media: dangers of misinterpretation of medical statistics
2006; Elsevier BV; Volume: 367; Issue: 9517 Linguagem: Inglês
10.1016/s0140-6736(06)68497-3
ISSN1474-547X
AutoresJohn Wright, Chris Bradley, T. Sheldon, Richard Lilford,
Tópico(s)Patient-Provider Communication in Healthcare
ResumoOn April 2, 2006, the BBC may broadcast an episode of its current affairs programme Panorama about the clinical management of women with breast cancer in Bradford, UK. The hospital has been refused any preview of the programme, but on the basis of correspondence from the Panorama team over the past year, we believe that it may make allegations about an individual surgeon's performance on the basis of an analysis of routinely collected medical statistics. The case has major implications about the use and misuse of routinely collected data to publicly accuse doctors of harming patients.In May, 2005, the Panorama team informed Bradford Teaching Hospitals Trust that they planned to broadcast a programme entitled “The surgeon who failed women”. Several allegations were made about a retired Bradford surgeon on the basis of a BBC commissioned report from Michel Coleman, which analysed data from 138 surgeons in the Yorkshire Cancer Registry.The analysis examined the survival of women diagnosed with breast cancer between 1982 and 2003. The Bradford surgeon's performance was compared with that of a selected sample of 20 peers who had similarly operated on large numbers of patients. His patients were shown to have had a lower than average rate of referral for radiotherapy after breast-conserving surgery in the years preceding 1994 compared with the regional mean and this sample of 20. From 1995 onwards there was no significant difference in referral rates when compared with the regional mean, although his rates remained lower than the sample average.The analysis adjusted for age and area of residence, but not for cancer stage or other measures of case mix. There was no excess risk for women treated by this Bradford surgeon with mastectomy. However, in one of the four periods analysed, women who were treated with breast-conserving surgery had a significant excess risk of death confined to the second to fifth years after diagnosis. The implication was that the reported lower survival was due to lower referrals for radiotherapy.The low referral rate for radiotherapy before 1995 was not news to the hospital. Yorkshire hospitals were among the first in the UK to audit the quality of care against explicit, evidence-based criteria of good practice. Bradford was informed in 1995 by the Yorkshire Cancer Registry that referral rates for radiotherapy were lower than the regional mean. Subsequent enquiries revealed that there had been concerns from some local clinicians about the side-effects of radiotherapy on the basis of some individual cases. These cases had prompted a very cautious approach to patient selection for radiotherapy. At this time there was no evidence that radiotherapy improved survival and no regional guidelines.1NHS Centre for Reviews and DisseminationThe management of primary breast cancer. Eff Health Care.http://www.york.ac.uk/inst/crd/ehc26.pdfGoogle ScholarThe hospital used the feedback of its outlying status constructively and reviewed service provision. Rapid changes were introduced with the introduction of a multidisciplinary cancer team to review all women with breast cancer. When the first regional evidence-based guidelines were issued in 1996, which for the first time explicitly identified patients requiring radiotherapy, they were universally accepted and implemented.10 years later, when approached by the BBC, the hospital clearly acknowledged that referral of patients was low before 1995 compared with other hospitals in Yorkshire. The hospital took the allegations of historical excess mortality extremely seriously. Independent reviews of the Coleman report were sought from an independent expert, Trevor Sheldon (see webappendix 1) and an in-depth audit of patient records was commenced by the clinical team.Sheldon's report highlighted flaws in Coleman's analysis and concluded that differences in observed mortality could be explained by chance, case mix, or selection bias in the comparison group and could not be attributed with any confidence to the performance of the individual surgeon.The Panorama team was invited to discuss the evidence. All requests for discussion off camera were refused. However, the team did request interviews on camera with medical staff and these were granted in July, 2005.Subsequent to these interviews, we suggested that the Panorama team seek further review of the Coleman report and we nominated David Spiegelhalter and Richard Lilford as suitable experts on the measurement of clinical performance. In view of the seriousness of the allegations, the hospital approached Lilford to provide further expert review. Lilford argued that it was incorrect to jump from the association between surgeon and outcome over a specific period of time to infer a cause-and-effect association.2Lilford R Mohammed MA Spiegelhalter D Thomson R Use and misuse of process and outcome data in managing performance of acute medical care: avoiding institutional stigma.Lancet. 2004; 363: 1147-1154Summary Full Text Full Text PDF PubMed Scopus (350) Google Scholar Even if this association was real, such an inference was unsafe.In December, 2005, a commentary prepared by Spiegelhalter on the Coleman report was received from the BBC. This commentary provided cautious support for the finding of lower 5-year survival in women treated by the Bradford surgeon, but expressed surprise that adjustment for radiotherapy referral in the analysis of survival did not explain the lower survival associated with surgery. He urged caution in the interpretation and cast doubt about the plausibility of under-referral of radiotherapy being the cause, since recent evidence indicated a survival benefit only at 15 years.3Early Breast Cancer Trialists Collaborative GroupEffects of radiotherapy and of differences in the extent of surgery for early breast cancer on local recurrence and 15 year survival: an overview of randomised trials.Lancet. 2005; 366: 2087-2106Summary Full Text Full Text PDF PubMed Scopus (3949) Google ScholarA final report from Coleman for the Panorama programme was also submitted. We reviewed this report and criticised it both for continuing weaknesses in method and incautious interpretation (see webappendix 2). Panorama have not made their reports available for publication on the internet.In view of the increasing divergence in expert opinion, further requests for discussion with senior members of the Panorama team were made but refused. We also invited the team to hold a televised public debate with an independent chair such as the Editor of The Lancet with the aim of increasing the public understanding of these complex issues. The team again refused. Their only attempt at dialogue was adversarial interrogation of the Trust on camera in January, 2006.The Coleman analysis linked the outcomes of a whole multidisciplinary breast cancer team to one surgeon. However, from 1995, cancer care in the UK has been coordinated by a wide range of specialists including oncologists, radiotherapists, and pathologists. Attribution of radiotherapy referral rates or outcomes after 1995 to an individual surgeon would be inappropriate. Investigating an association between an individual surgeon and his or her operative outcomes is fraught with difficulty. Investigating associations between complex multifaceted cancer management and individual clinical practice can be impenetrable.Chance, case mix, and biased comparison groups could all explain observed lower survival for the Bradford surgeon. However, as we explain below, one factor that could not explain lower survival was the lower referral for radiotherapy.The reported higher mortality rate was assumed to be attributable to the surgeon's lower rate of referral for radiotherapy for some groups of women. However meta-analysis of clinical trials indicates that significant differences in 5-year survival would not be plausible (it required a meta-analysis of 42 000 women in clinical trials to detect a significant survival benefit at 15 years).3Early Breast Cancer Trialists Collaborative GroupEffects of radiotherapy and of differences in the extent of surgery for early breast cancer on local recurrence and 15 year survival: an overview of randomised trials.Lancet. 2005; 366: 2087-2106Summary Full Text Full Text PDF PubMed Scopus (3949) Google Scholar The 15-year survival for the Bradford surgeon was not found to be higher than average when investigated by the Yorkshire Cancer Registry.To investigate the allegations and to determine other potential explanations for the lower survival among the surgeon's patients, a detailed audit of women treated between 1995 and 2000 was carried out by the Trust. This audit showed that regional guidelines were quickly adopted by the team and more importantly that there was no increased risk of local recurrence of breast cancer in patients treated in Bradford over this period.Three of us have spent much of our professional careers highlighting deficiencies in clinical practice and have been uncompromising in our efforts to improve quality of care. However, we are concerned that this episode is yet another example of data being used to point the finger at a clinician's clinical practice without fair process.The past 10 years have seen considerable emphasis placed on the collection and analysis of medical and in particular surgical data such as mortality. We are concerned that this information will increasingly be used inappropriately to make allegations about individual professional practice. In the hands of media that cannot appreciate the subtlety, complexity, and uncertainty surrounding these data, there is a danger that performance figures will be misunderstood or even misused.We argue that analysis of routine health outcomes has a potentially useful role to play in learning about variation and identifying cases for further investigation. However, the key principle is to use such data more as a starting point than an endpoint in investigating quality of care. We contrast this case with another in which careful monitoring led to early and remedial action that protected patients.4Mason S Nicholl J Lilford RJ What to do about poor performance in clinical trials.BMJ. 2002; 324: 419-421Crossref PubMed Google ScholarWe believe the way the Panorama team has handled this case may result in the further and unnecessary erosion of local and national public trust in health services.5O'Neill O A question of trust. Cambridge University Press, Cambridge2002Google Scholar On April 2, 2006, the BBC may broadcast an episode of its current affairs programme Panorama about the clinical management of women with breast cancer in Bradford, UK. The hospital has been refused any preview of the programme, but on the basis of correspondence from the Panorama team over the past year, we believe that it may make allegations about an individual surgeon's performance on the basis of an analysis of routinely collected medical statistics. The case has major implications about the use and misuse of routinely collected data to publicly accuse doctors of harming patients. In May, 2005, the Panorama team informed Bradford Teaching Hospitals Trust that they planned to broadcast a programme entitled “The surgeon who failed women”. Several allegations were made about a retired Bradford surgeon on the basis of a BBC commissioned report from Michel Coleman, which analysed data from 138 surgeons in the Yorkshire Cancer Registry. The analysis examined the survival of women diagnosed with breast cancer between 1982 and 2003. The Bradford surgeon's performance was compared with that of a selected sample of 20 peers who had similarly operated on large numbers of patients. His patients were shown to have had a lower than average rate of referral for radiotherapy after breast-conserving surgery in the years preceding 1994 compared with the regional mean and this sample of 20. From 1995 onwards there was no significant difference in referral rates when compared with the regional mean, although his rates remained lower than the sample average. The analysis adjusted for age and area of residence, but not for cancer stage or other measures of case mix. There was no excess risk for women treated by this Bradford surgeon with mastectomy. However, in one of the four periods analysed, women who were treated with breast-conserving surgery had a significant excess risk of death confined to the second to fifth years after diagnosis. The implication was that the reported lower survival was due to lower referrals for radiotherapy. The low referral rate for radiotherapy before 1995 was not news to the hospital. Yorkshire hospitals were among the first in the UK to audit the quality of care against explicit, evidence-based criteria of good practice. Bradford was informed in 1995 by the Yorkshire Cancer Registry that referral rates for radiotherapy were lower than the regional mean. Subsequent enquiries revealed that there had been concerns from some local clinicians about the side-effects of radiotherapy on the basis of some individual cases. These cases had prompted a very cautious approach to patient selection for radiotherapy. At this time there was no evidence that radiotherapy improved survival and no regional guidelines.1NHS Centre for Reviews and DisseminationThe management of primary breast cancer. Eff Health Care.http://www.york.ac.uk/inst/crd/ehc26.pdfGoogle Scholar The hospital used the feedback of its outlying status constructively and reviewed service provision. Rapid changes were introduced with the introduction of a multidisciplinary cancer team to review all women with breast cancer. When the first regional evidence-based guidelines were issued in 1996, which for the first time explicitly identified patients requiring radiotherapy, they were universally accepted and implemented. 10 years later, when approached by the BBC, the hospital clearly acknowledged that referral of patients was low before 1995 compared with other hospitals in Yorkshire. The hospital took the allegations of historical excess mortality extremely seriously. Independent reviews of the Coleman report were sought from an independent expert, Trevor Sheldon (see webappendix 1) and an in-depth audit of patient records was commenced by the clinical team. Sheldon's report highlighted flaws in Coleman's analysis and concluded that differences in observed mortality could be explained by chance, case mix, or selection bias in the comparison group and could not be attributed with any confidence to the performance of the individual surgeon. The Panorama team was invited to discuss the evidence. All requests for discussion off camera were refused. However, the team did request interviews on camera with medical staff and these were granted in July, 2005. Subsequent to these interviews, we suggested that the Panorama team seek further review of the Coleman report and we nominated David Spiegelhalter and Richard Lilford as suitable experts on the measurement of clinical performance. In view of the seriousness of the allegations, the hospital approached Lilford to provide further expert review. Lilford argued that it was incorrect to jump from the association between surgeon and outcome over a specific period of time to infer a cause-and-effect association.2Lilford R Mohammed MA Spiegelhalter D Thomson R Use and misuse of process and outcome data in managing performance of acute medical care: avoiding institutional stigma.Lancet. 2004; 363: 1147-1154Summary Full Text Full Text PDF PubMed Scopus (350) Google Scholar Even if this association was real, such an inference was unsafe. In December, 2005, a commentary prepared by Spiegelhalter on the Coleman report was received from the BBC. This commentary provided cautious support for the finding of lower 5-year survival in women treated by the Bradford surgeon, but expressed surprise that adjustment for radiotherapy referral in the analysis of survival did not explain the lower survival associated with surgery. He urged caution in the interpretation and cast doubt about the plausibility of under-referral of radiotherapy being the cause, since recent evidence indicated a survival benefit only at 15 years.3Early Breast Cancer Trialists Collaborative GroupEffects of radiotherapy and of differences in the extent of surgery for early breast cancer on local recurrence and 15 year survival: an overview of randomised trials.Lancet. 2005; 366: 2087-2106Summary Full Text Full Text PDF PubMed Scopus (3949) Google Scholar A final report from Coleman for the Panorama programme was also submitted. We reviewed this report and criticised it both for continuing weaknesses in method and incautious interpretation (see webappendix 2). Panorama have not made their reports available for publication on the internet. In view of the increasing divergence in expert opinion, further requests for discussion with senior members of the Panorama team were made but refused. We also invited the team to hold a televised public debate with an independent chair such as the Editor of The Lancet with the aim of increasing the public understanding of these complex issues. The team again refused. Their only attempt at dialogue was adversarial interrogation of the Trust on camera in January, 2006. The Coleman analysis linked the outcomes of a whole multidisciplinary breast cancer team to one surgeon. However, from 1995, cancer care in the UK has been coordinated by a wide range of specialists including oncologists, radiotherapists, and pathologists. Attribution of radiotherapy referral rates or outcomes after 1995 to an individual surgeon would be inappropriate. Investigating an association between an individual surgeon and his or her operative outcomes is fraught with difficulty. Investigating associations between complex multifaceted cancer management and individual clinical practice can be impenetrable. Chance, case mix, and biased comparison groups could all explain observed lower survival for the Bradford surgeon. However, as we explain below, one factor that could not explain lower survival was the lower referral for radiotherapy. The reported higher mortality rate was assumed to be attributable to the surgeon's lower rate of referral for radiotherapy for some groups of women. However meta-analysis of clinical trials indicates that significant differences in 5-year survival would not be plausible (it required a meta-analysis of 42 000 women in clinical trials to detect a significant survival benefit at 15 years).3Early Breast Cancer Trialists Collaborative GroupEffects of radiotherapy and of differences in the extent of surgery for early breast cancer on local recurrence and 15 year survival: an overview of randomised trials.Lancet. 2005; 366: 2087-2106Summary Full Text Full Text PDF PubMed Scopus (3949) Google Scholar The 15-year survival for the Bradford surgeon was not found to be higher than average when investigated by the Yorkshire Cancer Registry. To investigate the allegations and to determine other potential explanations for the lower survival among the surgeon's patients, a detailed audit of women treated between 1995 and 2000 was carried out by the Trust. This audit showed that regional guidelines were quickly adopted by the team and more importantly that there was no increased risk of local recurrence of breast cancer in patients treated in Bradford over this period. Three of us have spent much of our professional careers highlighting deficiencies in clinical practice and have been uncompromising in our efforts to improve quality of care. However, we are concerned that this episode is yet another example of data being used to point the finger at a clinician's clinical practice without fair process. The past 10 years have seen considerable emphasis placed on the collection and analysis of medical and in particular surgical data such as mortality. We are concerned that this information will increasingly be used inappropriately to make allegations about individual professional practice. In the hands of media that cannot appreciate the subtlety, complexity, and uncertainty surrounding these data, there is a danger that performance figures will be misunderstood or even misused. We argue that analysis of routine health outcomes has a potentially useful role to play in learning about variation and identifying cases for further investigation. However, the key principle is to use such data more as a starting point than an endpoint in investigating quality of care. We contrast this case with another in which careful monitoring led to early and remedial action that protected patients.4Mason S Nicholl J Lilford RJ What to do about poor performance in clinical trials.BMJ. 2002; 324: 419-421Crossref PubMed Google Scholar We believe the way the Panorama team has handled this case may result in the further and unnecessary erosion of local and national public trust in health services.5O'Neill O A question of trust. Cambridge University Press, Cambridge2002Google Scholar Web Extra Material Download .pdf (.43 MB) Help with pdf files Download .pdf (.14 MB) Help with pdf files Download .pdf (.43 MB) Help with pdf files Download .pdf (.14 MB) Help with pdf files
Referência(s)