How the adverse effect counting window affected vaccine safety calculations in randomised trials of COVID‐19 vaccines
2024; Wiley; Volume: 30; Issue: 3 Linguagem: Inglês
10.1111/jep.13962
ISSN1365-2753
Autores Tópico(s)COVID-19 Clinical Research Studies
ResumoSeveral articles published recently in the Journal of Evaluation in Clinical Practice raise important questions about how the efficacy and safety of COVID-19 vaccines have been and are being measured. First, Fung, Jones, and Doshi explained that statistical biases, particularly the case counting window bias, can greatly exaggerate the perceived effectiveness of COVID-19 vaccines in observational studies.1 A follow-up article by myself explained that this effect may be understated, due to the often accompanied definitional bias, and also that these biases could likewise exaggerate the perceived safety of COVID-19 vaccines in observational studies.2 Doshi and Fung then added further to the discussion by finding issues with case counting windows in COVID-19 vaccine clinical trials, relating to vaccine effectiveness.3 In this final article, I shall provide an example justifying my prior concern about counting windows in observational studies concerning safety, and further explain that there are also concerns around adverse effect counting windows in COVID-19 vaccine clinical trials, which may have exaggerated claims about vaccine safety. In my previous article, I expressed concern that issues around counting windows could lead to overstated vaccine safety in observational studies. Since then, an example of just that happening has appeared. Kitano et al., an observational study appearing in the American Journal of Epidemiology claims that, 'The benefits of mRNA COVID-19 vaccines in protecting against the omicron variant outweigh the risks, irrespective of age, sex, and comorbidity'.4 Interestingly, their calculated net benefits, expressed in terms of quality-adjusted life years (table 1), were minimal, leaving open the possibility that issues with counting windows could mean that there are no net benefits, and possibly even net deficits. To illustrate, the highest gain was stated as 939.8 QALY per 100,000 people. This equates to less than 3.5 days per person. The smallest gain was stated as 18.7 QALY per 100,000 people. This equates to less than 2 hours per person. Those results are further subject to the sorts of concerns raised by Fung, Jones and Doshi, and myself. This is how the authors categorised their subjects: Less than 5 months (Days 14–149) after the primary two doses versus no doses. Less than 4 months (Days 7–119) after a third dose versus 5–8 months (days 150–262) after the primary doses (no third dose). Less than 4 months (Days 7–119) after a fourth dose versus 4–7 months (days 120–232) after a third dose (no fourth dose) for adults. With no explanation provided, and none possible in the case of safety analyses, COVID-19 cases and adverse effects occurring from dose 1–14 days after dose 2 appear to be overlooked, in those receiving only the primary two doses. That this is not ideal is evidenced by cases where the vaccines have apparently been associated with deaths very soon after vaccination, such as those caused by anaphylaxis.5 Kitano et al. have themselves made reference in their article to a Japanese study, Suzuki et al., which concerns deaths 'within 7 days after COVID-19 vaccination', including myocarditis deaths, with the authors declaring that several of these deaths did 'show a causal relationship to vaccination'.6 Kitano et al. are omitting relevant data from their analysis, including data they do have access to. Furthermore, counting windows, concerning both cases and adverse effects, so affecting efficacy and safety estimates, are limited to 4–5 months after the last dose. Given ongoing concerns about waning effectiveness and even negative effectiveness on the one hand, and very short-term and also long-term adverse effects on the other, the potential exists that these already underwhelming results are exaggerated. It thus appears possible that with these concerns addressed, the authors could find that the COVID-19 vaccines are causing a net harm, now in the omicron era and beyond. In their latest article, Doshi and Fung have again added greatly to the scientific discussion about COVID-19 vaccines by identifying case counting window issues in the mRNA COVID-19 vaccine clinical trials. Their article once again focused on efficacy. And I again believe that they may have understated things once more, as numerous issues with the clinical trials and FDA briefing documents had gone unmentioned. For example, there are a significant number of trial participants lost to follow-up, and Pfizer also acknowledged '3410 total cases of suspected but unconfirmed COVID-19 in the overall study population' in the FDA briefing document on their vaccine trial, split almost evenly between the treatment and placebo groups, which would have drastically brought down treatment efficacy estimates.7 Nevertheless, my main focus is to here explain that the same trials include suspect adverse effect counting windows, which could lead to reported estimates of safety being inaccurate. Examining the same Pfizer and Moderna clinical trials as Doshi and Fung, I can thankfully report that it appears that one of the most common counting window issues does not apply here, in terms of adverse effects. While there are concerns about when case counting windows begin, in these clinical trials the adverse effect counting windows rightfully start from dose 1 (not 7–14 days after dose 2). However, there are still concerning issues with the adverse effect counting windows in the clinical trials. Namely, their limited lengths. For the Pfizer trial, adverse effect counting windows are explained thusly: 'The primary end points of this trial were solicited, specific local or systemic adverse events and use of antipyretic or pain medication within 7 days after the receipt of each dose of vaccine or placebo, as prompted by and recorded in an electronic diary in a subset of participants (the reactogenicity subset), and unsolicited adverse events (those reported by the participants without prompts from the electronic diary) through 1 month after the second dose and unsolicited serious adverse events through 6 months after the second dose'.8 Not only is it concerning that the researchers are relying on unsolicited contact (to say nothing of the aforementioned large number of trial participants lost to follow-up; and of course deceased trial participants will not be contacting the researchers to describe their issues), but even then the analyses extend to just 6 months after the second dose. They do assure us that their safety monitoring 'will continue for 2 years after administration of the second dose of vaccine', though the selected windows would not capture harms over the longer term if they exist, which might have implications on acceptability. It is also notable that 2-year follow-up is now not feasible as the trial was quickly unblinded with the treatment and placebo groups effectively merged. Adding to the concern, there were several cardiovascular deaths in the treatment group, and it is now reported in several (soon discussed) studies that the Pfizer vaccine is associated with cardiovascular deaths. Yet the researchers stated: 'No deaths were considered by the investigators to be related to the vaccine or placebo'. This further betrays an overreliance on what Pfizer and BioNTech consider to be adverse effects caused by their own vaccine. Similarly, in the Moderna trial, adverse effect counting windows are explained thusly: 'Safety assessments included monitoring of solicited local and systemic adverse events for 7 days after each injection; unsolicited adverse reactions for 28 days after each injection; adverse events leading to discontinuation from a dose, from participation in the trial, or both; and medically attended adverse events and serious adverse events from Day 1 through Day 759'.9 These counting windows are again very short, betray a heavy reliance on unsolicited reporting, and even the latter endpoint of approximately 2 years is now unfeasible due again to unblinding. These short adverse effect counting windows, and the issue of rapid unblinding and group mixing, were also noted in Fraiman et al., published in Vaccine, which additionally found that even with the data as presented, 'Pfizer and Moderna mRNA COVID-19 vaccines were associated with an excess risk of serious adverse events of special interest of 10.1 and 15.1 per 10,000 vaccinated over placebo baselines of 17.6 and 42.2 (95% confidence interval [CI]: −0.4 to 20.6 and −3.6 to 33.8), respectively. Combined, the mRNA vaccines were associated with an excess risk of serious adverse events of special interest of 12.5 per 10,000 vaccinated (95% CI: 2.1–22.9); risk ratio 1.43 (95% CI: 1.07–1.92). The Pfizer trial exhibited a 36% higher risk of serious adverse events in the vaccine group; risk difference 18.0 per 10,000 vaccinated (95% CI: 1.2–34.9); risk ratio 1.36 (95% CI: 1.02–1.83). The Moderna trial exhibited a 6% higher risk of serious adverse events in the vaccine group: risk difference 7.1 per 10,000 (95% CI: –23.2 to 37.4); risk ratio 1.06 (95% CI: 0.84 to 1.33). Combined, there was a 16% higher risk of serious adverse events in mRNA vaccine recipients: risk difference 13.2 (95% CI: −3.2 to 29.6); risk ratio 1.16 (95% CI: 0.97–1.39)'.10 Similar results and concerns are found in Benn et al., with the authors finding that based 'on the RCTs with the longest possible follow-up, mRNA vaccines had no effect on overall mortality despite protecting against fatal COVID-19', with their data indicating a net mortality deficit (though not statistically significant), and being disappointed that more accurate analyses are 'hampered by the short follow-up in these trials as the individuals from the control groups were offered vaccination after 2–6 months', finally recommending that while 'mass-vaccination programs with COVID-19 vaccines are rolled out, data on their effects on non-COVID-19 mortality should be collected'.11 Furthermore, the issue of additional, longer term, adverse effects eventually showing up significantly after the vaccines were at least provisionally approved led to a report in the BMJ, wherein the FDA had been 'criticised for taking more than a year to follow up a potential increase in serious adverse events in elderly people receiving Pfizer's COVID-19 vaccine'.12 It may be that short counting windows, as well as rapid unblinding, are becoming standard practice, not being limited to research on the RNA or DNA vaccines. In a clinical trial for a more traditional vaccine for COVID-19, QazCovid-in, reported on in a Lancet journal by Khairullin et al., the authors state: 'Here we report the results of the multicentre, randomised, single-blind, placebo-controlled, phase 3 clinical trial with a 180-day follow-up period starting from the 14th day after the first immunisation with QazCovid-in® to evaluate vaccine safety, immunogenicity, and efficacy, as well as the durability of the immune response to immunisation', later clarifying, 'safety outcomes included the incidence of all local reactions observed in participants within 2 h after the first and second doses of the vaccine, as well as the incidence of solicited adverse reactions within 7 days after the first or second vaccine administration, and the incidence of unsolicited adverse events AEs from the first vaccination up to Day 180 of the study'.13 They also report that unblinding 'was scheduled on Day 90'. Khairullin et al. further note 'Within 14 days after the first vaccination, 4 (0.17%) vaccine recipients and 0 (0%) placebo recipients were diagnosed with mild COVID-19', which could have greatly altered their effectiveness estimate, and they acknowledge that the single 'severe COVID-19 case was identified in a vaccine recipient', while their data (table 4) reveals—in line with Fraiman et al. and Benn et al.—that there were considerably more 'Hospitalisation, deaths and other secondary outcomes' in the vaccinated group, adding to our previous concerns about the potential for inaccurate estimates and the possibility of negative effectiveness. Returning to the mRNA COVID-19 vaccines, it is worth nothing that while they were developed, trialled, and approved within the same year that the pandemic was declared, and while authorisation was granted with only days and months of safety data, the FDA makes clear on its website that safety monitoring for new pharmaceutical products typically takes several years.14 Similarly, the CDC notes that vaccine development 'often takes 10–15 years of laboratory research', revealing just how atypical the present situation is.15 While development, trials, and authorisation might reasonably be expedited in an emergency, it is worth inquiring into whether there truly was an emergency for all age groups, and whether now, with the pandemic generally declared over, there is still an emergency, and one that somehow prevents ongoing safety monitoring and revised safety analyses, based on counting windows designed to collect more relevant data. To illustrate, one of the more significant adverse effects now being discussed in the literature, which can take more than 2 months to appear, is, especially if it is preclinical, vaccine-related myocarditis (VRM). While the issue is still not fully understood, particularly for the long-term, some of the research in this area presents cause for concern. In this section, further developing on a rapid response appearing in BMJ Open,16 I shall explain why the risks of myocarditis associated with COVID-19 vaccination could indicate that the adverse effect counting windows in several observational studies and clinical trials are inadequate. There has been increasing research done on myocarditis associated with the mRNA COVID-19 vaccines, being particularly concerning for young males. Kitano et al. note: 'In the United States, among persons vaccinated with BNT162b2 there were 26.7 reported cases of myocarditis per 100,000 males aged 12–17 years within 21 days after the second dose, and many studies have indicated that the risk is higher for mRNA-1273 than for BNT162b2'.17 The Australian TGA reports 'Rates of likely myocarditis cases following Comirnaty (Pfizer)' as high as 13.2 per 100,000 doses (second doses), for males aged 12–17. 'Rates of likely myocarditis cases following Spikevax (Moderna)' are as high as 23.6 per 100,000 doses (second doses), for males aged 12–17.18 Reporting on an Israeli study, the American Heart Association stated: 'The risk of developing myocarditis among males ages 16–19 after a third dose was about 1 in 15,000'.19 A recent study published in BMJ Open, by Alami et al., found that their 'meta-analysis indicates that within 30-day follow-up period, vaccinated individuals were twice as likely to develop myo/pericarditis in the absence of SARS-CoV-2 infection compared to unvaccinated individuals, with a rate ratio of 2.05 (95% CI: 1.49–2.82)'.20 Also of note is a South Korean study, Cho et al., published very recently in the European Heart Journal, which helps shed some light on postvaccination myocarditis risks. The authors found, out of 44,276,704 vaccinated people, that 'COVID-19 VRM was confirmed in 480 cases (1.08 cases per 100,000 persons)'. Many of these were serious, as the authors explain: 'Severe VRM was identified in 95 cases (19.8% of total VRM, 0.22 per 100,000 vaccinated persons), 85 intensive care unit admission (17.7%), 36 fulminant myocarditis (7.5%), 21 extracorporeal membrane oxygenation therapy (4.4%), 21 deaths (4.4%), and one heart transplantation (0.2%). Eight out of 21 deaths were sudden cardiac death (SCD) attributable to VRM proved by an autopsy, and all cases of SCD attributable to VRM were aged under 45 years and received mRNA vaccines'. Inadequate counting windows may also apply with this study, with the authors stating that 'Acute myocarditis developed within 42 days after COVID-19 vaccination was considered as COVID-19 VRM'. Furthermore, VRM 'incidence was highest in males between the ages of 12 and 17 years (5.29 cases per 100,000 persons)', which is roughly 1 case per 19,000 persons.21 These figures may even be larger, with the study's limited timeframes; possible undercounting of such deaths, as for eight of the deaths, 'VRM was not suspected as a clinical diagnosis or a cause of death before performing an autopsy'; and the risks of subclinical VRM being currently unknown. That myocarditis cases may occur after the short counting windows in the clinical trials is no small matter. In the United Kingdom, a government report from early 2023 discussed the number needed to vaccinate (NNV) to prevent severe COVID-19 hospitalisation. Table 4 in the report reveals this number to be over 100,000 for very young age groups (primary series, around 200,000 for boosters). The number for those specifically considered 'no risk', and aged 30–39, is over 300,000, with no estimate provided for younger age groups (primary series), presumably because such severe COVID-19 is very rare in the young and healthy. The figures approached 1,000,000 for boosters, for various 'no risk' groups.22 Somewhat similar results can be gleaned from the analysis in Kitano et al. Given the aforementioned figures for QALY gains in young and healthy males, combined with vaccine effectiveness that is below 100% (for various reasons we can expect that the vaccines now offer far fewer benefits than in 2020—a topic for another day), we can again expect number needed to vaccinate figures in the hundreds of thousands, or even millions. Recall that their net benefits for vaccination in young and healthy males amounted to only a few hours per person. While the risk of vaccine-induced myocarditis may indeed be very small, it appears that the risk of serious COVID in the young and healthy is smaller still. Estimating the NNV for preventing deaths is unfortunately necessary, on account of the limited data, partly due to short counting windows. Given the difference between 'severe COVID-19 hospitalisation' and death, and given that vaccine effectiveness wanes rapidly (data from Kitano et al. notes effectiveness as low as 14%) and may even become negative (as with our prior discussion on negative effectiveness, further discussed in a BMJ rapid response),23 a conservative estimate of the number needed to vaccinate to prevent a COVID-19 death in healthy young males might be 1 million (which aligns well with some of Pfizer's own estimates, as shall be explained soon). Very roughly, then, to prevent the death of one young healthy male from COVID-19, we might need to vaccinate 1 million healthy young males. Assuming a conservative rate of VRM of 1 in 20,000, we could be causing 50 cases of VRM. If 4.4% of them die (the death rate, based on quite limited timeframes, from VRM found in Cho et al.), approximately 2 healthy young males would be killed for every one saved from COVID-19. This result only concerns myocarditis. Nothing here is said of other adverse effects, such as pericarditis, blood clotting,24 neurological symptoms,25 or kidney damage,26 with many of these adverse events occurring after the employed counting windows. For but one example of how myocarditis can be a longer term issue, a recent paper by Kim et al. explains that while a large proportion of myocarditis associated deaths occur within 1 month of diagnosis, many myocarditis associated deaths occur several years later.27 More research is needed to confirm if the same is true with VRM. The currently employed counting windows would become more questionable if more data and wider timeframes reveal the myocarditis (and other adverse effect) incidence rates as being higher than is currently understood. Critics could note that COVID-19 also carries with it a risk of myocarditis, but that is not relevant when the vaccines are nonsterilising, and might display negative effectiveness over time, so that the vaccines could actually add to the already existing baseline and COVID-19 risks of myocarditis. It is also not clear how a vaccine that is associated with increased myocarditis can help prevent myocarditis caused by COVID-19. More data are needed. This estimation is plagued with uncertainties due to a lack of data. We simply do not yet understand fully the effectiveness of the vaccines over time or the potential long-term side effects, partly due to the short case and adverse effect counting windows in the observational studies and clinical trials. What is needed are long-term controlled clinical trials or at least long-term observational studies that account for the biases and counting window issues mentioned throughout this series of articles. Finally, more justification for these concerns about adverse effect counting windows comes from Pfizer. In 2021, via another FDA briefing document, Pfizer effectively acknowledges in tab. 14 that per 1 million 5–11-year-old males as little as 0 lives will be saved by the vaccine, weighed up against 179 cases of excess myocarditis. These 179 cases Pfizer expects to result in 98 hospitalisations and 58 ICU admissions. Despite myocarditis being potentially fatal, and the aforementioned research that this also appears to be the case with COVID-19 vaccine-related myocarditis, Pfizer estimates that the number of deaths from these 179 cases would be 0.28 Pfizer also admits to short counting windows in their associated clinical trial on 5–11-year-old children, explaining that limitations 'include the lack of longer-term follow-up to assess the duration of immune responses, efficacy, and safety'. They further acknowledge that their 'study was also not powered to detect potential rare side effects of BNT162b2 in 5–11-year-olds'.29 Additionally, reporting on the mRNA COVID-19 vaccines, NBC News stated: 'The Food and Drug Administration has required that the drugmakers conduct several studies assessing the potential long-term impacts of myocarditis, as part of its approval of the mRNA COVID vaccines in the United States. Early findings from the research could be published as early as next year, sources told NBC News. Some of the trials will follow those who developed the condition for as long as 5 years, according to the FDA's approval letters. The trials will be monitoring for myocarditis and subclinical myocarditis, which doesn't cause symptoms'.30 However, Pfizer's ongoing research again appears to be beset with counting window issues. One of their inclusion criteria is: 'Received either the 1st, 2nd, 3rd or booster dose(s) of COMIRNATY ≤ 7 days of symptom onset, even if a different brand of COVID-19 vaccine had been administered in earlier vaccinations Retrospectively ascertained participants must be enroled within 2 years of diagnosis'. Another is: 'Probable or confirmed myocarditis/pericarditis as per the contemporaneous CDC case definition at the time of diagnosis'. This is again not an ideal counting window, and is not at all set up to detect subclinical myocarditis. No results have been posted as of yet. Nevertheless, Pfizer is confident that 'This will help us determine if COMIRNATY is safe and effective, and if there is a myocarditis/pericarditis association that should be noted'.31 As indicated by Cho et al., these determinations and notes may arrive too late for some. Doshi and Fung once again have greatly added to the discussion about how mRNA COVID-19 vaccine effectiveness is measured, by highlighting the potential issues with case-counting windows in the clinical trials. I further stressed the importance of these issues, again noting similar issues with adverse effect counting windows in the clinical trials, partly justifying this concern with a discussion on how recent research on the risks of vaccine-related myocarditis, which in some cases would only be apparent after the counting windows, has the potential to single-handedly nullify the claim that the benefits of the vaccines still outweigh the risks in all populations. Open access publishing facilitated by The University of Sydney, as part of the Wiley - The University of Sydney agreement via the Council of Australian University Librarians. The author declares no conflict of interest. Data sharing is not applicable to this article as no datasets were generated or analysed during the current study.
Referência(s)