Covid Vaccine Effectiveness at 6 months

Straightforward analysis of hospitalizations

The most straightforward analysis to do is to look at everyone who was hospitalized with COVID-19, match them up with vaccination records, and calculate the risk of hospitalization for the vaccinated and unvaccinated populations. In practice, age has such a strong effect on COVID outcomes that you at least need to control for age among the different population.

This study from New York state (https://www.cdc.gov/mmwr/volumes/70/wr/mm7034e1.htm?s_cid=mm7034e1_w#F1_down) looked at 9,000 hospitalizations due to COVID between May and July 2021. Looking at the Pfizer and Moderna vaccines together, they found a VE of about 95% for preventing hospitalizations for adults 18-49, 50-64 and >65, with no change over time as the Delta variant took over.

The data from Israel (https://www.covid-datascience.com/post/israeli-data-how-can-efficacy-vs-severe-disease-be-strong-when-60-of-hospitalized-are-vaccinated) of 500 hospitalized COVID patients that’s been kicking around the internet suggests a VE of 90%+ for preventing severe disease among adults <50 and 85% effective among adults over 50.

Another CDC study (https://www.cdc.gov/mmwr/volumes/70/wr/mm7037e1.htm?s_cid=mm7037e1_w) does a very straightforward analysis (looking at the number of hospitalized people who are vaccinated/unvaccinated), and concludes a VE of ~90% against hospitalization.

In-depth analysis of hospitalizations

A study from Kaiser Permanente (https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(21)02183-8/fulltext) is a bit of a weird fish, as it doesn’t have a comparison group, but has some rather complex modeling to account for differences between the vaccinated and unvaccinated. They find no decline in vaccine effectiveness for the Pfizer vaccine at 6 months post vaccination (between 80 and 90%). They looked at 12,000 COVID hospitalizations, and controlled for a huge host of potential confounding variables, but didn’t have an explicit comparison group. Pfizer was involved in the study, and there’s some rather complex modeling, so I might take the data with a grain of salt.

Studies with a comparison group

To look at vaccine effectiveness in more detail, it’s helpful to have a group to compare against. For most of these studies, this was people admitted to a hospital for something other than COVID-19, or people admitted with COVID-like symptoms, but with a negative COVID test.

This study (https://www.cdc.gov/mmwr/volumes/70/wr/mm7038e1.htm?s_cid=mm7038e1_w) seems pretty reasonable - it’s a case control study, covered 3,600 patients, and covers adults without immunocompromising conditions. They say that >120 days after full vaccination, Moderna has a VE of 92%, Pfizer has a VE of 77% against hospitalizations.

This study (https://www.cdc.gov/mmwr/volumes/70/wr/mm7034e2.htm?s_cid=mm7034e2_w) of 18 hospitals reports a VE of 90% against hospitalization for patients without immunocompromising conditions, with no trend over time.

Yet another CDC study (https://www.cdc.gov/mmwr/volumes/70/wr/pdfs/mm7037e2-H.pdf) looking at hospitalizations for July and August 2021 reports a VE against hospitalizations of 80% for Pfizer, and 95% for Moderna. The study did not explicitly look at how VE changed over time. The study used a test-negative design, meaning they are comparing people who tested positive for COVID-19 against people who tested negative for the disease.

A large study from Qatar reported a VE of 80-90% for the Pfizer vaccine against hospitalizations, 4-6 months after vaccination, with no clear trends over time. There’s maybe some hint of decreasing protection, but the error bars also get wider as time goes on. (https://www.nejm.org/doi/full/10.1056/NEJMoa2114114).

This preprint looking at Israel data (https://www.medrxiv.org/content/10.1101/2021.08.24.21262423v1.full.pdf) gives a VE against hospitalization of 85-95%, 6+ months after vaccination. This study was a little different, in that their primary comparison was between people who received the vaccine in January versus people who received the vaccine in latter months. There’s some hint of declining effectiveness over time (91% to 86% for adults 60+, 98% to 94% for adults 40-59), but it does not appear statistically significant.

I’ve seen some additional references to data out of Israel suggesting decline protection against severe disease (e.g., https://www.nytimes.com/2021/09/17/science/cdc-pfizer-vaccine-efficacy.html), but the best numbers I’ve been able to find are in the above preprint, and they don’t seem as dire as the news article indicates. There’s this set of slides from Isreal (https://www.gov.il/BlobFolder/reports/vpb-12082021/he/files_publications_corona_vpb-12082021-01.pdf), that seem to show a drop in VE against severe COVID for those 65+ over time, but I think I trust the pre-print numbers more.

Summary of hospitalization data

My summary from these numbers that that, 6 months out against Delta, Pfizer provides 80 - 90% protection against hospitalization due to COVID-19. I’m putting more weight on the studies with a comparison group than those just looking at crude hospitalization rates. Of those studies, 4 of them reported a VE of between 80 and 90%, while one study reported a VE of 77%. Most of the studies reported no significant trend over time, making it unclear why these numbers are lower than what was found in the initial clinical trials.

Summary of infection

My takeaway from all this is that there does seem to be a real decline in VE against symptomatic infection. Unfortunately, there’s a lot of variation in the absolute VE numbers between the different studies. I’m seeing numbers between 22 (in Qatar) and 75% (in the UK), which is a pretty dramatic difference. Throwing all the numbers together into a weighted linear regression, I find that at 6 months post vac, the average vaccine effectiveness is 54 or 42%, depending if I include Qatar or not. There’s a lot more someone could do to better pool these results, but I’m not going to, I’m just going to say that at this point, vaccines are probably about 50% effective at preventing infection, but honestly, it could easily be as high as 75%, or possibly as low as 25%.

Figure 1: Vaccine effectiveness against infection over time, based on data from Kaiser, Qatar, UK (twice), and Israel

Background information.

As far as I can tell, there are 3 ways of evaluating how infectious someone with COVID-19 is. Going from least to most reliable, they are: Ct values, percent culture positive, and contact tracing. Unfortunately, the more reliable methods are more difficult to do, so fewer studies use them.

In a little more detail:

1. Cycle Threshold (Ct) values refer to the number of PCR cycles that are needed before the virus can be identified. Lower numbers mean more virus was present in the initial sample. Generally there is some maximum number of cycles that a PCR test will do, and after that number of cycles, if SARS-CoV-2 is not identified, the test comes back negative; if the virus is detected before that number, the test comes back positive. In general, a lower Ct values means a higher viral load, and in theory should mean a greater likelihood of transmission. However, Ct values can really only be compared between samples collected in the same way and run using the same PCR procedure, since things like how much virus was collected in the initial sample can affect the results. Ct values only tell you about viral genetic material, which can be present even when there isn’t any infectious virus. (https://www.tn.gov/content/dam/tn/health/documents/cedep/novel-coronavirus/Ct_Fact_Sheet.pdf)

2. Percent culture positive means taking a sample from an infected individual and trying to use it to grow additional copies of the virus in a lab. That needs to have pretty strict biohazard controls around it, so relatively few places can perform this test. The upside though is that this test only detects viruses that can produce other viruses, so it should be more reliable than looking at Ct values. (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7427302/)

3. Contact tracing is exactly what it sounds like - looking at close contacts of infected individuals and seeing how many of them came down with COVID. Effective contact tracing requires a lot of resources, and close contacts could be infected from other sources if the virus is circulating widely. However, in theory it’s the only approach that directly answers the question we’re really interested in, of how likely an infected person is to infect other people.

Studies on transmission

I’ve found two studies that used contact tracing to look at vaccine effectiveness against transmission.

One study from the Netherlands uses contract tracing to examine the spread of COVID from an index case to other members of their household. They have about 5K index cases. They report a Vaccine Effectiveness against transmission of 40-60%, lumping all the vaccines together. 55% of index cases and 60% of contacts received the Pfizer vaccine. They also say that “However, our data does not allow detailed analysis of VET waning due to small numbers and strong correlation with age and time since vaccination of the household contacts”. Overall, they find about a 10% “secondary attack rate” for a vaccinated index case to infect vaccinated contacts. (https://www.medrxiv.org/content/10.1101/2021.10.14.21264959v1.full.pdf)

Then there’s an absolute beast of a study from the UK (https://www.medrxiv.org/content/10.1101/2021.09.28.21264260v2.full.pdf) that reports a whole ton of information of transmission of COVID. It’s a retrospective cohort study on the contacts of COVID index cases. They looked at both asymptomatic and symptomatic index cases. Now, this is a pretty loose cohort, meaning that contacts were only included if they accessed testing, and it’s a little unclear to me how they determine who exactly was a contact. To me, excluding un-tested contacts suggests that the secondary attack rate will be strongly biased upwards. Only 26% of the contacts underwent testing.

The main question they had in the study was how does index case vaccination status affect onward transmission. Key results from the study for Delta were that 2 doses of Pfizer reduced the risk on onward transmission by 50% 2 weeks after the second dose, but this declines to only 25% effectiveness 14 weeks post second dose:

Figure 2: Vaccine effectiveness against transmission over time, from the UK study (BNT162b = Pfzier vaccine)

A second key number is that contacts of asymptomatic index cases were less likely to test positive, but with less of an effect than I would expect (adjusted risk ratio = 0.73, so ~25% less likely to transmit).

Beyond those two studies, the evidence becomes a lot more confusing.

In general, a number of studies have shown that Ct values (sometimes reported as viral load) are similar between vaccinated and unvaccinated individuals. This includes the giant UK study, a study of 200 patients from Singapore (https://www.medrxiv.org/content/10.1101/2021.07.28.21261295v1.full.pdf), and a study of 700 people from Wisconsin (https://www.medrxiv.org/content/10.1101/2021.07.31.21261387v5.full.pdf), and the study of the Provincetown outbreak (https://www.cdc.gov/mmwr/volumes/70/wr/mm7031e2.htm?s_cid=mm7031e2_w).

As far as I know, there’s one study suggesting that vaccinated people tend to have higher Ct values (meaning lower viral load) than unvaccinated people. A study from the UK that used random testing of individuals (https://spiral.imperial.ac.uk/bitstream/10044/1/90800/2/react1_r13_final_preprint_final.pdf) found that “median Ct value was higher for vaccinated participants at 27.6 (25.5, 29.7) compared with unvaccinated at 23.1 (20.3, 25.8)”. However, there’s some ambiguity in how you define “positives” - if you restrict the data to stronger positives, the difference between vaccinated and unvaccinated goes down. I’m not quite sure how to interpret all that - after all, if you looking over a narrower band, I’d expect any differences to get compressed.

However, it’s unclear what to make of these different numbers, given that a number of studies have found that Ct values either aren’t indicative of transmissability or mean different things in vaccinated and unvaccinated people.

This study from the Netherlands (https://www.medrxiv.org/content/10.1101/2021.08.20.21262158v1.full.pdf) finds that even at a same Ct value, vaccinated cases were less likely to have infectious virus (measured by percent culture positive). Eyeballing their curves from Figure 1 suggests that vaccinated people are 33% less likely to have culturable viruses at similar Ct values:

Figure 3: Percent Culture Positive versus Ct values (viral load) for vaccinated and unvaccinated individuals

The giant UK study says directly that viral loads (Ct values) are not very predictive of onward transmissibility. They hypothesize that “Vaccination may act by facilitating faster clearance of viable infectious virions, but leaving damaged ineffective virions behind that still contain PCR-detectable RNA”

My takeaway from this is that Ct values are not a very useful guide in estimating vaccine effectiveness against transmission, and we should be looking at contact tracing or percent culture positive instead. This is rather awkward since the Provincetown study which led the CDC to state that vaccinated people could transmit Delta as easily as unvaccinated people was looking entirely at Ct values.

I’ve found two studies that looked at percent culture positive between vaccinated and unvaccinated people. The study from the Netherlands reports that “The SARS-CoV-2 culture of nasopharyngeal swabs was positive in 68.6% of vaccinated HCWs versus 84.9% of unvaccinated HCWs with primary infections (p = 0.005, t-test)”. This corresponds to a 20% vaccine effectiveness against transmission. However, most of the vaccinated infections were Delta strain and most of the unvaccinated infections were pre-Delta, making it difficult to interpret this number. Just eyeballing their figure of percent culture positive v. viral load suggests that vaccination reduces the risk of transmission by about 33%.

The Wisconsin study (https://www.medrxiv.org/content/10.1101/2021.07.31.21261387v5.full.pdf) found that percent culture positive was similar between vaccinated and unvaccinated individuals. However, they cultured few enough samples that the error bars are quite large.

Summary of transmission numbers

I was able to fnid 3 and a half studies looking at vaccine effectiveness against transmission. Together they report numbers of 40-60%, 25%, 20-33%, and -28%. The negative number is from the Wisconsin study, and I don’t weight that one very heavily due to the small sample size. The other three all suggest somewhere in the 20 - 50% range, probably near the lower end of that range