by revere, cross-posted from Effect Measure

The latest study on flu vaccine effectiveness in children has been well discussed in the MSM and the flu blogs, so I’ll point you to those excellent pieces (Branswell, crof, Mike Coston at Avian Flu Diary) and just add some things not covered elsewhere. The full text of the article is available for free at JAMA and it’s a pretty good read, so if you want to see for yourself what is involved I urge you to read it, too. First, let me back up a bit and connect this to the controversy about observational and randomized clinical trials we’ve been discussing here of late (before my grant writing interfered, anyway).

There is ample evidence, both biological and observational, that immunization for seasonal influenza “works.” But how well does it work? And what is the best way to do it? This is not just a scientific question (although it is a scientific question) but also a policy question. If it doesn’t work at all or only minimally, then the policy of many national health departments is misguided and possibly dangerous. The question of whether flu vaccination works has been raised by (in our view) misguided zealots who demand standards of proof difficult to obtain (blinded and randomized trials) under the guise this is the only acceptable kind of evidence. We have previously registered our disagreement with this view (here, here), not least because even the availability of this evidence won’t answer the questions. In fact reading some of the literature and commentary about this makes us wonder if the RCT zealots even understand why randomization is useful (and when possible, it is; see here, here, here, here, here, here). Just as importantly, the JAMA study illustrates why RCTs won’t answer the question. More about that in a moment.

A quick synopsis of the JAMA study (you can read more details in the paper itself or in the linked articles, above). Suppose you wanted to do a randomized trial of whether vaccinating children protects unvaccinated older adults (on the theory that the children are good spreaders of the virus). How would you do a randomized trial of a community, not of the people in it? The beauty of the JAMA paper is that the authors found a way to do that, using fairly isolated and self-contained colonies of an Anabaptist fundamentalist sect in Canada, the Hutterites (I guess it shows that religious extremism is good for something). Instead of randomizing people, they randomized the colonies, vaccinating the children in some against flu, in others against hepatitis A. Not every Hutterite colony wanted to participate and many refused or dropped out after initially agreeing (but before allocation to either vaccine). Look at Figure 1 in the paper for the reasons why some colonies were included or not. It’s a little lesson in how difficult these things are to do. Moreover this is a highly unusual situation and would simply not be feasible in almost any other set of communities. So good for these researchers for taking advantage of this opportunity. It’s one of the things that makes epidemiology as much an art as a science.

The idea was to vaccinate a large proportion (in excess of 80% as it turns out) of the healthy children between 3 and 15 years old in each of the study colonies (each colony had from 60 to 120 people of all ages) to see if vaccination against flu, as opposed to hepatitis, protected the older, unvaccinated members of the colony against flu. And because this is part of the mantra of the RCT crowd, it was done “blinded” to the vaccine status. This is not as easy as it sounds because the flu vaccine came in a multidose vial and the hepatitis vaccine in a single dose vial, so the nurses who gave it knew which was which. To remedy this the nurses who gave the vaccine where then switched out of the study for new nurses who didn’t know which vaccine was used. The new nurses did the assessment of outcomes. There were some other details related to blinding, too, that are worth reading (middle column on p. 945 of the .pdf), just to get a flavor of what you have to do to do a conventional double-blinded RCT. You might think about it next time someone says, “Why didn’t they just do an RCT? How stupid are they?”

There are a lot of these pesky details in this paper. It’s a non-trivial task. In our view they did an excellent job. I wasn’t sure it was even possible to do something like this and they showed it was. You can read the results in all the other accounts but in sum it was that if you vaccinate 80% of the healthy school aged children you obtain a very significant herd immunity effect: you protect the older unvaccinated population to the same extent as if they had themselves been vaccinated. Unfortunately there weren’t many in the usual target group of over 65, so we don’t know what would have happened if the vaccine campaign had been extended even further for the target group, or, for that matter how generalizable this result is to the more complex setting of crowded, non-isolated urban populations. In other words, this is a valid and highly important result, but it doesn’t bullet proof the controversy over flu.

Indeed the controversy continues. CDC has had a policy of targeting high risk groups, particularly the over 65. This group (to which I belong) may or may not benefit as much because our immune systems may be getting like our memories — senescent. So that’s the second question this study addresses. Is there a benefit to the over 65s if we instead target the under 15s? Yes. But this study doesn’t completely answer the policy question because the over 65s weren’t targeted. Maybe the result would have been even more dramatic. It is this age group that is most at risk, so maybe it would save more lives to target them.

Bottom line: flu vaccines work to significantly reduce outbreaks when school aged children are vaccinated in isolated populations. Although we knew that already, now the nay sayers will have to find another objection. And they will be able to do so, as I just demonstrated. Because just doing an RCT isn’t a definitive answer. It just produces more questions.

Science is like that.