I don't find this paper convincing.
Recall that for a synthetic control, one region is matched with a weighted combination of other regions, to best mimic the behavior of the region of interest. In Germany, different regions adopted mandatory face mask rules at different times; this allows regions tat adopt face mask rules to be compared with synthetic controls that adopted rules later. The assumption that a region is otherwise identical to its synthetic controls is a very strong assumption, and worth challenging.
 |
| Figure 2 from Mitze et al. Note that the new cases in Jena drop to nearly zero shortly after the introduction of mandatory face masks April 6. This is too strong of an effect. Note also in Panel B that the synthetic control on matched Jena for a brief period before the interventions of interest. |
Above I have copied one of the key figures from Mitze et al. Jena adopted face mask rules well before other states or regions in Germany. A few days later, the number of cases in Jena flatlined; to take these data at face value, we conclude that wearing face masks is nearly enough of an additional measure to stop the spread of COVID. This effect size is implausible to me, and I don't believe is supported by experiences elsewhere. Therefore, we can conclude something else must be going on.
This gets back to the strawman hypothesis I suggested in my previous post. Perhaps when Jena became the first-in-the-nation to mandate face mask usage, people started conforming to other health-related restrictions better? I don't know how plausible this explanation is, having never been to Jena, and not having set foot in Germany for nearly 20 years.
In defense of Mitze et al., they acknowledge part of this issue. "..the Covid-19 spread in a single municipality may still be driven by certain particularities and random events that may prevent a generalization of estimated effects. We therefore also test for treatment effect in districts that introduced face masks after Jena but still before they became compulsory in the corresponding federal state."
The results from this more expansive study are shown below:
 |
| Part of Figure 5 from Mitze et al. The left panel shows the average treatment effect over 32 regions (Left) and a subsample of 8 larger cities (Right). Top row shows the regions with mandatory face masks vs mandatory controls, and the bottom row shows the difference between the two lines. |
Note that the sudden drop-off of new cases observed in Jena is not reproduced; this is further evidence something special happened in Jena. The left column show a modest decrease; one that I'm not convinced isn't just a statistical artifact of matching the synthetic region in the period of time before the intervention. This is clearly shown in Panel B in the first figure, where the match between Jena and the synthetic controls breaks down in the days before the label 53 (I'm guessing around March 22). In the focus on the 8 larger cities, I find the sudden jump up in the synthetic controls unusual; especially since this jump, more than a decrease in the cities instituting the bans, seems to drive the effect.
Finally, the effect is to block 6-12 infections (depending on the group) per region. Do we trust the fidelity of our statistical models to be able to measure differences of this scale?
Despite these reservations, if I were a reviewer, I'd recommend this paper for publication. The authors are clear and open in their methods. There are a few points I'd like to see addressed in more detail (e.g. the complete drop-off of new cases in Jena), but these issues are easily fixed in revision. These complaints I'm making are easy to make because the authors are so open with what they did, what they found, and why they made the decisions they made. I believe this paper is a good contribution to the discussion; me not being convinced does not mean there isn't value in trying to figure out what you can from what has happened.
No comments:
Post a Comment