Checking the assumptions that led to headlines reporting 8.7 million infected with COVID

Media reports, based on a Penn State University press release, have discussed "a new study [that] estimates that the number of early COVID cases in the U.S. may have been more than 80 times greater and doubled nearly twice as fast as originally believed." This has led some to conclude that "By the time governors in the U.S. forced lockdowns, COVID-19 had already extended beyond a point in which lockdowns could be effective in slowing the spread" The key piece to this chain of reasoning is the "may have been," and I believe this is a great case study in the challenges of scientific communication.

The study attempts to answer the question of how many undiagnosed cases of COVID are in the US. The answer is based on analyzing an existing Center for Disease Control (CDC) database of Influenza-Like Illness (ILI), estimating how much ILI has increased over previous years, and attributing an appropriate amount of the excess ILI to COVID cases. In transforming excess ILI at participating hospitals to COVID cases in the general population, a number of assumptions are necessary; I think the answer is highly dependent on these assumptions. I am very pleased that the authors of the study properly caveat their conclusions, and call for the appropriate follow-on studies. They are also careful to summarize their major assumptions and highlight most of them. However, I believe a number of their assumptions are flawed, and as a result, they run the risk of greatly over-estimating the true number of total cases. The only way to tell the actual number of early COVID cases in the U.S. is through further testing, as suggested by the authors, because these numbers are, after all, extrapolated from multiple assumptions. 

What this study is not is definitive proof that the shutdowns were an example of proverbially locking the barn door after the horse left.

The study highlights three key assumptions that it makes:

1. The patient population reported by the participating health care providers is representative of the state's population.
2. Medical care-seeking behavior is the same for populations with ILI and those without.
3. The total number of people seeking medical attention is unchanged since 2018.

Given that there are about 1 billion trips to the doctor each year in the US, I think assumption #3 is reasonable, even when the impact of a million or more new COVID cases is factored into this assumption. 

I have reservations about the other two.

For assumption #1, we need to assume that the 2600 health care providers reporting ILI visits to the CDC are a representative sample of all health care providers. This is unlikely, as creating representative samples is hard. In March, slightly under half the COVID-related deaths were related to nursing homes; any distortion in how nursing homes are represented in the CDC data will have a huge impact. If better-resourced and urban hospitals are more likely to participate with the CDC surveillance program, then rural populations and poorer urban neighborhoods may be under-represented...and it was the poorer neighborhoods of Brooklyn and Queens that were another large component of the NYC outbreak.  

For assumption #2 to hold, we need to believe that people without ILI were not avoiding hospitals, when people were scared of contracting COVID, and, in NYC, some hospitals were overflowing with COVID patients. Similarly, because the news was covering a global pandemic with flu-like symptoms, I think it is more reasonable to believe people with ILI were more likely to seek medical care. For these reasons, I don't believe assumption #2 holds. 

As the paper explains, both a belief that those with mild ILI symptoms are more likely to seek medical care, and a belief that those without ILI symptoms are less likely to seek medical care, leads to the estimates presented being over-estimates of the true COVID rate. We already have two paths to viewing the work as an over-estimate by the logic helpfully laid out by the authors. 

Further, to scale up the estimates from the health care providers that report ILI symptoms to the CDC to the general population, the study uses the following equation:

w = 5*p*m*\lambda*b/10000

where w is the per-state weighting factor, 5 is for the number of days in the week, p is the state's population, m=20.2 is the average number of patients seen by a physician in a day, \lambda=0.55 is a discount factor based on previous research, b is the number of physicians per 10,000 people in a state, and the 10000 scales b to be relevant to the rest of the equation.  Changes to any of these numbers will make a proportional change to the estimated prevalence of COVID; e.g. if physicians average 18 patients per day rather than 20.2, the estimated number of people with COVID is decreased by about 10%.

Finally, not all people with COVID seek medical help. The study increases its estimate to account for this, based on what was observed on the Diamond Princess cruise ship. This final factor increases the estimate from 2.8 million patients seeking medical care to 8.7 million infected Americans at the end of March.

The study cites supporting evidence. In April, New York State estimated that it had a 14% infection rate; 12.5% of those tested had antibodies, and given the estimated properties of the test, this led to a conclusion that 14% of the people sampled had COVID antibodies. There are two issues with this study. The first is that it was a convenience sample of people shopping at grocery stores. This would be equivalent to asking people at grocery stores to fill out secret sample ballots for how they want to vote in an upcoming election and using this to forecast the election. The other is that they assumed the test was very reliable; I recall that back in April there were issues of the reliability of most COVID tests. This may not have been an issue, but I'm suspicious about how high performing they reported their test was.

I think the authors of the Penn State study did an excellent job trying to get value out of an existing database. While I believe they greatly under-reported the uncertainty of their analysis by not having distributions on a lot of their assumptions, they communicated their assumptions and the sensitivity of their conclusions clearly within the scientific community. Moreover, they make a number of testable predictions, and encourage these tests to be run. The University press release drops most of the details, as press releases tend to. The headline number of 8.7 million infections is then picked up by other outlets. Journalists and bloggers then helpfully add their own analysis to this 8.7 million number, without going back to the original source and understanding the assumptions behind it. 

I believe the 8.7 million number should be treated as interesting, if true. I could as easily argue for a number closer to 500,000 (drop a factor of 3 for people without ILI being less likely to seek medical care in a pandemic; drop a factor of 4 for people with ILI being much more likely to seek medical care in a pandemic focused on ILI, and put in another 50% for non-representative samples, physicians seeing fewer than 20.2 patients/day, and \lambda being mis-estimated). The New York State study was not conducted on a randomly sampled population, so I'm hesitant to draw any conclusion from it. Until we are testing random samples of the population, we will not have a good way of estimating the true rate of COVID in the population.