The currently statistical methods for identifying specific risk factors and possible beneficial treatments for Covid-19 are asking the wrong question. Many, if not most, of these studies use analysis of patient hospital records as their primary data source. The standard statistical analysis asks what rate people die in the hospital under different conditions. Unfortunately, under a simple scenario, this can conclude a beneficial medicine that also reduces the length of hospital stays of patients who will recover is harmful.
Most models work on predicting some quantity of interest (e.g. death rates), based on measurements of different factors we think will be predictive of the quantity (e.g. age, Body Mass Index [BMI], does the person smoke?, did they take a drug of interest?). A statistical model does this by positing an equation relating the values of interest. We get specific numbers for the effect sizes based on these values of interest through fitting the model to the data. By following well-established procedures to estimate the unknowns, what is associated with harmful and beneficial outcomes can be inferred. When the size of the effect is so large that it is unlikely to be due to random fluctuations, we say that a result is statistically significant.
A lot of papers analyzing risks and potential treatments of Covid-19 use the Cox Proportional Hazards Model to estimate the contributions from the measured factors. The model posits that there is an underlying rate of death. This rate is then multiplied by some amount for each measured factor. For instance, if you are looking at the death rate of people, some of whom smoke, and some of whom take a drug of interest, you will estimate the rate to be:
Baseline rate * Smoking factor * Drug factor
Where Smoking factor and Drug factor are 1 for non-smokers and for people not taking the drug, and some unknown number that needs to be estimated for smokers and people taking the drug. Factors greater than 1 are interpreted as harmful (increase the rate of dying), and factors less than 1 are interpreted as beneficial (decrease the rate of dying)
Between knowing how long people are in the study population, typically their time in the hospital, and which people died, all the unknown factors can be estimated. Measurements such as BMI can be treated as either a present/absent factor when combined with a threshold (e.g. BMI>40 vs, BMI<40), or as a continuous variable to multiply by the to-be-estimated factor.
This has a number of issues when drawing conclusions. Beyond the usual challenges of statistical modeling of dealing with a model that may or may not be of the right mathematical form, there is the additional challenge of whether this model is asking the right question. This model, when applied to hospitalization records, has an interesting confounder with people being hospitalized.
Let’s say upon hospitalization there exists two populations of people - those that will be discharged alive (group A), and those that will die in the hospital (group D). There is a drug we want to evaluate. On people in group A, it helps them improve, so they get out of the hospital in half the time of those untreated. It has no effect on people in group D. This is not a life-saving drug, but will improve the quality of life of the people in group A that take it, and reduce the usage of hospital resources, so probably a net benefit to society.
What happens if you estimate the effect of this drug on mortality using the Cox Proportional Hazards model? You’ll have the same number of deaths in the group getting the drug, but with the drug recipients spending on average half the time in the hospital. So how does this affect the estimated death rate for the people getting the drug vs. those that aren’t? Simplified a bit, it looks like as follows:
For the people getting the drug, the rate per time is:
R(drug)=#deaths/(total time in hospital for people getting the drug)
For those not getting the drug, they will, on average, spend twice the time in the hospital (a bit less since those that will die are hospitalized as long), and have, on average, the same number of deaths:
R(control)=#deaths/[(total time in hospital for people getting the drug)x2]
The actual math is a bit more complex, and has a lot of sources of variation we need to take into account, which is why the Cox Proportional Hazards model is popular (or any other model, for that matter!). The example given here is designed to be relatively straight-forward, and this explanation is oversimplifying a bit...but this is the back-of-the-envelope calculation I used to design a simulation illustrating the dangers of using models to answer the wrong question.
Some high school algebra shows that R(control) is one half of R(drug), meaning the rate of death per unit time in the hospital of the drug recipients will be higher, and the natural conclusion is that the drug is harmful. In my simulation I chose to have 10% of the population belong to group D, and got estimates that the death rate with the drug is about twice that of the people not taking the drug - so that the Cox Proportional Hazards Model will lead you to infer this drug is VERY dangerous!
We can add a smaller beneficial effect (say, save 10% of the people from group D), and still conclude the drug is harmful!
How can we improve on this situation?
- Unless you are looking to model usage of hospital resources, what matters is the end-point, not how long it took to get there. So death vs. recovery is what matters, not time to death or time to recovery. With this measurement, we would conclude the hypothetical drug has no effect, which is correct.
- This then suggests models that take a bunch of measurements, and predict an outcome...these are the same models used to pick adverts to show people online, and a branch of Statistics and Computer Science that has seen a lot of innovation in recent decades!
- Follow-up is important. Hospital records are the easiest to access….but if somebody is hospitalized, discharged to a skilled nursing facility (or home!), and later dies from the disease of interest, do we really want to count them as surviving?
Is the fundamental error in the simplified example a matter of treating "time in hospital for survivors" as part of the death rate measure at all, whereby a drug that lowers the survivor time in hospital but has no affect on those die increases the stated death rate measure since that time contributes in inverse-proportion to the measure?
ReplyDeleteYes.
DeleteWhy are people doing this? I think it comes from using models without understanding their assumptions. The reason hazard models were developed in the first place was, I believe, to deal with studies where people drop out. If people dropping out of a study is unrelated to the measure of interest (e.g. a drug for heart disease, and somebody either moves for a new job or dies in a car crash), you want to gain information from the time they were in the study, in this case indicating that there were no bad side effects. In the COVID studies of hospitalized populations, people leave the study by getting better....but are treated as independent drop-outs. So the assumptions of the model are being badly violated.
This comment has been removed by the author.
ReplyDeleteIt seems that a different basis is needed. Cost per casualty might be interesting except you would not want to find the minimum of such a function. ($0/manyDeaths)
ReplyDeleteDeaths per engagement is blind to extended stays which is a finite communal resource bad.
...it would seem that you want the minimum of E1(1+$$)*E2(1+TIME)*E3(1-Death%)...the hospital record should contain all of this even if the $$ value is worthless analytically. In principle you could throw in a (1+PAIN)E4 but E4 vs E3 is a very long running debate.