Be careful who you measure

One of the challenges with the observational studies conducted on COVID-19 is that the studied population may be different from the larger population, limiting the validity of drawing conclusions for making policies for general populations. Beyond that, many of the studies either take advantage of differences that may have unusual or undocumented causes, or attempt to match to other, similar people. However, differences may exist that were not measured or used in the matching that can affect the conclusions.




In “Low dose of hydroxychloroquine reduces fatality of critically ill patients with COVID-19” by Yu et al., 550 critically ill COVID-19 patients were studied, of which 48 received Hydroxychloroquine (HCL). All the patients were at the same hospital, and otherwise received the same standard of care. The patients averaged 68 years of age, and had surprisingly low levels of heart disease or diabetes; overall this was a very healthy cadre of seniors. There was no discussion of how they decided which patients were to receive HCL.

So what? If the 48 patients differed in some systemic, but unmeasured, manner from the other 502, the beneficial effects of HCL may be an artifact of that systemic difference. This is an otherwise healthy population of patients; how or why are they different from the general population? Does this matter? We cannot tell from the study as written.

Another example of these issues is illustrated by “Smoking and the risk of COVID-19 in a large observational population study” by Israel et al. The records of a large Israeli HMO are studied to look at the COVID-19 risks of smokers vs. non-smokers; for each COVID-19 patient, 5 patients were matched based on gender, ethnicity, and 5-year age-bracket. The conclusion was that smokers got COVID-19 at half the rate of non-smokers, with no difference for former smokers. Biological mechanisms for this differentiation were proposed to explain this difference. However, smokers differ from non-smokers in more ways than how otherwise identical exposures to coronavirus affect them. Do people tend to avoid smokers, or gather close to them? Are smokers more or less likely to work in health care? Are smokers more likely to spend time indoors, in close proximity to others, or outdoors where it is easy to stay distanced? Finally, they were looking at infections as opposed to hospitalizations or extreme outcomes. Is there any relation between smoking and the likelihood of getting tested for coronavirus? I do not know how large of an effect size is plausible for any of these differences, nor do I know Israeli society well enough to even make reasonable guesses. But based on the study design, any of these effects could show up as a difference between smokers and non-smokers.

A related challenge in conducting medical studies is in extending the results from the studied population to the larger population. An extreme example of this is with many psychology experiments, where the experiments are conducted on student volunteers. 18-22 year-olds at research universities, typically majoring in psychology, are not representative of the larger population. In designed clinical trials, there are often criteria to keep the variation between subjects down, so that there is (hopefully!) less variation in outcomes, making it easier to detect effects. However, how comfortable should we be extending the conclusion to untested populations? What can we do about this? The perfect should not be the enemy of the good. I am glad people are doing these observational studies, trying to figure out what we can learn from the experiences we are having. However, conclusions need to be tentative, and the ideal scientist should point out these issues, rather than leaving them to the reader to notice or figure out. An exemplar of this is the authors of a recent Public Health Britain study, who prominently pointed out confounders (such as occupation) that could explain their observed racial differences in health outcomes, but which they lacked the data to control for. We should also view observational studies as suggestions of where we should focus our randomized clinical trials and detailed laboratory work...especially in cases where there are significant costs from following the implied recommendations (e.g. known side-effects of HCL, known hazards of smoking).