The Henry Ford study on Hydroxychloroquine

The Henry Ford Health System published a study claiming Hydroxychlorquine (HCQ) saves lives in COVID patients. I don't find the study convincing. The study was well-analyzed based on how I was trained such studies should be analyzed. However, the study is an observational study of the patients of one health system. Then need to be careful about who they measure when drawing larger conclusions. Moreover, the patients getting HCQ met certain clinical guidelines, and thus were not like the patients not getting HCQ, so they needed to have been more careful about what they measured.

Patients had electrocardiograms assessed, and those that did not show a heart issue were treated with HCQ. Azithromycin was administered to patients with more severe COVID cases.

Treatment	#Cases	#Deaths	Percent
HCQ	1202	162	13.5%
HCQ+azith.	783	157	20.1%
azith. only	147	33	22.4%
Neither	409	108	26.4%

As these populations differed in important demographics (age, ethnicity, and co-morbidities), a Cox Proportional Hazards model was used to attempt to correct for these differences.  People over 65 years of age had 2.6 times the fatality rate of those under, white people at 1.7 times the fatality rate, and those that immediately needed a ventilator had 2.2 times the fatality rate. HCQ reduced the fatality rate by 66%, and HCQ and azithromycin together by 71%.

The people not receiving HCQ averaged 4 fewer days in the hospital than those receiving HCQ; they were also, on average, 4 years older than those receiving HCQ. Importantly, since the HCQ population averaged about 8.5 days in the hospital, the higher death rate of the non-HCQ population is not sufficient to explain the differences in length of hospital stay; if the groups were otherwise identical, and the deaths happened immediately, the non-HCQ group would have about 2 days shaved off their average stay due to dying quickly. The difference in median stay is about 3 days, so I don't think a few super-long hospital stays in the HCQ population is driving this difference. Given the issues with the Cox Proportional Hazards model I've discussed before, I am nervous about how the statistical correction may be creating artifacts. That the people not receiving HCQ are less healthy to begin with (heart issues) and older means that the baseline death rates should not be used to draw conclusions on their own. This skepticism is supported by the fact that none of the comorbidities in the model had statistically significant effects, unlike most other studies. 

To address the issue of the patients being unlike each other, the study tries creating statistical matches, and analyzing the comparable populations. They do a good job creating matches based on what they are matching on; however, they do not take into account the cardiac differences that led to the HCQ vs. non-HCQ differences in treatment at the start.

Importantly, the study reports no significant HCQ-based side-effects:

A review of our COVID-19 mortality data demonstrated no major cardiac arrhythmias; specifically, no torsades de pointes that has been observed with hydroxychloroquine treatment. This finding may be explained in two ways. First, our patient population received aggressive early medical intervention, and were less prone to development of myocarditis, and cardiac inflammation commonly seen in later stages of COVID-19 disease. Second, and importantly, inpatient telemetry with established electrolyte protocols were stringently applied to our population and monitoring for cardiac dysrhythmias was effective in controlling for adverse events.

So what does this all mean? I'm not sure. I'd need to see the data to check for a couple of statistical artifacts. Getting good data to analyze is hard. The Henry Ford Health System is, I assumed, primarily interested in doing what is best for its patients, and research is secondary their health care work. I assume they believed treating with HCQ is in the patients' best interest, assuming good cardiac health. So they did that. And set up careful checks to make sure no harm would come from the known side effects of HCQ. If I were a patient, that is what I'd want my doctors to be doing. The problem is that this destroys the assumptions needed for standard statistical analysis. Their estimated effects are different from what I've seen in other studies; this increases my confidence that the statistical artifacts I'm questioning may be causing significant changes in the estimated effects. I'm glad they've shared their research with the world. If I were an author, I'd probably have expanded the limitations section, and I would have used a model that just tried to predict survive or die, rather than worry about how long the patients were at risk for, but I think those are the only changes I would make.