An article by Johannes Haushofer and Jessica Metcalf is a full-scale handbook for policy makers. Researchers tried to assess the existing evidence-based methodology on its applicability in evaluation of policy interventions. The major focus is placed on the non-pharmaceutical interventions` assessment, as even though they proved to be the most effective in constraining coronavirus spread in theory, the practice may be strikingly different
Johannes Haushofer, C. Jessica E. Metcalf, Science
The only approaches currently available to reduce transmission of the novel coronavirus severe acute respiratory syndrome–coronavirus 2 (SARS-CoV-2) are behavioral: handwashing, cough and sneeze etiquette, and, above all, social distancing. Policy-makers have a variety of tools to enable these “nonpharmaceutical interventions” (NPIs), ranging from simple encouragement and recommendations to full-on regulation and sanctions. However, these interventions are often used without rigorous empirical evidence: They make sense in theory, and mathematical models can be used to predict their likely impact (1, 2), but with different policies being tried in different places—often in complicated combinations and without systematic, built-in evaluation—we cannot confidently attribute any given reduction in transmission to a specific policy.
Because many of these interventions differ from each other in terms of their economic and psychological cost—ranging from very inexpensive, in the case of interventions based on behavioral economics and psychology, to extremely costly, in the case of school and business closures—it is crucial to identify the interventions that most reduce transmission at the lowest economic and psychological cost. Randomized controlled trials (RCTs) are one of several methods that can be used for this purpose but surprisingly have received little attention in the current pandemic, despite a long history in epidemiology and social science. We describe how RCTs for NPIs can be practically and ethically implemented in a pandemic, how compartmental models from infectious disease epidemiology can be used to minimize measurement requirements, and how to control for spillover effects and harness their benefits.
How can RCTs be practically and ethically conducted in a pandemic? In a typical RCT, a subset of randomly chosen individuals or regions receives an intervention, and a randomly chosen control group receives no intervention or a different intervention. The random assignment ensures that any later differences between the groups can be attributed to the intervention. During an outbreak, policy-makers must decide which interventions to impose when, and when to loosen them again. It will rarely be feasible in this context to omit individuals or regions entirely. However, policy-makers can use systematic timing of such interventions to both protect the population and understand the impact of the intervention. For example, when experts begin to think that measures can be loosened, this can be done gradually, so that evaluation is possible: A subset of randomly chosen locations (such as counties or municipalities) begins, and others gradually follow suit. Comparison of the “early” to the “late” regions makes it possible to estimate the effects of the intervention.
This “phase-in” or “stepped-wedge” approach can be used at any point during the pandemic. At the beginning, protective measures can begin early in some areas and somewhat later in others. During the pandemic, periods of loosened measures may be necessary to restore a sense of normality and keep essential services working, or measures may have to be tightened to limit further spread of the virus; these periods can also be systematically timed to evaluate their impact. In extended versions, different interventions can be tested against each other, and different locations can tighten or loosen different subsets of restrictions; for example, schools could be opened back up, whereas nonessential businesses remain closed.
Governments and organizations could work with scientists to choose an experimental design, implement and keep track of the treatment assignment, and measure outcomes. Studies of this kind can now often be done in nimble and practicable ways, reducing the oversight and time burden on implementing partners. Interventions could range from messaging campaigns to promote social distancing to laws and regulations. Where full randomization (without phase-in) is possible, this may be desirable to increase statistical power (3).
RCTs are, of course, not the only method for estimating the impact of NPIs. Where randomization is not feasible, the “natural experiments” created by some policies can be exploited, such as quasi-arbitrary cutoffs (for example, the reopening of stores below a certain square footage). Observational studies, often integrated with mathematical models have also contributed important insights.
Great care must be exercised to make RCTs ethical. Several considerations are relevant: The approach may be ethically justifiable because there are two sources of uncertainty around most interventions. For any intervention, it may be uncertain whether its benefits in terms of reducing disease transmission exceed its economic and psychological costs or how these costs and benefits relate to those of other interventions. At the same time, it is difficult to identify a single “correct” moment to loosen or tighten protective measures, as illustrated by ongoing policy debates. Thus, equipoise may be satisfied in terms of costs, benefits, and timing. Policy-makers are therefore neither knowingly withholding a beneficial intervention from constituents nor knowingly imposing a harmful one. This uncertainty is likely to make staggered tightening or loosening of an intervention more acceptable to the public.