Introduction
Social policies often have broad impacts. We consider them to be powerful potential mechanisms to improve population health and promote health and racial equity. Research on the causal effects of social policies on health can deliver critical evidence on whether or not to adopt a particular policy.
One way that researchers evaluate policies is by comparing changes in health outcomes over time in places that did and did not adopt a policy of interest. However, one important methodological challenge with this approach is that new policies are rarely adopted or implemented in isolation. Co-occurring policies happen when multiple related policies are adopted in the same place simultaneously or in close succession. For example, a state government that moves to overhaul its social safety net is likely to change multiple related policies in a single wave.
Co-occurring policies create a conundrum because if all the policies are related to the health outcome of interest, then the co-occurring policies confound one another. So, in our causal analyses, we will have confounding bias if we don’t adjust for them. But if we do adjust for them, we often encounter problems with imprecise or unstable estimates. What all this means is that in many cases, our studies will be underpowered, or we will think we are isolating the effect of an individual policy, when in fact what we’ve measured is the combined effect of the individual policy with some other unmeasured but co-occurring policy.
This methodological problem is particularly significant for research seeking to advance racial equity. Policies are one of the most critical drivers of and solutions to racial health inequities. Disentangling the effects of co-occurring policies to provide evidence about which policies affect health—and for whom – is one of the most important steps the scientific community can take to help achieve racial health equity.
Example Research
An E4A grantee evaluated the effect of state-level college affirmative action bans on smoking and alcohol use among underrepresented minority adolescents. They did this by leveraging variation in when and where the policy was adopted. The authors measured how smoking and alcohol use changed over time in states implementing affirmative action bans between 1991 and 2015 (Arizona, California, Florida, Michigan, Nebraska, New Hampshire, Oklahoma, Texas, and Washington) compared to states that did not implement bans during the same time period.
Policies are rarely randomized, but comparing places that adopted a policy to places that did not offers a promising alternative to randomization, because comparing trends over time in states with and without the policy helps to separate the effects of the policy from other factors driving changes in smoking and alcohol use over time. In prior posts we have discussed some advantages of this approach versus conventional observational studies. Yet for this study design, we still must grapple with the challenge of co-occurring policies. In this case, the validity of the study would be threatened if other policy changes related to smoking or alcohol use occurred concurrently with affirmative action bans. The authors addressed this concern using a range of sensitivity tests, and by accounting for other related policy changes—such as alcohol and cigarette taxes—in the analysis.
Possible Approaches
We, the E4A Methods Lab researchers, recently published two scientific papers on how co-occurring policies pose a problem for research on the health effects of social policies. The first paper examines the pervasiveness of the problem across diverse social policy domains (e.g., tobacco control, immigration, education) and quantifies how it impedes the estimation of health effects. The second paper outlines a range of analytic solutions investigators can incorporate to help address the problem of co-occurring policies in their research.
Major takeaways include:
- Policy co-occurrence is pervasive and substantial across many social policy domains.
- Appropriately controlling for co-occurring policies will often make effect estimates for the policy of interest very imprecise. In other words, the confidence intervals will be wide and results may not give much new information about the impact of the policy.
- Study designs that take advantage of variation in when and where a policy is adopted are most compelling when policy adoption is truly plausibly like-random or arbitrary—for example, because of the timing of elections or legislative sessions.
- Researchers can take numerous analytic approaches to help address the problem of co-occurring policies. Careful selection of the research question, guided by deep substantive knowledge and creativity, can go a long way. For example, we can focus on subpopulations affected by the policy of interest but not affected by other co-occurring policies. Another approach to address co-occurring policies is to identify clusters of policies and evaluate their combined effects.
- Randomization is the best solution to clearly delineating the causal effects of policies. When possible, randomly staggering the rollout of policy changes can help reveal whether and to whom the policy is beneficial.
We hope these papers will help investigators recognize this important challenge in their policy research and incorporate techniques that will yield more compelling evidence. Understanding which policies work for whom will be particularly critical as we consider concrete strategies for dismantling structural racism and advancing health and racial equity. Stay tuned for a future post featuring the work of one of our grantees deliving into this issue in greater detail.