Introduction
E4A funds research to determine whether an intervention (e.g., policy, program, or practice) is likely to impact population health or health equity. When thinking about whether a study is designed to demonstrate the causal influence of an intervention, it’s important to consider whether it was adequately powered – meaning enough people participated in the study for the results to indicate whether the intervention is effective.
There isn’t a universal sample size that yields adequate power. The appropriate sample must be determined for each research study based on the expectations regarding how much the intervention will impact outcomes.
Example Research Questions
When thinking about the likely impact of an intervention, one thing to consider is whether the effect on the outcomes of interest is likely to be small, medium, or large. When considering a research question about an intervention that has a larger effect (e.g., the effect of smoke free air policies on secondhand smoke exposure) the sample size, or number of study participants, may only need to be in the hundreds. When considering a research question about an intervention that has a smaller effect size (e.g., the impacts of compulsory schooling laws on mortality and obesity) the study would need to have a much larger sample size, perhaps in the thousands to tens of thousands.
Possible Approaches
Large effect sizes may be unrealistic for social interventions as the mechanisms of changes are acting on intermediary levers and are more distal to the outcomes. For example, social interventions are usually designed to impact primary outcomes such as housing or economic security, so health and other outcomes will be one or more steps removed from that primary outcome and there will be a smaller effect size when considering these secondary outcomes.
At E4A, we’re particularly interested in population health outcomes. Population-level effects, rather than individual-level effects, depend on how much of the population was exposed to the intervention, how frequently the outcome occurs, and whether similar effects can be expected in portions of the population not participating in the study. Small effects can matter a lot when considering population health, especially if the intervention can be broadly implemented at little cost.
For example, one E4A-funded study is evaluating the effects of price disclosure on use of health services. This intervention could be broadly implemented for very little cost. Therefore, even a small benefit of the intervention could justify widespread adoption. In contrast, another grant is evaluating a youth development intervention for adolescents, with a fairly intensive program to enhance social and emotional learning. This intervention is likely to be widely adopted only if it has large benefits.
Population health researchers are often evaluating programs with small effects. This implies that large sample sizes are essential and that primary data collection may be unrealistic. Enhancements to existing data infrastructure such as timely collection of high quality, individual-level, geographically detailed measures in administrative and surveillance data and incorporating measures of social interventions into existing large-scale primary data collection efforts would support better-powered studies of social interventions.
Putting Evidence into Practice
Finally, given the limited resources available to implement and evaluate interventions, it may be appropriate to consider what the most meaningful effect would be (versus the smallest detectable effect). If there are significant costs associated with implementing an intervention at scale, it may be necessary to demonstrate a larger effect to justify the expense or make the intervention publically or politically viable.
Tools & Resources
Key considerations for policy-makers when thinking about whether a study should be informing decisions are:
- Did the study ensure that key inputs (e.g., sample size, the population or segments of the population that may be exposed to the intervention, the anticipated effect size, etc.) are used in their calculations and presented transparently?
- Is the hypothesized effect size large enough to impact population health and therefore justify studying and/or implementing the proposed policy or intervention?
These considerations are the subjects of our two latest Methods Notes. The first Note reviews the key inputs for conducting power calculations (Guidance for Sample Size, Statistical Power, and Smallest Detectable Effect Size Calculations). The second discusses what effect sizes (a key input to power calculations) are plausible for social interventions and important for population health (Considerations for Plausible and Important Effect Sizes in Population Health Research).
We welcome your comments and feedback on these ideas!