Introduction
While a variety of study designs can be used to estimate the population health impacts of social interventions, randomized control trials (RCTs) are uniquely well-suited to assess causal relationships between an intervention and outcomes. These RCTs, which call for random assignment of study participants to treatment and control groups, are often considered a gold standard approach when feasible, because randomization helps ensure study groups are comparable to each other and reduces the chance that factors outside the intervention could affect outcomes. In controlled environments, RCTs produce exceptionally reliable results; but the real world is not a highly controlled environment. Consequently conducting an RCT under real world conditions is a complex process with numerous challenges. E4A has funded several individual and cluster-randomized control trials, and the shared experiences of our grantees offer insights into how researchers and practitioners can navigate these complex issues to produce high quality evidence. These insights can also help policy makers better understand and act on evidence from RCTs.
Common Challenges to Navigate
Recruitment and retention. Even in laboratory settings there are barriers to enrollment, such as lack of interest, reliable transportation, or paid time off to participate in study activities. RCTs run in the real world face additional challenges. For example, in the context of school-based RCTs, school administrators need assurance of tangible benefits to their school. For app-based enrollment, data privacy concerns can discourage participant consent.
Compensation may be offered to offset some of these barriers, but in addition to legal and ethical considerations, it can be difficult to anticipate how much compensation is needed to effectively incentivize participation without inadvertently creating suspicion, which may deter participants.
If too few people participate, the study risks being underpowered, meaning it may not be possible to detect program effects, even if they exist. Attrition, or loss of study participants over the duration of the study, poses another threat to power and can bias results if there are differences in which participants are lost to follow up.
Tradeoffs between research and program or policy objectives. RCTs in the real world are often forced to balance priorities between the study rigor and the objective(s) of program or policy implementors. For instance, RCTs require a control group, which may raise concerns among practitioners over withholding or delaying services or resources to people who are in need. Addressing research and program priorities simultaneously requires a coordinated approach across multiple stakeholders.
Unexpected shocks. RCTs are frequently conducted in lab settings under highly controlled conditions. But in the real world any variety of external shocks – from natural disasters to political or economic turmoil – can disrupt the conditions under which an intervention is being implemented and evaluated. COVID-19 is an extreme example of an unexpected shock that has upended the settings in which real world RCTs are being carried out.
Possible Approaches
Despite the obstacles that real world dynamics pose to conducting RCTs, there are practices that can help mitigate these challenges and enhance the strength and credibility of RCT findings.
Productive partnerships. Researchers who have strong rapport with program implementors have found that these close partnerships make it easier to navigate competing priorities and deal with changes. When faced with unexpected shocks, researchers and program implementors are able to better navigate program disruptions together.
Partner(s) on the ground. Ensuring that there is a team member or point person who can engage directly with study participants and provide coordination can greatly improve recruitment and retention. For instance, E4A supported researchers evaluating the country’s first city-led guaranteed income pilot, the Stockton Economic Empowerment Demonstration (SEED), attribute much of their success in recruiting and retaining participants to having a dedicated team member who is in constant communication with program participants.
Contingency Planning. Planning ahead for attrition or potential shocks when determining recruitment targets can help ensure that studies have adequate sample sizes and power, even in the face of real world challenges.
Putting Evidence into Practice
Strong evidence is needed to understand which interventions drive change. RCTs are widely considered the most rigorous way to assess causal relationships between an intervention and outcomes; but even a well-designed RCT can face challenges when implemented in real world conditions. To increase their confidence in research findings, decision makers weighing the credibility of an RCT should be attuned to common pitfalls, and whether best practices were applied in carrying out a study.