Introduction
Evidence for Action (E4A) strives to fund research that will contribute to improving health and racial equity. At E4A we require preregistration of all funded studies. Preregistration involves recording the planned research hypotheses, methods, and statistical analyses before collecting or analyzing data.
A preregistration plan, written in advance and made available when the research findings are reported, allows readers to identify any discrepancies between what was planned and what actually happened over the course of the research project. This transparency helps strengthen the credibility of research findings. For example, preregistration can help to identify and mitigate issues such as poorly planned analyses, “HARKing” (i.e., hypothesizing after the results are known), undisclosed multiple hypothesis testing, “p-hacking” (i.e., repeated data analysis until a statistically significant p-value is obtained), and unreported null findings that might lead to publication bias (see also the "Details & Workflow" tab here).
Preregistration is a critical component of “open science”, a movement that recognizes the importance of independent researchers reproducing prior studies in an attempt to replicate findings. A research study can only be reproduced precisely if it is fully documented with transparent scientific protocols. Preregistering such protocols reduces the chances of substantial, undocumented deviations from the original plan. Additionally, a preregistered study protocol can be critiqued by colleagues and other experts who can recommend potential improvements. Preregistration also helps when distinguishing between confirmatory and exploratory hypotheses.
We acknowledge that preregistration is not necessarily appropriate in all research. For example, early in the arc of research on a topic, preregistration may stymie exploratory analyses or limit researchers’ capacity to revise and develop appropriate analytic strategies based on their growing understanding or discovery of data limitations. However, there are a variety of different types of preregistrations that align with different research approaches, so it’s worth considering whether preregistration could be beneficial, even for more exploratory studies.
The stage of research funded through E4A is ideal for preregistration, as we typically fund later stage research meant to drive policy, practice, and programmatic decision-making. Given the aim of driving decision-making, it is also critical that the research plan, implementation, and findings all be clear, transparent, and otherwise above board, all of which preregistration helps establish.
Example Questions
A selection of example questions to answer in a preregistration document include:
- What are the main hypotheses tested in this study? Are they confirmatory or exploratory?
- What is the data source to be used, what eligibility or exclusion criteria will be adopted, and how large is the sample size?
- What are primary and secondary dependent/outcome variables and how will each be measured?
- What is the exposure/independent variable that will be evaluated, how will it be measured, and what variations in measurement will be evaluated (e.g., non-linear effects or threshold effects)?
- What covariates are considered potential confounders and how will they be measured?
- What subgroups or effect modifiers will be tested and how will they be measured?
- What specific analyses will be conducted to examine the main research question/hypothesis, including how missing data will be handled?
Each research plan must be tailored to the specific research study. The Open Science Framework (OSF) provides several preregistration templates, including their standard form. This standard form requires providing information on the background of the study, design plan, sampling plan, measured variables, and analysis plan.
Putting Evidence into Practice
The preregistration plans for E4A funded projects are on the E4A website (see the Funded Projects here). For example, E4A grantees evaluated the effectiveness of communication strategies to increase public support for state investments in affordable, accessible, and high-quality childcare for all. The study’s OSF preregistration plan is here. The study’s main hypothesis was whether different messaging strategies (e.g., a simple pro-policy message) increased support for state investment in childcare policies. The preregistration plan described the study design (i.e., randomized, controlled trials), key dependent variables and how they were measured (e.g., a 7-point Likert scale ranging from strongly oppose to strongly support the policy), the methods used in the study (i.e., simple ordinary least squares regression, a generalized form of analysis of variance [ANOVA]), and a description of the data collected (i.e., respondents to an online survey).
The published research can be found here. The study found that members of the public exposed to messaging strategies had higher support for state investment in childcare policies. For instance, one finding was that those individuals who received a “narrative” message (i.e., a story about how early childhood education policies can achieve favorable social outcomes or avoid unfavorable ones) had higher support than those who read an “inoculation” message (i.e., a warning that others might try to persuade them to refute early childhood education policies) and those who were exposed to no message.
The authors reference their preregistration plan in the published article and, throughout the study narrative, note where hypotheses and methodological decisions were included in the preregistration. Furthermore, the authors include notes when there were non-preregistered hypotheses or methods included in the study.
While not all authors explicitly address their preregistration in a published journal article, the availability of the preregistration plan allows any reader to compare the preregistered plan to the published study and identify deviations from the plan. Research that adhered closely to the preregistration plan (e.g., evaluated the planned hypotheses with the planned methods) is more credible. If numerous deviations from the preregistered plan occurred, it is plausible that the findings reflect the outcome of a long fishing process and that the results are unreliable. Comparing published methods to preregistered plans can be valuable when synthesizing a body of literature or explaining discrepancies between findings in different studies. The study herein is an excellent example of how research can effectively use preregistration and communicate it in the published findings.
Tools & Resources
There are several websites where researchers can post their preregistration plans. The most well-known is the Open Science Framework (OSF), which is where E4A grantees are expected to register. However, other options are available for preregistration, including https://clinicaltrials.gov/, the American Economic Association’s registry, AsPredicted, and the WHO Registry Network.
For more information on preregistration, we recommend the following resources: