Practice of research
Sounds like a plan: pre-specified analysis plans in epidemiologic research Amanda Irish* Amanda Irish Aayush Khadka Jillian Hebert Whitney Wells Jade Benjamin-Chung Anusha Vable
Introduction. Pre-analysis plans (specified prior to beginning analysis) are a valuable component of the research lifecycle, helping to increase reproducibility and transparency and avoid pitfalls such as hypothesizing after results are known, data dredging, and cherry-picking. The use of pre-analysis plans has become standard practice in clinical trials, but has not yet been widely adopted in observational epidemiology.
Methods. We polled a convenience sample (N = 79) of people engaged in epidemiologic research asking whether, how, and why (or why don’t) they use pre-analysis plans. We also performed a scoping review to assess the extent to which completing a pre-analysis plan is reported in original research papers in the epidemiologic literature. We searched the full text of research articles in the American Journal of Epidemiology (AJE) published between Jan 2014 to Dec 2023 for terms relating to pre-analysis plans (e.g., “analysis plan”, “pre-spec*”). We further restricted to articles that specified using an analysis plan prior to conducting analyses.
Results. Among the 79 respondents to our poll, 32 (40.5%) were faculty, 27 (34.2%) were students or post-docs, and 18 (22.8%) were staff. Most (88.6%) had used a pre-analysis plan at some point; of these, 25.7% reported using a pre-analysis plan for all papers. A minority (29.1%) had seen a pre-analysis plan mentioned when reviewing manuscripts. In our AJE scoping review, of the 2555 research articles published during this period, we found 8 that included their pre-analysis plan either in the main text or linked to the plan, and an additional 7 articles that mentioned using a pre-specified analysis plan but did not include (or link to) the plan itself. Of these 15 articles, 11 were observational studies.
Conclusion. Our poll suggests there is considerable interest in and informal use of pre-analysis plans; however, this has not yet translated into widespread or standardized use in observational epidemiology.
