Big Data/Machine Learning/AI
Evaluation of Research Credibility in Alzheimer’s Disease: A Comparative Analysis of Amyloid and Tau Sarah Ackley* Sarah Ackley Ackley Ackley Ackley Ackley Ackley Ackley Department of Epidemiology, Brown University, Providence, RI
Amyloid and tau are implicated in Alzheimer’s disease (AD). Scientific and popular sources have raised concerns that amyloid has received disproportionate attention relative to tau and that fraud has occurred in the amyloid literature—conditions that may increase incentives for biased statistical practices. However, systematic evidence on these subtler threats to research credibility remains limited. Advances in large language models (LLMs) and variable-power p-curve methods enable formal evaluation of bias. We analyzed PubMed-indexed abstracts referencing amyloid or tau published 1992-2024. P-values were extracted using validated LLM-based tools and analyzed. We first quantified the magnitude of the discontinuity at 0.05 using a model-free “cliff” statistic defined as the difference in probability mass between 0.04-0.05 and 0.05-0.06. We then estimated excess probability mass from 0.01 to 0.05 using variable-power p-curve models fit while excluding this interval, under the assumption that analysis manipulation operates primarily within this region. Both p-value distributions have a cliff at 0.05 (Figure). Amyloid exhibited a larger discontinuity at the 0.05 threshold than tau, with an estimated cliff-size difference of 0.015, 95% CI: (0.008, 0.023); this difference was smaller and not statistically distinguishable from zero when restricting to a later time period [difference=0.008, 95% CI: (-0.005, 0.020)]. Excess probability mass from 0.01 to 0.05 ranged from 0.18 to 0.38 depending on model, biomarker, and time period. Amyloid-tau differences ranged from 0.046-0.049; this difference was substantially attenuated when restricting to a later time period (difference range: 0.020-0.026). Differences between amyloid and tau likely reflect publication timing and evolving norms in statistical reporting, as bias signatures are comparable recently. Strong signatures of bias for both biomarkers underscore the need for rigorous evaluations of research credibility.