Even if a research claim in a scientific study is accepted as true, it may, in fact, be false. There are many contradictory claims in the published science literature, and some end up being taken as true and canonized. These situations can be considered research puzzles. Which of the two opposing claims is correct? Maybe neither is correct. How these puzzles come about and what might be done to see how puzzle parts fit together is an important current question. In this research protocol, we give parts/steps whereby false claims can be identified and hopefully resolved. Claim A or B could be true, or both could be true but under different circumstances.
➤ Version 3 (2024-09-28) |
Stan Young and Warren Kindzierski (2024). Specification Curve Analysis Protocol. Researchers.One. https://researchers.one/articles/24.09.00001v3
Dena ZeraatkarSeptember 19th, 2024 at 11:02 pm
I support the issues laid out in this manuscript and the suggested solutions.
I am a research methodologist and often work in the field of nutritional epidemiology. Nutritional epidemiology, and observational epidemiology in general, has long been criticized for producing sensational and conflicting findings, which has overall eroded confidence in the discipline. Anomalous findings in the literature—findings that attract the greatest attention and controversy—may only represent results from a minority of plausible and justifiable methods and that differences in analytic choices may explain inconsistencies in findings in the literature.
This issue, though massive, is remediable. Investigators can improve the robustness of their findings by subjecting their data to multiple analyses. One approach for achieving this is multiverse analysis or specification curve analysis. Through this approach, investigators can consider the range of all plausible results, express more confidence in results that are consistent across all or most justifiable analytic specifications, and identify analytic decisions that are most consequential in defining inferences.
A major barrier to applying this approach is the lack of access to datasets. Often, these datasets are withheld, possibly to prevent other investigators from publishing on similar topics or challenging findings, although the justification is often framed as protecting identifying information. For the advancement of science, these datasets should be made available, especially when they are generated, even partially, with public research funds.
Stan YoungSeptember 19th, 2024 at 06:53 pm
I received this comment from Nate Breznau:
"It is a good idea to establish protocols for how to interpret results going in 'every' direction due to analytical flexibility (what Gelman and Loken call the garden of forking paths). A specification curve, as far as I know, is used to visualize results. We can only go so far with this. In particular specification curves only reveal how single modeling choices impact the outcomes. In science, especially social science, the processes are hyper-complex. It is likely that it is an interaction of unique model specifications that lead to certain outcomes, and therefore we may be limited in what we can know from a specification curve. To get at this we need more than visual tools in my opinion. We need meta-analysis, or improved forms of multiverse analysis or global sensitivity analysis. But of course, visual tools are a good start."
I agree that looking into interactions is a good way forward.
Kathryn KellySeptember 7th, 2024 at 01:03 am
A thoughtful contribution to a vexing problem facing us in published literature. As this paper suggests, journal publishers could greatly improve the quality of published papers by 1) requiring the availability of underlying data so that others may review and replicate and 2) requiring adequate statistical analysis, perhaps independently, of claims of causality in topics with potentially significant consequences. Revisiting the initial PM2.5 papers comes to mind, both for revisiting the underlying data as well as claims of causal associations.
© 2018–2025 Researchers.One