LAUSR

In a recent editorial, Deborah Mayo (2022) argues that journal editors “should avoid taking sides” regarding “heated disagreements about statistical significance tests.” Particularly, they should not impose bans on statistical methods, such as Wasserstein et al.’s (2019) proposed ban on citing statistical significance and using p-value thresholds. Were journal editors to adopt such proposals, Mayo argues, they would be acting under a conflict of interest (COI) of a special kind: an “intellectual” conflict of interest. Mayo’s invocation of COIs invites consideration of how disputes over statistical method arise and are treated, and of COIs as a problem statistical methods must address. We considered these questions in the contexts of experimental high energy physics (HEP) and oncology. A potential objection to Mayo’s claim is that journal editors are entrusted with decision-making power precisely to adopt and enforce standards on publication. Journal editors are responsible for making informed, reasoned judgments about standards that distinguish credible research conclusions. This requires making personal judgments about methodological standards. To have an intellectual interest in a policy is simply to think it is a good idea. Journal editors should act on good ideas when they have them. This objection neglects crucial features of the disputes over significance testing and the “statistics wars.” These disputes involve, in Mayo’s words, “philosophical presuppositions.” These concern fundamental aspects of scientific inquiry such as examining the purpose of a statistical test and whether beliefs of investigators matter to the results of inquiry. Philosophical debates are often thought to be never-ending or unresolvable in principle and thus tend to have a bad reputation among people who are not philosophers. Perhaps some are never-ending, but even in these cases (and we do not think this is one) progress in clarifying what is at stake and eliminating nonviable positions is possible. So long as competing methodological approaches rest on differing philosophical presuppositions, to preclude the use of an approach as a matter of editorial policy would foreclose on the possibility of engaging that philosophical dispute at the level of scientific practice. The consequences of that foreclosure for a scientific discipline would be impoverishing. Our consideration of statistical issues in HEP and oncology indicates that encouragement of philosophical reflection on statistical methods is both viable and important for addressing COIs that arise in the conduct of research. Although the field of conservation biology differs from oncology and HEP in numerous significant respects, some points drawn from our examination of these fields are relevant. First, statistical methods and standards must evolve to respond to problems that arise in practice, particularly with respect to potential biases and COIs. Second, methods and standards adopted collectively by a scientific community are also susceptible to criticism and revision by that community. Finally, the specific policy of banning significance and thresholds would discourage methods that require explicit treatment of error probabilities and thus (properly used) provide useful checks on COIs. In HEP, there is a working principle governing the publication of discovery (or observation) claims requiring p-values corresponding to at least a 5σ deviation from the null prediction. The objection under consideration here would hold that the editors of Physical Review ought to be free to impose the 5σ requirement on discovery claims, even though there are some