LAUSR

Archives of Sexual Behavior signaled its collective interest in joining the discourse regarding replicability of scientific findings—already ongoing both within sexual science (see Sakaluk, 2016; Sakaluk & Graham, 2018; Seto, 2019) and beyond (Nelson et al., 2018; Spellman, 2015; Vazire, 2018)—with its publication of the invited Guest Editorial by Lorenz (2020). Proximally, Wisman and Shrira (2020) published an article in Archives describing a trio of experiments ostensibly evidencing “sexual chemosignals” and their impact on directing men’s sexual arousal and motivation. I critiqued both of these papers (Sakaluk, 2020), out of concern that Archives was engaging in performative methodological reform: that Archives wanted to reap the optical benefits of talking the reform-talk, but wasn’t yet capable of (or even interested in) walking the reform-walk. To summarize my critiques beginning with Wisman and Shrira (2020): I subjected the focal effects from their three experiments to 10 different tests of “evidential value,” a concept that for me is agnostic as to whether a given set of effects are “significant” or not (or how big or large they are), but rather attempts to describe how compelling or convincing a set of effects is, significant or not, big or small. Those looking for a one-paragraph rehashing of the outcomes of these tests can find them summarized in Sakaluk (2020, p. 2749), but the gist is they suggest that the results of Wisman and Shrira (2020) are irreproducible and un-credible. Although some of the evidential value testing techniques I used to evaluate Wisman and Shrira (2020) were more novel and complex, many were not (e.g., eye-balling the implausibly repetitious “borderline/marginally significant” nature of the p values of their three focal tests, p = .01, .044, and .054, respectively). Indeed, distributions of p values, whether informally or formally analyzed, have been a go-to statistical canary in the replicability coal mine for years now (e.g., Simonsohn et al., 2015). This is what stung about the Lorenz (2020) Editorial: that a paper failing such an obvious replicability-minded check could be published in Archives so soon after Archives was signaling that it now finally invested in appraising and increasing the replicability of effects. I took further umbrage with the exclusively individual-level focus of the replicability promoting strategies described by Lorenz, particularly in absentia of any discussion (or implementation) of systems-level strategies that would be much more effective. I could not put the sentiments I felt at the time better than they are described by Lorenz and Holland (2020): the Editorial felt entirely “milquetoast” (p. 2761) at best, and collectively deceptive, at worst, if Archives were not to adjust any of its standard operating procedures following its increasingly replication-friendly rhetoric.1 And so, I engaged in a little scholarly catharsis, though not soon enough to pre-empt Wisman and Shrira (2020) from being featured in high-impact articles as a reliable source (see Hofer et al., 2020). I have since had the pleasure of reading three response commentaries. The first, by Imhoff (2020), argues that salvation for “solid” sexual science is to be found through the pursuit of multi-study papers.2 The second, by Lorenz and Holland (2020), lodges a counter-critique of my Commentary for having given short-shrift to qualitative methodologists and their place in the replicability/methods-reform conversation. The final, and most recent, by McCarthy et al. (2021), draws attention to the limits of heuristic appraisals of evidential value. I am grateful to both Imhoff, Lorenz and Holland, and McCarthy et al. for engaging constructively with my critical commentary. I wish to first respond to Imhoff and McCarthy et al., before engaging in more detail with the substance of Lorenz and Holland (as well as further related concerns of my own).