LAUSR

In their paper ‘Responsibility, second opinions and peerdisagreement: ethical and epistemological challenges of using AI in clinical diagnostic contexts’, Kempt and Nagel discuss the use of medical AI systems and the resulting need for second opinions by human physicians, when physicians and AI disagree, which they call the rule of disagreement (RoD). The authors defend RoD based on three premises: First, they argue that in cases of disagreement in medical practice, there is an increased burden of proof (better to be conceived as a burden for justification) for the physician in charge, to defend why the opposing view is adopted or overridden. This burden for justification can be understood as an increased responsibility. In contrast, such burden does allegedly not arise, when physicians agree in their judgement. Second, in those medical contexts where humans collaborate with humans such justification can be provided, since human experts can discuss the evidence and reasons that have led them to their judgement, through which the sources of disagreement can be found and a justified decision can be made by the physician in charge. Third, unlike humantohuman collaboration, such communicative exchange is not possible with an AI system. Due to AI’s opacity, the physician in charge has no means of illuminating why the AI disagrees. Conclusively, the authors propose RoD as a solution. RoD suggests that a second human expert should be consulted for advise in cases of human–AI disagreement. Once AI systems become more widespread in clinical practice, it can be expected that such type of disagreement occurs more frequently. AI, after all, is being implemented, because it promises, among others, higher accuracy, which implies that some abnormalities will be detected that the physician would have missed. Hence, it is laudable to discuss the moral implications of disagreement for clinical practice and consider whether these cases are analogous to those of in which human experts disagree. In the following, we will focus in particular on the first premise of the argument, consider whether the stated asymmetry between agreement and disagreement indeed holds and what the implications are for RoD. We propose a more refined idea of medical expertise and we outline some concerns regarding the efficiency of medical AI, if RoD was adopted. The authors’ view on disagreement is best expressed in the following paragraph: ‘Our main question of moral responsibility emerges in cases of disagreement between the initial and the second opinion. Without a disagreement, the physicianincharge has no reason to assume they could be mistaken, as all available evidence and the physician’s own diagnosis are reaffirmed by the second opinion, independent of the correctness of said diagnosis. As far as responsibility goes, physicians are epistemically justified in their diagnosis if another physician comes to the same conclusions, barring unusual circumstance. A disagreement between initial and second opinion, however, establishes the burden of proof as falling on the physicianincharge: as the bearer of responsibility for the final decision, their disagreement with a peer opinion on the same diagnosis ought to be justified’. This view is strongly impacted by the idea that physicians are experts and agreement between the initial judgement and second opinion ought to increase their confidence levels in the rightness of the diagnosis. The reverse happens in cases of disagreement: confidence in the rightness of the diagnosis should decline, if they disagree and, additional justificatory weight (the weight of the burden of proof, as they call it), ends up on the shoulder of the physician in charge. First, this picture neglects the fact that the very step of requesting a second opinion already requires a justification as to why confidence has been low, why specific colleague has been chosen for consultation and what gains and insights have emerged from this collaboration. These are justificatory burdens that are on surface not at all less weighty in cases of agreement than they are in the cases of disagreement. Second, while having more confidence in a medical judgement might in general be justified in cases of agreement, each individual case of agreement requires a separate justification: a reason for reaching the same conclusion. Without understanding these reasons, we cannot know whether coming to the same conclusion should indeed be called agreement, or must rather be seen as an effect of the employment of different heuristics by both physicians, as conclusions emerging from different reasons or even just mere coincidence. Through the process of exchanging reasons, medical professionals examine each other’s levels of expertise and quality of judgement and, thereby, validate each others expertise. In those cases that induce the need for collaboration, expertise has to be continuously reestablished through reason exchange both in cases of agreement and disagreement. Expertise without such exchange of reason is a label that might suffice in public settings, where nonexperts might have good reasons to frequently rely on expert knowledge, but cannot suffice when a particular medical situation leads to uncertainty and poses epistemic problems with farreaching consequences. In short, expertise in medical practice should not be understood as a label that guarantees epistemic certainty. So, contra Kempt and Nagel, even in cases of agreement, medical professionals must be wary that their judgement and that of their colleagues is fallible. We largely concur with the author’s second premise that the burden for justification poses a challenge for human–AI collaboration, because medical AI systems are typically opaque, meaning that they are inscrutable for humans. If it is true— as established before—that in collaborative settings, justificatory burdens arise both in cases of agreement and disagreement, we see now that neither can be sufficiently fulfilled in human–AI collaboration; physicians cannot understand or explain the AI and, therefore, cannot identify why an AI system came to agree or disagree with the initial judgement. The severity of this explanatory problem varies depending on the context. Arguably, algorithmic opacity seems less concerning in relatively simple diagnostic AI applications, where link uncertainty—the relation between the actual phenomenon in Medical Humanities, University Medical Center Utrecht, Utrecht, The Netherlands Department of Values, Technology and Innovation, TU Delft, Delft, Netherlands