LAUSR

Purpose Many models of competency-based medical education (CBME) emphasize assessing entrustable professional activities (EPAs). Despite the centrality of EPAs, researchers have not compared rater entrustment decisions for the same EPA across workplace- and simulation-based assessments. This study aimed to explore rater entrustment decision making across these 2 assessment settings. Method An interview-based study using a constructivist grounded theory approach was conducted. Gastroenterology faculty at the University of Toronto and the University of Calgary completed EPA assessments of trainees’ endoscopic polypectomy performance in both workplace and simulation settings between November 2019 and January 2021. After each assessment, raters were interviewed to explore how and why they made entrustment decisions within and across settings. Transcribed interview data were coded iteratively using constant comparison to generate themes. Results Analysis of 20 interviews with 10 raters found that participants (1) held multiple meanings of entrustment and expressed variability in how they justified their entrustment decisions and scoring, (2) held personal caveats for making entrustment decisions “comfortably” (i.e., authenticity, task-related variability, opportunity to assess trainee responses to adverse events, and the opportunity to observe multiple performances over time), (3) experienced cognitive tensions between formative and summative purposes when assessing EPAs, and (4) experienced relative freedom when using simulation to formatively assess EPAs but constraint when using only simulation-based assessments for entrustment decision making. Conclusions Participants spoke about and defined entrustment variably, which appeared to produce variability in how they judged entrustment across participants and within and across assessment settings. These rater idiosyncrasies suggest that programs implementing CBME must consider how such variability affects the aggregation of EPA assessments, especially those collected in different settings. Program leaders might also consider how to fulfill raters’ criteria for comfortably making entrustment decisions by ensuring clear definitions and purposes when designing and integrating workplace- and simulation-based assessments.