LAUSR

Introduction Validated competency assessment tools and the data supporting milestone development during gastroscopy training are lacking. We aimed to assess the validity of a novel formative direct observation of procedural skills (DOPS) assessment tool in gastroscopy, and study competency development using DOPS. Methods This was a prospective multicentre analysis of formative gastroscopy DOPS assessments. Internal structure validity was tested using exploratory factor analysis and reliability estimated using generalisability theory. Item and global DOPS scores were stratified by lifetime procedure count to define learning curves, using a threshold determined from receiver operator characteristics (ROC) analysis. Multivariable binary logistic regression analysis was performed to identify independent predictors of DOPS competence. Results 10086 DOPS were submitted for 987 trainees from 275 UK training centres. Exploratory factor analysis identified three distinct item groupings: representing ‘pre-procedure’, ‘technical’ and ‘post-procedure non-technical’ skills. From generalisability analyses, sources of variance in overall DOPS scores included: trainee ability (31%), assessor stringency (8%), assessor subjectivity (18%), and trainee case-to-case variation (43%). The combination of 3 assessments from 3 assessors was sufficient to achieve the acceptable reliability threshold of 0.70. On ROC analysis, a mean score of 3.9 provided optimal sensitivity and specificity for determining competency. This threshold was attained in the order of: ‘pre-procedure’ (10–24 procedures), ‘technical’ (15–74 procedures), ‘post-procedure non-technical’ skills (20–24 procedures), and overall competency (22–49 procedures) [Abstract PWE-107 Figure 1]. Higher lifetime procedure count (P<0.001), DOPS count (P<0.001), trainee specialty (P=0.028) and assessor specialty (P=0.002), trainee seniority (P=0.011) and lower case difficulty (P<0.001) were multivariable predictors of DOPS competence.Abstract PWE-107 Figure 1 Learning curves in gastroscopy as assessed by the constructs identified in factor analysis: a) Pre-procedure, b) Technical (covering Insertion & Withdrawal and Visualisation domains), c) Non-technical (covering Management, Post-Procedure and ENTS domains). Conclusion This study establishes milestones for competency acquisition during gastroscopy training and provides novel validity and reliability evidence to support gastroscopy DOPS as a competency assessment tool.