LAUSR

Dear Editor: We are happy to respond to Dr. Sabour’s criticism that is aimed at a portion of the statistical methods that we used in our recent study (Mears et al., 2017 JOR). We agree with Dr. Sabour that Pearson correlation coefficients are not actually a measure of intra-observer and inter-observer reliability, and that we should have used intraclass correlation coefficients (ICCs), which are specifically reliability measures. We have since computed these and display them in tables below. The results of this analysis lead to the same conclusions stated in our study. We disagree with Dr. Sabour’s advice for assessing accuracy. He suggested comparing our method (CFM, circle fit method) with a gold standard or reference method using a Pearson r or Spearman correlation coefficient. If a reference method was even available to do that, which is not, a correlation coefficient would actually assess validity, not accuracy. For example, one method could consistently give a score that was higher by five points, and so inaccurate, and the correlation would still be perfect (r1⁄41.0). Accuracy is correctly assessed with a Bland–Altman analysis. A Bland– Altman analysis determines if two measurement methods provide measurements that are similar on average, and with small variability in the differences between them, since if so, they could be used interchangeably. However, if a method fails to have reliability, it cannot possibly have accuracy. Since the other two methods (Tingart and Mather) failed to have reliability, we cannot compare our CFM to either of them to assess accuracy using a Bland–Altman analysis. The CFM we presented, where we quantified how different on average each method was between observers and how variable these differences were, was sufficient to rule out accuracy of the other two methods (Tingart and Mather). To assess intra-observer reliability, or test–retest reliability, three of our observers measured the same image (n1⁄4 10) on two occasions in a random order, one week apart. These results, now expressed in terms of ICCs, are shown in Table 1. It is clear to see that even the same observer cannot get consistent results with the other two methods. To assess inter-observer reliability, all five of our observers measured the same images (n1⁄4 33 images). For inter-observer reliability, all five raters were included in the computation, so the reliability is among fiver raters. These are shown in Table 2. Applying the Cicchetti and Sparrow guideline, which is frequently quoted in the reliability literature for interpreting the ICC coefficient, CFM achieved “excellent agreement” (0.75 ICC 1.00), while the other two methods only achieved “poor agreement” (ICC<0.40).