LAUSR

Dear Editor, We thank Sabour et al for their interest in our paper and welcome their suggestions. They pointed out that Cohen's kappa coefficient is influenced by the prevalence of each category. We agree that the magnitude of kappa is affected by the prevalence of the attribute. Obtaining PABAK (prevalence-adjusted bias-adjusted kappa) would address this problem. However, many experts are critical of the use of PABAK because the effects of bias and prevalence on the magnitude of kappa are themselves informative, and the PABAK coefficient on its own is uninformative because it relates to a hypothetical situation in which no prevalence or bias effects are present. Indeed, it is believed by some experts that the effects of the prevalence “penalize” the value of kappa in an appropriate manner making it more relevant to the real-world setting. Sim et al recommended using PABAK in addition to kappa. We do agree however with Sabour et al that we should have reported bias, prevalence and PABAK in our manuscript. Sabour et al also claim that weighted kappa is a better statistic when a variable with more than two categories or an ordinal scale is used. Although we agree that weighted kappa is better for ordinal variables, their concern is not valid for our study as all our variables involved categories on a nominal scale. Weighted kappa is an appropriate study when researchers are more interested in agreement across major categories in which there is a meaningful difference, or certain disagreements are considered more serious than others. Weighted kappa is typically used for categorical data with an ordinal structure, such as the categories high, medium, or low presence of a particular attribute. For example, we may not care whether one pathologist categorizes a histopathological specimen as normal and another categorizes it as benign, but we do care if one categorizes it as normal and the other as cancer. However, in a pediatric ICU setting, all abnormal lung findings are equally important, making the weighted kappa redundant. Therefore, our research methodology reflects clinical practice by computing Cohen's kappa instead of weighted kappa. We also agree with Sabour et al's assertion that validity/accuracy (sensitivity in our study) and reliability/precision (Kappa in our study) are two different concepts. We have indeed evaluated and reported them separately (Tables 2 and 3). In summary, we appreciate the recommendations by Sabour et al. Further studies on this topic should report the bias, prevalence, and PABAK along with kappa coefficient.