LAUSR

Background Stability is a critical consideration in the implementation of an automatic algorithm. Some measurements can typically be more accurate than expert manual measurements but still make large errors in the presence of confounders. This can be especially true in situations where signal averaging or high sampling rates are present, there is a low signal to noise ratio or there is an atypical morphology. When a consistent accurate measurement is difficult to make manually, then it is also difficult to manually validate that measurement's automatic measurement. This increases the criticality of stability further. JTp Measurements with high natural variability (like R to R intervals) need to have their stability evaluated with respect to truth annotations. Measurements that change more slowly like JTp can be evaluated even in the absence of truth annotations. Slow change allows one to substitute consistency for stability. First we measure JTp consistency over time under near ideal conditions. Then we repeat the analysis in the presence of different categories of confounders. Each test set's lessened consistency helps to illuminate the algorithm's difficulty with the properties common to that test set. Methods We used the PhysioNet QT Database. Additionally Noise Stress Test data was added to the records in controlled amounts. Measurement consistency was calculated in each test set. To relate the impact of variation on performance accuracy in each of those test sets, QTe measurements are also performed. Calculating JTp relative consistency with respect to the companion algorithm's annotated QTe measurement's consistency normalizes the analysis. This relates the analysis to a better understood measurement. Results It was found that changing from QTe to JTe generally increased the variability of the intervals over a record. The combined record average fared worse than the median due to an increased number of outliers. Similarly changing from QTe to QTp resulted in higher overall variability. Worst of all was replacing both end-points and comparing QTe to JTp. Additionally lower signal to noise ratios adversely effected the JTp consistency more than the QTe consistency. Performing a similar analysis on the expert involved annotations is difficult to directly compare. The annotated sections were brief, stable and relatively clean so the contribution of noisy areas was not equal. Further it was not possible to further add noise, however, the trends were similar. Discussion Of course, a lowering of consistency is not proof of absolute difficulty since enhancements or other algorithms might perform better. For instance, signal averaging over longer periods of time could reduce variability at the expense of locating dynamic changes quickly. However, identified areas where current repolarization measurements in a similar computing environment perform better may help inform suitability of measurement in those environments or direct further developmental work.