Agreement Statistics Based on Total Error Per CLSI EP21-A And CLIA 1171 Using Web Tools
View Presentation View Presentation
*Lawrence I Lin, JBS Consulting Services Inc. 

Keywords: CLSI EP21-A, CLIA 1171, CLIA88 Final Rule, TDI, CP, Web Tools

Per CLSI EP21-A and CLIA 1171, validation of a test device compared to a gold standard (reference or target) device should be focusing on agreement at the individual (sample, subject) level based on total error. Such total error should be shown to be within a pre-specified or documented allowable total error (ATE). Specifically, CLIA 1171 suggests that we want 95% of data of the test device that are within ATE from paired data of the target device. Similarly, The CLIA88 Final Rule required that at least 80% of assay’s measurements must be within a proficient testing criterion (PTC), either percent and/or absolute units, from their target values. For clinical chemistry and hematology analytes, CLIA 1171 suggests to use PTC as ATE. Mirrored statistical tools such as coverage probability (CP) and total deviation index (TDI) were developed to satisfy guidance like CLIA1171 and CLIA88 Final Rule. For example, for evaluating agreement of triglycerides between paired data of a test and reference devices, the PTC is 25%. Based on CLIA 1171, we can capture CP of data within 25% (CP25%) and show that it’s greater than 0.95, or compute TDI% to cover 95% of data (TDI%0.95) and show that it’s less than 25%. Furthermore, CLIA 1171 suggests that the 95% one-sided confidence limit of CP should be > 92%. We can compute CP25% and show that its 95% lower confidence limit is > 0.92, or compute TDI%0.92 and show that its 95% upper confidence limit is < 25%. In addition, when there is disagreement, it’s important to evaluate if the disagreement is coming from inaccuracy and/or imprecision. Inaccuracy can usually be corrected thru calibration. Imprecision can usually be corrected using variation reduction exercise which is often more complex. User friendly web tools, free for public to use, had been developed using R for the above evaluation. Examples of clinical chemistry and hematology analytes with various degrees of accuracy/precision will be shown. The use of web tools will be demonstrated. For binary and ordinal data, the web tools are also available and will only be briefly mentioned.