Beyond Kappa: Comparing Methods for Assessing Agreement of Cross-Sectional, Replicated, and Longitudinal Binary Outcomes

• Activity Number: 71
• Type: Contributed
• Date/Time: Sunday, July 31, 2011 : 4:00 PM to 5:50 PM
• Sponsor: Biometrics Section
• Author(s): Yi Pan*+ and Charles Rose and Yan Ma and Joseph Carrasco and Michael Haber and Wendy Keitel and Harry Keyserling and Robert Jacobson and Gregory Poland and Michael M. McNeil
• Companies: Centers for Disease Control and Prevention and Centers for Disease Control and Prevention and Cornell University and University of Barcelona and Emory University and Baylor College of Medicine, Houston TX and Emory University and Mayo Clinic and Mayo Clinic and Centers for Disease Control and Prevention
• Keywords: agreement ; binary ; kappa ; longitudinal ; adverse events ; anthrax vaccine

Abstract

Cohen's kappa coefficient, which was introduced in 1960, serves as the most widely employed coefficient to assess inter-observer agreement for categorical outcomes. However, kappa can only be applied to cross sectional binary measurements and is unavailable when both observers evaluate the same subjects at repeated intervals. This study reviews and compares existing methods of assessing agreement of cross sectional, replicated and longitudinal binary outcomes. Our motivating example is to assess agreement of adverse events after vaccination between clinical medical records and patients' diaries in a longitudinal vaccine trial database. First we evaluate the traditional kappa and weighted kappa. Then two extended kappa estimations to assess agreement for both replicated and repeated binary measurements, one based on a logistic model using the generalized estimating equation (GEE) and the other based on U statistics, are also considered. In addition, the intraclass correlation coefficient (ICC) and concordance correlation coefficient (CCC) which were originally developed for assessing agreement on continuous outcomes are included. We used data from a CDC randomized, double-blind placebo-controlled phase 4 clinical trial to assess safety and serological noninferiority of alternate schedules and routes of administration of anthrax vaccine adsorbed (AVA). Preliminary results indicate that moderate to good agreement was obtained comparing adverse events between clinical records and patients' diaries with the above methods.