Often, agreement studies utilize limited raters, and frequently only two. Therefore, understanding the underlying mechanisms that contribute to interrater agreement are important to the effective use and appropriate interpretation of traditional methods. One approach for studying these mechanisms is by examining interrater agreement in the context of diagnostic interpretation by true disease status. Notably, Cohen’s kappa can be rewritten to reveal insights as to how kappa relates to other quantities, such as the odds of observed agreement under independence. Hence, this re-expression of kappa yields insight into how the joint distribution of diagnostic interpretation by true disease status affects the marginal probability of each rating. Further, samples typically selected for diagnostic testing may not provide an adequate representation of the population of interest, which may affect inferential assumptions. We show how kappa can be rewritten, examine different sampling paradigms and study the advantages of competing designs. These new, yet fundamental, insights provide a different lens with which to study and interpret interrater agreement.