Exploring rater agreement: configurations of agreement and disagreement

At the level of manifest categorical variables, a large number of coefficients and models for the examination of rater agreement has been proposed and used for descriptive and explanatory purposes. This article focuses on exploring rater agreement. Configural Frequency Analysis (CFA) is proposed as a method of exploration of cross-classifications of raters’ judgements. CFA allows researchers to (1) examine individual cells and sets of cells in agreement tables; (2) examine cells that indicate disagreement; and (3) explore agreement and disagreement among three or more raters. Four CFA base models are discussed. The first is the model of rater agreement that is also used for Cohen’s (1960) (kappa). This model proposes independence of raters’ judgements. Deviations from this model suggest agreement or disagreement beyond chance. The second CFA model is based on a log-linear null model. This model is also used for Brennan and Prediger’s (1981) n. It proposes a uniform distribution of ratings. The third model is that of Tanner and Young (1985). This model proposes equal weights for agreement cases and independence otherwise. The fourth model is the quasi-independence model. This model allows one to blank out agreement cells and thus to focus solely on patterns of disagreement. Examples use data from applicant selection.