Academic Journals Database
Disseminating quality controlled scientific knowledge

Exploring rater agreement: configurations of agreement and disagreement


Journal: Psychology Science
ISSN 1614-9947

Volume: 48;
Issue: 1;
Start page: 69;
Date: 2006;
VIEW PDF   PDF DOWNLOAD PDF   Download PDF Original page

Keywords: rater agreement | Configural Frequency Analysis (CFA) | exploration | agreement types and antitypes | base models

At the level of manifest categorical variables, a large number of coefficients and models for the examination of rater agreement has been proposed and used for descriptive and explanatory purposes. This article focuses on exploring rater agreement. Configural Frequency Analysis (CFA) is proposed as a method of exploration of cross-classifications of raters’ judgements. CFA allows researchers to (1) examine individual cells and sets of cells in agreement tables; (2) examine cells that indicate disagreement; and (3) explore agreement and disagreement among three or more raters. Four CFA base models are discussed. The first is the model of rater agreement that is also used for Cohen’s (1960)  (kappa). This model proposes independence of raters’ judgements. Deviations from this model suggest agreement or disagreement beyond chance. The second CFA model is based on a log-linear null model. This model is also used for Brennan and Prediger’s (1981) n. It proposes a uniform distribution of ratings. The third model is that of Tanner and Young (1985). This model proposes equal weights for agreement cases and independence otherwise. The fourth model is the quasi-independence model. This model allows one to blank out agreement cells and thus to focus solely on patterns of disagreement. Examples use data from applicant selection.

Tango Rapperswil
Tango Rapperswil

     Affiliate Program