Presentation
Unit Agreement and Reliability Generalization in Management and Organizational Research
78th Annual Meeting of the Academy of Management
(2018)
Abstract
With wider recognition that most data in the organizational sciences are nested, multilevel research has flourished in management and organizational research. Two persistent and related issues that lack convergence are how to empirically justify aggregating lower-level responses to form higher-level constructs and whether aggregated higher-level variables can reliably differentiate one unit from another. Various agreement and reliability indices and their corresponding thresholds have been proposed over the years, but the extent to which these indices and cut-off values are empirically supported, generalizable across constructs, or are influenced by unit characteristics has not been fully understood. Drawing from reliability generalization research, we meta-analyze established unit agreement and reliability indices – i.e., rwg, ICC(1), and ICC(2) – for the most commonly studied unit-level constructs. Based on 667 samples (62,114 units), results show that the levels of rwg and ICC(1) reported in the literature were generally above suggested arbitrary thresholds, whereas the levels of ICC(2) were generally lower than the suggested cut-off value of .70. There was some degree of variability and stability in the aggregation indices within and across different constructs. Additionally, relevant study features accounted for varying portions of the variability in the aggregation indices for different constructs. Overall, the results allow for researchers to contextualize their findings in terms of the variables they seek to aggregate, characteristics of their sample, and the specific indices they use.
Publication Date
August 13, 2018
Location
Chicago, IL
Citation Information
Gang Wang, Liam Patrick Maher, Brandon Mead, Ernest O'Boyle, et al.. "Unit Agreement and Reliability Generalization in Management and Organizational Research" 78th Annual Meeting of the Academy of Management (2018) Available at: http://works.bepress.com/liam-maher/20/