Article
Judging facts, judging norms: Training machine learning models to judge humans requires a modified approach to labeling data
Science Advances
(2023)
Abstract
As governments and industry turn to increased use of automated decision systems, it becomes essential to consider
how closely such systems can reproduce human judgment.We identify a core potential failure, finding that
annotators label objects differently depending on whether they are being asked a factual question or a normative
question. This challenges a natural assumption maintained in many standard machine-learning (ML) data
acquisition procedures: that there is no difference between predicting the factual classification of an object and
an exercise of judgment about whether an object violates a rule premised on those facts. We find that using
factual labels to train models intended for normative judgments introduces a notable measurement error. We
show that models trained using factual labels yield significantly different judgments than those trained using
normative labels and that the impact of this effect on model performance can exceed that of other factors (e.g.,
dataset size) that routinely attract attention from ML researchers and practitioners.
Keywords
- machine learning,
- norms,
- automated decision making,
- labeling,
- bias
Disciplines
Publication Date
2023
DOI
DOI: 10.1126/sciadv.abq0701
Citation Information
Aparna Balagopalan, Gillian Hadfield, Dylan Hadfield-Menell, David Madras, et al.. "Judging facts, judging norms: Training machine learning models to judge humans requires a modified approach to labeling data" Science Advances Vol. 9 (2023) Available at: http://works.bepress.com/ghadfield/74/