The accuracy of four machine learning methods in predicting narrative macrostructure scores was compared to scores obtained by human raters utilizing a criterion-referenced progress monitoring rubric. The machine learning methods that were explored covered methods that utilized hand-engineered features, as well as those that learn directly from the raw text. The predictive models were trained on a corpus of 414 narratives from a normative sample of school-aged children (5;0-9;11) who were given a standardized measure of narrative proficiency. Performance was measured using Quadratic Weighted Kappa, a metric of inter-rater reliability. The results indicated that one model, BERT, not only achieved significantly higher scoring accuracy than the other methods, but was consistent with scores obtained by human raters using a valid and reliable rubric. The findings from this study suggest that a machine learning method, specifically, BERT, shows promise as a way to automate the scoring of narrative macrostructure for potential use in clinical practice.
Article
An Exploration of Automated Narrative Analysis Via Machine Learning
PLoS One
Document Type
Article
Publisher
Public Library of Science
Publication Date
10-31-2019
Disciplines
Creative Commons License
Creative Commons Attribution 4.0
Abstract
Citation Information
Jones S, Fox C, Gillam S, Gillam RB
(2019) An exploration of automated narrative
analysis via machine learning. PLoS ONE 14(10):
e0224634. https://doi.org/10.1371/journal.
pone.0224634