Skip to main content
Article
Using the Amazon Mechanical Turk for Transcription of Spoken Language
Computer Science Department
  • Matthew R Marge, Carnegie Mellon University
  • Satanjeev Banerjee, Carnegie Mellon University
  • Alexander I Rudnicky, Carnegie Mellon University
Date of Original Version
1-1-2010
Type
Conference Proceeding
Abstract or Description

We investigate whether Amazon's Mechanical Turk (MTurk) service can be used as a reliable method for transcription of spoken language data. Utterances with varying speaker demographics (native and non-native English, male and female) were posted on the MTurk marketplace together with standard transcription guidelines. Transcriptions were compared against transcriptions carefully prepared in-house through conventional (manual) means. We found that transcriptions from MTurk workers were generally quite accurate. Further, when transcripts for the same utterance produced by multiple workers were combined using the ROVER voting scheme, the accuracy of the combined transcript rivaled that observed for conventional transcription methods. We also found that accuracy is not particularly sensitive to payment amount, implying that high quality results can be obtained at a fraction of the cost and turnaround time of conventional methods.

Citation Information
Matthew R Marge, Satanjeev Banerjee and Alexander I Rudnicky. "Using the Amazon Mechanical Turk for Transcription of Spoken Language" (2010)
Available at: http://works.bepress.com/alexander_rudnicky/83/