The analysis of spoken language has been integral to a breadth of research in social science and beyond. However, for analyses to occur with efficiency, language must be in the form of computer-readable text. Historically, the speech-to-text process has occurred manually using human transcriptionists. Automated speech recognition (ASR) is advertised as an efficient and inexpensive alternative, but research shows this method of speech-to-text is prone to error. This paper investigates the viability of using error prone ASR transcriptions as part of the methodological process of language analysis. Results show that at the individual feature level, analysis of ASR transcriptions differ dramatically from human transcriptions. However, when the same features are used for classification, a common machine learning task, performance results between ASR and human transcriptions are similar. We present these findings and conclude with a discussion on the methodological considerations for researchers who opt to use automated speech recognition for social science research.