Skip to main content
Automatic Video Self Modeling for Voice Disorder
Multimedia Tools and Applications
  • Ju Shen, University of Dayton
  • Changpeng Ti, University of Kentucky
  • Anusha Raghunathan, Intel Corp.
  • Sen-ching S. Cheung, University of Kentucky
  • Rita Patel, Indiana University - Bloomington
Document Type
Publication Date

Video self modeling (VSM) is a behavioral intervention technique in which a learner models a target behavior by watching a video of him- or herself. In the field of speech language pathology, the approach of VSM has been successfully used for treatment of language in children with Autism and in individuals with fluency disorder of stuttering. Technical challenges remain in creating VSM contents that depict previously unseen behaviors. In this paper, we propose a novel system that synthesizes new video sequences for VSM treatment of patients with voice disorders. Starting with a video recording of a voice-disorder patient, the proposed system replaces the coarse speech with a clean, healthier speech that bears resemblance to the patient’s original voice. The replacement speech is synthesized using either a text-to-speech engine or selecting from a database of clean speeches based on a voice similarity metric. To realign the replacement speech with the original video, a novel audiovisual algorithm that combines audio segmentation with lip-state detection is proposed to identify corresponding time markers in the audio and video tracks. Lip synchronization is then accomplished by using an adaptive video re-sampling scheme that minimizes the amount of motion jitter and preserves the spatial sharpness. Results of both objective measurements and subjective evaluations on a dataset with 31 subjects demonstrate the effectiveness of the proposed techniques.

Inclusive pages
Document Version

Document available for download is the authors' accepted manuscript, provided in compliance with publisher policy on self-archiving. Permission documentation is on file.

Peer Reviewed
Citation Information
Ju Shen, Changpeng Ti, Anusha Raghunathan, Sen-ching S. Cheung, et al.. "Automatic Video Self Modeling for Voice Disorder" Multimedia Tools and Applications Vol. 74 Iss. 14 (2015)
Available at: