Audiovisual Integration in Speech Perception

Audiovisual Effects

The way we perceive speech can be dramatically affected by visual information from a speaker's face. For instance, if you hear the consonant /b/ while watching the face of a speaker saying /g/, you are likely to hear the sound as a /d/ (McGurk & MacDonald, 1976).

reference: McGurk, H., & MacDonald, J. (1976). Hearing lips and seeing voices. Nature, 264, 746-748.

Try this example: Watch these movies, then play them again with your eyes closed. What did you hear each time?

Movie 1.

Movie 2.

Stephens & Holt (submitted) Behavioral Data

It is likely that you heard the first as /aba/ and the second as /ada/ (or possibly /atha/). However, if you listened with your eyes closed, you may have discovered that the sounds were actually identical! Information from the movements of the speaker's mouth changed your perception of the sounds.

Researchers are uncertain whether effects like these arise through experience, or are essentially "hard-wired" from birth. We have investigated the effects of experience on audiovisual integration in speech perception by training individuals to use novel visual cues for speech. In our experiments, participants watched and listened to an animated cartoon robot that moved as it produced speech sounds. Over several training sessions, participants learned about a systematic relationship between the robot's movements and the consonants /b/, /d/, and /g/.

Here are some examples of the animated robot:

/aba/
/ada/
/aga/

After training, most participants learned the relationship between the robot's movements and the speech sounds well enough that they could identify which consonant the robot produced just by watching it move, without sound.

Among participants who learned the audiovisual correlation, the robot influenced identification of speech sounds in manner similar to what you experienced above. That is, when an ambiguous sound was accompanied by video of the robot producing /aba/, the consonant was identified as /b/ more often than when the sound was accompanied by video of the robot producing /ada/. These results indicate that experience may indeed play an important role in audiovisual speech perception.

Phonetic Context Effects
Lexical Context Effects
Audiovisual Effects