Lip reading

Lip reading, also known as lipreading or speechreading, is a technique of understanding speech by visually interpreting the movements of the lips, face and tongue when normal sound is not available, relying also on information provided by the context, knowledge of the language, and any residual hearing. Although primarily used by deaf and hard-of-hearing people, people with normal hearing generally process visual information from the moving mouth at a subconscious level.

Process

In everyday conversation, people with normal vision, hearing and social skills sub-consciously use information from the lips and face to aid aural comprehension and most fluent speakers of a language are able to speechread to some extent (see McGurk effect). This is because each speech sound (phoneme) has a particular facial and mouth position (viseme), and people can to some extent deduce what phoneme has been produced based on visual cues, even if the sound is unavailable or degraded (e.g. by background noise).

Lipreading while listening to spoken language provides the redundant audiovisual cues necessary to initially learn language, as evidenced by Lewkowicz who in his studies determined that babies between 4 and 8 months of age pay special attention to mouth movements when learning to speak both native and nonnative languages. While after 12 months of age enough audiovisual cues have been attained that they no longer have to look at the mouth when encountering a native language, hearing a nonnative language spoken again prompts this shift to visual and auditory engagement by way of lipreading and listening in order to process, understand and produce speech. 

Lipreading has been proven to not only activate the visual cortex of the brain, but also the auditory cortex in the same way when actual speech is heard. Research has showed that rather than have clearcut different regions of the brain dedicated to different senses, the brain works in a mutisensory fashion, thus making a coordinated effort to consider and combine all the different types of speech information it receives, regardless of modality. Therefore, as hearing captures more articulatory detail than sight or touch the brain uses speech and sound to compensate for other senses.

Speechreading is limited, however, in that many phonemes share the same viseme and thus are impossible to distinguish from visual information alone. Sounds whose place of articulation is deep inside the mouth or throat are not detectable, such as glottal consonants and most gestures of the tongue. Voiced and unvoiced pairs look identical, such as [p] and [b], [k] and [g], [t] and [d], [f] and [v], and [s] and [z]; likewise for nasalisation (e.g. [m] vs. [b]). It has been estimated that only 30% to 40% of sounds in the English language are distinguishable from sight alone.

Thus, for example, the phrase "where there's life, there's hope" looks identical to "where's the lavender soap" in most English dialects. Author Henry Kisor titled his book What's That Pig Outdoors?: A Memoir of Deafness in reference to mishearing the question, "What's that big loud noise?" He used this example in the book to discuss the shortcomings of speechreading.

As a result, a speechreader must depend heavily on cues from the environment, from the context of the communication, and a knowledge of what is likely to be said. It is much easier to speechread customary phrases such as greetings or a connected discourse on a familiar topic than utterances that appear in isolation and without supporting information, such as the name of a person never met before.

Difficult scenarios in which to speechread include:

• Lack of a clear view of the speaker's lips. This includes:
  • obstructions such as moustaches or hands in front of the mouth
  • the speaker's head turned aside or away
  • dark environment
  • a bright back-lighting source such as a window behind the speaker, darkening the face.
• Group discussions, especially when multiple people are talking in quick succession. The challenge here is to know where to look.
• use of an unusual tone or rhythm of speech by the speaker

Tips for Lip Reading

Lip reading, also know as speechreading, is difficult because only 30% of the speech can be seen, the other 70% is inferred by context clues. Thus, there are little things that can be done to make the process a little easier. Learning to lip read is like learning to read a book. A novice lip reader will concentrate on each sound, and may miss the meaning. Lip reading will be more effective if you receive the message as a whole rather than each individual sound.

• Make sure you can see the speaker’s face clearly.
• Hold the conversation in a quiet environment, with good lighting, and not a lot of visual distractions.
• Make sure that light is behind you, not the person you are trying to lip read.
• Gently remind people that you need to see their face when they forget and look down or away from you.
• Ask for the topic of the conversation, if you are not sure.
• If the speaker over–exaggerates, or talks too loudly, gently request that they speak normally.
• Remind speakers to move their hands or other objects away from their face.
• If you still don’t understand after a repeat, ask the speaker to rephrase.

Lipreading is a skill that is easier to develop in those who have experience with spoken language. In one study by Tonya R.Bergeson adults who progressively became deaf, are able to read lips much better than those who suddenly became deaf.

Use of speechreading by deaf people

Speechreaders who have grown up deaf may never have heard the spoken language and are unlikely to be fluent users of it, which makes speechreading much more difficult. They must also learn the individual visemes by conscious training in an educational setting. In addition, speechreading takes a lot of focus, and can be extremely tiring. For these and other reasons, many deaf people prefer to use other means of communication with non-signers, such as mime and gesture, writing, and sign language interpreters.

To quote from Dorothy Clegg's 1953 book The Listening Eye, "When you are deaf you live inside a well-corked glass bottle. You see the entrancing outside world, but it does not reach you. After learning to lip read, you are still inside the bottle, but the cork has come out and the outside world slowly but surely comes in to you." This view—that speechreading, though difficult, can be successful—is relatively controversial within the deaf world; for an incomplete history of this debate, see manualism and oralism.

When talking with a deaf person who uses speechreading, exaggerated mouthing of words is not considered to be helpful and may in fact obscure useful clues. However, it is possible to learn to emphasize useful clues; this is known as "lip speaking".

Speechreading may be combined with cued speech—movements of the hands that visually represent otherwise invisible details of pronunciation. One of the arguments in favor of the use of cued speech is that it helps develop lip-reading skills that may be useful even when cues are absent, i.e., when communicating with non-deaf, non-hard of hearing people.

See also

• Audio-visual speech recognition
• Forensic speechreading
• Motor theory of speech perception
• Mouthing
• Read My Lips (disambiguation)
• Reading (process)
• Silent speech interface
• Ventriloquism
• Visual capture

External links

• CSAIL: Articulatory Feature Based Visual Speech Recognition - To develop a visual speech recognition system that models visual speech in terms of the underlying articulatory processes.
• an MRI video of a person speaking, showing tongue movements not visible to a lip reader