Tuesday, 18 July 2017

Emotional communication in speech and music in hearing impaired and hearing aided individuals

Written by: Rachel Hottle

The roar of a train, the laughter of children, the swell of a symphony. All are colorful aspects of daily life that are communicated to us via our sense of hearing. Hearing is an important asset that helps us respond to and interact with the world around us. Age-related hearing loss is a pervasive problem that often affects older adults’ quality of life. Such hearing loss can make it difficult to perform everyday tasks, such as determining where a sound is coming from, distinguishing speech from background noise, and understanding emotions conveyed in speech—all of which can lead to isolation and depression. Over the last few years, researchers in the SMART (Science of Music, Auditory Research, and Technology) lab at Ryerson University have become especially interested in the interaction between hearing loss and perception of emotion from auditory cues.
Hearing is an important asset to help us respond to and interact with the world.
Perceiving and correctly identifying emotion in speech is important for interacting with and responding to others, and the loss of this ability can cause social isolation in older adults. Emotion is conveyed in speech through features such as pitch contour and dynamic contrasts. For example, emotions such as anger and fear tend to be conveyed in a loud voice, while sadness and tenderness are conveyed via softer tones. In hearing impaired individuals, constrictions in the dynamic range may cause “loud” emotions to be less well differentiated from “quiet” emotions such as fear. This dynamic range is often limited in individuals with sensorineural hearing loss due to a problem known as recruitment. In sensorineural hearing loss, some of the hair cells in the inner ears die and are no longer able to convey sound information to the brain. Our brains “recruit” adjacent hair cells to vibrate at the frequency of the dead hair cells, as well as their original frequency. This causes the vibration reaching the brain to sound louder than usual. It can also cause hearing distortion, as hair cells for multiple frequencies can be vibrating at the same time due to their now double or triple function. As a result, hearing aids can be uncomfortable for individuals with severe recruitment.
It is important to investigate the effects of hearing aids on perception of emotion in speech.
To deal with the problem of recruitment, hearing aids must both amplify quiet sounds and compress loud sounds to make them less uncomfortable and minimize distortion. Since emotion in speech often relies on a wide dynamic range, it is important to investigate the effects of hearing loss as well as hearing aids on perception of emotion in speech. Our lab has found small, but promising, results regarding the effect of hearing aids on emotional perception. A study led by PhD student Gabe Nespoli found that hearing aided individuals have similar emotional responses to emotional speech as normal hearing individuals (as measured by skin conductance), but are slower and less accurate at identifying emotions presented in speech (although still better than hearing impaired individuals without hearing aids) (Nespoli, Singh, & Russo, 2016). A study conducted by Dr. Huiwen Goy, a postdoctoral fellow in the lab, found that hearing aids yielded small but significant improvements in emotion identification in speech compared to hearing impaired individuals without hearing aids (Goy, Pichora-Fuller, Singh, & Russo, 2016). While hearing aids may not be able to increase emotion recognition levels to that experienced by normal hearing listeners, we see encouraging results when compared with individuals with hearing loss who do not use hearing aids. Nonetheless, there is certainly room for improvement.
Musical emotion differs from spoken emotion in some important ways.
Although self-report questionnaires that measure hearing loss and its associated limitations do exist, until recently, no scale existed to assess the experiences of individuals with hearing loss with respect to emotional communication. In collaboration with Dr. Gurjit Singh and Stefan Launer, research scientists at Phonak, we developed the EmoCheQ (Emotional Communication in Hearing Questionnaire) to address this gap (Singh, Liskovoi, Launer, & Russo, submitted). Our seventeen-item questionnaire includes questions relating to characteristics of talkers, speech production, listening situations, and socio-emotional wellbeing. After validating our scale, we tested it with older adults with normal hearing, hearing aids, and hearing impairments without hearing aids. We found that, in the domain of talker characteristics, individuals with normal hearing reported significantly less handicap than those with hearing aids or hearing loss, while in the domain of situational factors, there was no difference between the performance of individuals with normal hearing and those with hearing aids. This provides yet more evidence that hearing aids in their current form can help with some features of emotional perception in speech, but not others.
While musical emotions rely on some of the same prosodic cues as spoken emotions (dynamic range, pitch contour, speed), musical emotion differs from spoken emotion in some important ways. Emotions in music are often redundantly coded, meaning that multiple emotional features are present at once. For example, a “sad” musical passage may be simultaneously soft, slow, and low in pitch. Since instrumental musical passages are lacking in the semantic content of emotional speech (the words we speak, which may give clues to the emotion we are attempting to convey), the emotions presented in music need to be more exaggerated to be distinguishable to the audience. All this might mean that it should be easier for hearing impaired and hearing aided individuals to parse musical emotion compared with spoken emotion. However, it is also possible that recruitment and dynamic range compression may make it more difficult for hearing impaired and hearing aided individuals to appreciate the emotion conveyed by music, which, especially in the classical genre, relies on a wide dynamic range from pianissimo to fortissimo. 
Parsing emotion is a key part of speech communication and musical enjoyment
A new preliminary study led by undergraduate Domenica Fanelli found promising results regarding the parsing of musical emotion by hearing aided individuals (Fanelli, Good, & Russo, unpublished). Her study involved the presentation of music stimuli judged to be either happy, sad, angry, fearful, or tender/calm to older adults who were either hearing impaired, hearing aided, or had normal hearing. She found that the hearing aided group was slightly better than the other two groups at distinguishing low arousal emotions such as sadness and tenderness, and performed just as well as the normal hearing group at judging high arousal emotions of happiness, anger, and fear.
While these results are preliminary, it appears from our recent work that hearing aids are beneficial for those with hearing loss in understanding emotional cues from speech and music. This is encouraging, as parsing emotion is a key part of speech communication as well as musical enjoyment. More research remains to be done to investigate if there is a way to improve hearing aid technology to further increase emotional understanding.

Fanelli, D., Good, A., & Russo, F.A. (unpublished). Perception of Emotion in
Music by Hearing-Impaired and Hearing-Aided Listeners. Paper
submitted in 2017 for partial fulfillment of the requirements of an
 undergraduate thesis project at Ryerson University. Toronto, ON.
Goy, H., Pichora-Fuller, M. K., Singh, G., & Russo, F. A. (2016). Perception
of emotional speech by listeners with hearing aids. Proceedings of Acoustics
Week in Canada. Canadian Acoustics, 44, 182-183, Vancouver, BC.
Nespoli, G., Singh, G., & Russo, F.A. (2016). Skin conductance responses to
emotional speech in hearing-impaired and hearing-aided listeners.
Proceedings of Acoustics Week in Canada. Canadian Acoustics, 44,
184-185, Vancouver, BC.
Singh, G., Liskovoi, L., Launer, S., & Russo, F.A (submitted). The Emotional
Communication in Hearing Questionnaire (EMO-CHeQ): Development
and validation.

Rachel is a fourth-year undergraduate at Swarthmore College in Philadelphia, Pennsylvania, where she is studying music and biology. During summer 2017 she volunteered as a research assistant in the SMART Lab.

Edited by: Dr. Frank Russo
Formatted by: Fran Copelli

Also published in HearingHealthMatters.org on Tuesday, July 25, 2017.


  1. The details of the speech and music connectivity which you have share within this posting that could be really amazing and informative to know. Thanks a lot. Hearing aid | speech therapy

  2. Great!! Nice post. Thank you so much for your valuable post and keep doing this. All are benefit in Hearing aid price in Chennai. https://www.adrohearingaid.com/