How Music Moves Us

This is a reproduction of an article that was originally published in themarknews.com (January 28, 2011):

Music may be an art, but science can help explain why it makes us feel powerful emotions.

A number of reasonable proposals have been tabled over the years in an effort to explain how we experience emotion in music. This article outlines a few of these proposals and one in particular concerning movement, which has been gaining some attention recently in the growing field of music cognition.

“Ring a bell and I’ll salivate/how’d you like that?”

– “Brian Wilson,” Barenaked Ladies

Conditioning, or learning by association, is no doubt involved in our emotional response to music. We can develop a learned response to a neutral stimulus when it is paired with another stimulus that is clearly positive or negative. It’s remarkable how quickly those awkward feelings of middle school can come flooding back when we’re exposed, willingly or not, to that romantic last song of the last dance.

“It goes like this/the fourth, the fifth/the minor, fall the major lift”

– “Hallelujah,” originally recorded by Leonard Cohen, with covers by K.D. Lang and Rufus Wainwright

Many structural aspects of music will give rise to expectations. Some of these expectations are culturally specific, while others appear more universal. For example, a large leap in a melody – such as the octave that occurs at the start of “(Somewhere) Over the Rainbow” – will give rise to the same expectation regardless of whether it’s experienced in Manhattan or Mumbai. The expectation is that the melody will change direction following the leap. If this expectation is denied by a continuation of the melody in the same direction, our autonomic nervous system gets a jolt, much as it would if a tiger were to enter the room. This jolt opens a window to emotional experience. However, other aspects of the music, which lie beyond the expectancy-denying event, are necessary in order to shape the interpretation of the emotional experience.

“Wild thing/I think you move me”

–“Wild Thing,” originally recorded by The Wild Ones, with covers by The Troggs, Bryan Adams, and others

The etymology of the word “emotion” can be traced back to emovere from medieval Latin, which means “to move.” Indeed, emotions tend to have their own characteristic patterns of movement – for instance, the slow and heavy movements associated with sadness or the intense and abrupt movements associated with anger. Sensitivity to the patterns of movement displayed by those around us provides us with important nonverbal insight that helps us to navigate our social world.

The idea of music conveying a sense of movement has a long history. Movement can be implied in music through the speed of event onsets, the range of pitches, and the smoothness of transitions between pitches, to name but a few variables. Making sense of all this implied movement may require the mirror neuron system, a collection of neurons in the frontal and parietal cortex that responds to the execution and observation of goal-directed actions. My students and I believe that the mirror neuron system runs a simulation of the actions necessary to create the music, which in turn activates primitive emotion networks.

“Keep smiling through/just like you always do/till the blue skies/drive the dark clouds far away”

–“We’ll meet again,” originally recorded by Vera Lynn, with covers by Johnny Cash, The Byrds, and others

It’s interesting to note that people observing song will routinely display subtle muscle activation that is consistent with a singer’s facial movements. This muscle activation is normally too subtle to observe by eye, but it can be detected by recording changes in the electrical potential at the surface of the skin using electromyography. Within approximately 100 milliseconds of seeing and hearing a singer smile, previous research in our lab suggests that the audience is already smiling in response (albeit subtly). This example of social contagion is likely an automatic feed-forward consequence of the movement simulation referred to above. It stands to reason that if this automatic process is kept up for long enough, the smiling will rub off and influence our own mood.

In sum, we experience emotion in music through conditioning, through culturally specific as well as universal expectations, and through the internal simulation of movement.

-- Frank Russo

The Benefits of Creative Arts Therapies for Autism

**If you would like to be involved in autism research happening in our lab, or hear updates about our new autism video game, click here to join our mailing list!**

Autism Spectrum Disorder (ASD) is a neurodevelopmental disorder that affects a growing number of individuals all over the world. It is characterized by difficulties in communication. These include perceiving emotions in other individuals, language production, and emotion regulation.

In our social world, we connect to people by subtly imitating each other. Many dating and job websites often recommend mimicking gestures of an employer or a person of interest in order for that person to feel closer to you. This is a behaviour that actually happens naturally and non-consciously in many everyday social interactions. Mimicry facilitates closeness, as a result of us perceiving the other person as being more akin to ourselves, and therefore we accept them more readily. In addition, it is thought that we understand their emotions more readily, because we are producing echoes of those same emotional expressions ourselves. In autism, this tendency to imitate across verbal and body language is impaired, leading to deficits in social understanding and acceptance.

Photo source: http://www.hanen.org

Arts-based therapies provide an abstract route of communication that reaches individuals with autism. The enjoyment factor also creates a safe environment to experiment with imitation. It’s often been found that autistic individuals respond more towards emotion that is directed through song and creating visual art. While there is lots of room to conduct more research, initial research findings are incredibly promising. In the present article we will address a few of these findings and discuss the mechanisms by which these arts therapies may work. We will also discuss these findings in relation to work in our own laboratory, where we have found beneficial effects of song and movement therapy for children with autism.

Music

Music therapy provides a promising route to emotional communication training and behaviour regulation in autism. In 1964, Norddoff and Robins developed Improvisational Music Therapy, which incorporated the use of musical instruments as voice during conversation. This therapy was later developed into Creative Music Therapy, yielding positive results in communicative responses from once non-verbal children.

Photo source: www.hanlen.com

Drawing from Nordoff and Robins, Miller and Toca developed Melodic Intonation Therapy in 1979. They paired signs, then words with musical tones to assist in speech production with a 3-year-old nonverbal autistic boy. After 35 sessions, the child was capable of producing, responding, and understanding simple sentences. 

Auditory Integration Training therapy has been used to treat self-destructive behaviour. A particular study that adopted this methodology managed to eliminate head-jerking in an 11 year-old. The method teaches beat entrainment to metered music that mimics the rhythm of a relaxed heartbeat. The goal effect is for the listener to relax with entrainment. In a way, it is teaching listeners how to relax by bopping to Pink Floyd rather than Metallica.

Combining elements of Dance and Music therapy may prove to be effective in enhancing imitation and behaviour regulation, as well as emotional control and social interaction. In a more recent preliminary study conducted in 2013, sixteen adults with severe autistic syndrome participated in activities that involved drum and dance, expressive story-telling, and learned genre-specific dances such as the rhumba. Although more testing needs to be done, the results after 4 months of training have shown remarkable changes in the participants. See our review paper here for more information on the potential benefits of dance/movement therapy for autism.

Dance

In Dance and Movement Therapy (DMT), an individual is able to express internal impulses and emotions through guided and controlled movement. Through synchronous movement with others, dance has also been shown to enhance empathic understanding and social connectedness. In two studies that involved group dance and “follow the leader” activities, both autistic groups showed a decrease in stimming behaviour and aversive touch responses. There was also an increase in eye contact and the use of communicative gesture. These studies span from 1978 to 2001, suggesting that the success of Dance Movement Therapy is not just a recent finding. 

Photo source: www.fitfutures.com

Drama 

Drama has been found to be a promising avenue to promote prosocial behaviours in autistic children. In one recent study, ASD participants aged 6-17 were paired with 8 typically developing children (peers). The children participated in a musical theatre production, and peers performed the participant’s roles on video for the participants to watch and practice at home via video modelling. Children with autism showed improvement on identification of faces and theory of mind skills. 

Photo source: blog.camperoo.com/

In another study, children with ASD were provided with drama therapy in their schools. The drama therapists introduced structured ways of saying hello/goodbye, singing songs, playing ball games and drama games, storytelling, imaginary play, movement work, and relaxation. According to parent/caregiver and teacher feedback, improvements in five themes were found: feelings (a safe place to explore), peers (being included and making friends), social skills (role play provides a short cut to learning about and practicing social skills), structure (predictability lessens anxiety), and family (drama therapy supports families as well). These findings suggest that drama therapy is a suitable intervention for people with autism.

In a recent study involving children with Asperger’s syndrome, each student was given time on stage to explore different roles that involved improvisation, role-play, and movement work. The students reported greater self-confidence, self-esteem, cooperation, and emotional expression. It appears that drama therapy is a prosocial venture that could help kids with autism to flourish socially.

There are further case studies that demonstrate individual situations in which acting, song, visual art, or a combination, have helped families to make breakthroughs with their ASD children. While future research should use groups of individuals in order to be sure of the beneficial effects of arts therapies, current findings are incredibly promising.

Future Directions

In our own lab, we are currently using song, slowly scaled back to speech, in order to facilitate understanding of emotions in sentences for children with autism. We also ask the participants to mimic the emotional expressions of individuals in videos. This action may help them feel as though they were expressing the emotion themselves, and project how they feel during that type of expression while discerning the expression of the other person. So far, we have found that many of the children improve following 2 weeks of playing a video game that capitalizes on these methods. On the basis of our preliminary results, we have received a government grant to further develop the video game for tablet devices (eg., iPad). The new version of the game will be released in October, 2014. 

              

       

How does it work?

What exactly is it that makes these arts therapies so effective? There are a few different mechanisms at play here. The opportunity to become specialized in a certain activity is definitely one – there is plenty of research showing that mastery of a discipline enhances self-esteem. Even after one class, these kids can feel proud about having learned something new. In time, their talents will increase, along with their confidence.

A second mechanism involves the areas of the brain that are stimulated when people engage in artistic endeavours, especially music and dance. The mirror neuron system, which is noted to be deficient in activity during perception of other people in autism, is engaged during dance and music. It is even engaged by just watching dance or listening to music, but this is increased when actually participating. Synchronizing one’s movements with another person makes this system become especially engaged. Our theory is that when these kids engage in music or dance therapy, they are exercising the part of their brain that is important for social understanding.

Music presents itself on a multi-synchronous level due to the coordination of vocal inflection, entrainment to beat and meter, and facial manipulation in order to articulate notes. By pairing music with imitation therapy, a multi-modal system of synchrony in imitation is produced. Entrainment to the beat may help with the efficiency of movement production. In addition, since many individuals with ASD respond positively to music, this pairing seems promising in social development training

A third mechanism relates to synchronization. Spontaneous, or non-conscious mimicry of postures and positions of peers is an example of matching yourself to the surrounding social environment. Known as the chameleon effect, mimicry during social interaction creates rapport and encourages more prosocial behaviour. Kids with autism exhibit some deficits when it comes to timing, as well as binding of audio and visual information. Music and dance both rely heavily on proper timing. In many cases, learning of music and dance involves moving in synchrony with an instructor or moving in synchrony with a group of peers. Synchronous movement has been shown to assist in cooperation, liking, trust, and empathy for the person we are moving in synchrony with

Think of when you were in school and were part of a choir, or had to sing “Oh Canada” with your classmates in the morning. It’s been found that this kind of synchronous behaviour enhances camaraderie. In terms of autism, it is quite possible that practice with synchronous behaviour will help with two things:

          1.  It will help them practice moving synchronously, and to practice their general timing skills.

          2.  Through practice with synchronous movement, they may carry this skill over to their everyday life, which could benefit them in terms of enhancing emotional understanding and trust for other people.

You can participate!

The message is a positive one when it comes to arts therapies and autism. If you are interested in learning more about research going on in autism therapy, and new research applications, click here to join our mailing list. You can also participate in a study we are currently running in the SMART lab at Ryerson University. We are looking for kids aged 8-12, with high-functioning autism, as well as neurotypical kids who do not have autism for our comparison group, to take part in a video game singing study aimed at enhancing social skills and emotional understanding. We are measuring changes in brain activity, emotional responsiveness, and social behaviours that evolve as a result of the therapy. So far, we are seeing some promising benefits of synchronous singing and speaking on social deficits in autism.

Written by Lucy McGarry, Emma Bortolon-Vettor, Abby Tong, and Frank Russo

Music as social glue

In Martin Scorcese's The Wolf of Wall Street, a primitive chant whips Jordan Belfort's stock brokers into a euphoric state of solidarity. (Note: NSFW. Chanting starts at about 1:00.) The chest-beating rhythm and wordless tune (consisting of just four pitches) evoke our early ancestors singing and dancing before a hunt.


In fact, recent psychological findings are at least consistent with the idea that human music-making evolved as a social bonding mechanism. Rhythmically moving together is unique among cooperative activities in its ability to promote feelings of social affiliation and empathy. Young children are more likely to engage in helpful behaviour with someone after they have rhythmically moved together.

 Why should rhythmically moving together act as such a powerful social glue? One idea increasingly supported by neuroscience is that the act involves simulating or "playing out" another person's actions without necessarily producing them yourself. Simulating another person's actions may allow you to gain insights into their emotional state and make predictions about their future actions in a more direct way than through observation. For this reason, simulation may have had survival value for our early ancestors in deciding whether someone is friend of foe. Likewise, performing easily-simulated, predictable rhythmic movements, like clapping your hands or stomping your feet at regular intervals, may be an effective means of signalling good will towards others.

One interesting recent study showed evidence for simulation among musicians using transcranial magnetic stimulation (TMS). TMS sends electrical impulses to the scalp that can be used to selectively impair parts of the brain temporarily and harmlessly. The experimental task was to play a right hand piano part in synchrony with an audio recording of the left hand part. Prior to the experiment, one group of musicians had no exposure to the left hand part. The other group practised playing the left hand part, which, upon subsequent hearing, would presumably elicit a motor simulation of those actions required to produce it.

In the experiment, both groups played the right hand part in synchrony with an audio recording of the left hand part. The recording contained accelerations and decelerations in tempo typical of an expressive music performance. At tempo changes, TMS was delivered to the right motor cortex (which controls the left hand). Remarkably, only the group that had practiced the left-hand part beforehand were impaired in their ability to follow the tempo changes. This suggests they had been attempting to simulate the actions required to produce the audible left-hand part. More strikingly, extent of impairment was greatest for those individuals that showed a stronger tendency to adopt others' perspectives on an empathy questionnaire.

Predicting a listener's emotional response to music

Why do we listen to music? While there are several answers to this question, one common answer is that listening to music provides us with an emotionally rich experience. If music is rich with emotional content, would it be possible to predict a listener’s emotional assessment of a specific song? Additionally, to what degree is this assessment of emotion common across listeners with different musical interests and backgrounds, and to what degree is it subjective to each listener? Gaining a better understanding of this relationship between music and the listener, in the context of emotion, is an important topic in the domain of music composition and music recommendation applications. A good understanding of this relationship would enable music software applications to be customized for different types of listeners. Emotion assessments are collected either as predetermined categories (e.g. happy, sad, peaceful, angry for a music excerpt) or as ratings on a continuous scale (e.g. calm-to-excited, unpleasant-to-pleasant).

Photo Source: www.brainpickings.org (Maria Popova)

Studies have shown that it is possible to successfully predict emotion assessments using computational modelling methods. These approaches involve gathering ratings from several listeners for large datasets of music samples, and applying machine learning algorithms to train predictive models on these labeled ratings using features. The features are a combination of low-, mid-, and high-level audio features (spectral and temporal) and music features (tonal and rhythmic). Previous work in the SMART lab involved using neural networks based on audio features to successfully predict emotion ratings of listeners for a small dataset of 12 classical music excerpts. While this approach is valuable for a general listener, it has its limits with respect to each specific listener.

Every listener brings his or her own cultural background, preferences, and familiarity to the music listening experience. In some cases, your sad could be my happy. One of the ways to hone in on a listener’s subjective emotional experience is through his or her physiological responses. Studies have shown that listeners experience physiological changes during music listening which parallel the kinds of physiological changes they experience with everyday emotion. This allows us to pursue the relationship between the types of physiological responses experienced by a listener and the reported emotion induced by the music they are listening to. An interesting question to consider is whether a listener's subjective assessments of felt emotion can be predicted from physiological responses occurring during music listening. In a paper we published in Frontiers in Psychology, neural networks were used to predict emotional responses of listeners based on five physiological features. Results from this paper suggest that physiological responses are powerful indicators of a listener’s emotional state, and a non-linear relationship may be used to map musical emotion onto physiological responses.

Further studies are currently in progress between the SMART lab and WaveDNA, a music software development company in Toronto, to explore and answer interesting questions at the intersection of musical emotion, musical features, listener physiological responses, and listener experience.

Written by: Naresh Vempala

Plasticity in the deaf brain: Effects on the perception of music

When it comes to music, how is the deaf brain different than the hearing brain?

For more info see our article in Brain Sciences (2014).

A number of researchers have explored how sensory deprivation in one modality may affect the development of the remaining modalities. They have uncovered an extraordinary capacity of the brain to adapt and adjust to the environment. When one sense is unavailable, the sensory responsibilities appear to shift and the processing of the remaining modalities may become enhanced. Although investigations of brain plasticity in deaf participants have led to some discrepant findings, the deaf brain seems to be structurally and functionally different than the hearing brain.

Arla Good

         This shift in sensory responsibility, and the enhancements in visual and tactile abilities, results in a non-auditory sensory experience that is unique to individuals who are deaf. This unique sensory experience may include the manner in which music is perceived (see Figure above).

Expanding our definition of music?

Frequently cited definitions of music, such as "organized sound" or "an ordered arrangement of sounds and silences” emphasize the supremacy of sound. These definitions necessarily imply that the deaf population has limited access to emotional, social, and pleasurable aspects of music. However, a definition that focuses exclusively on sound fails to incorporate multi-modal aspects of music that extend beyond hearing. Non-auditory aspects of musical performance make music accessible to people of all hearing abilities.

How is music more than sound?

During a musical performance, the movements of musical performers, including hand gestures, body movements and facial expressions, can influence an audiences’ perceptual and aesthetic assessment of the music.

Furthermore, loud music may elicit vibrations felt on the walls, on the floor and even in/on the body. Humans have the capacity to perceive musical information, including rhythm, voice and instrumentation, through vibrotactile sources alone. These vibrations may support a unique non-auditory experience for individuals who are deaf.

In addition, music can be manipulated through assistive, multi-modal technologies, such as music visualizations and vibrating chairs, which have been created to enhance the accessibility of music in the deaf population.

So, can deaf people experience music?

A wealth of non-auditory information is available that helps to convey the structure and emotion in music to a deaf audience. The strengthened visual and tactile skills in individuals who are deaf may even provide enhanced processing of these non-auditory aspects of musical performance.

The definition of music does not have to rely solely on sound; music is capable of incorporating important visual and tactile elements that communicate structural and emotional information. This realization of music as a multi-modal experience has the potential to be advanced by the deaf community, leading to new forms of music that may transcend our current conceptualizations and ultimately to the acceptance that music is so much more than “organized sound.”

Film review: Alive Inside

An elderly woman sits before the camera, her face in close-up. A man off-screen leans over and places a pair of headphones over her head. With a press of a button, music fills the woman’s ears. The song playing is When The Saints Go Marching In by Louis Armstrong. She immediately recognizes the song, a smile spreading across her face. She moves her head along to the beat and we watch as she is transported into a distant world, a world that was lost to her until now. “Oh God, that’s beautiful,” she says as she allows herself to become absorbed into the music.

Alive Inside: A Story of Music and Memory came to life after filmmaker Michael Rossato-Bennett agreed to film social worker Dan Cohen for a day. Dan happened to discover that a surprisingly wonderful thing happened when he gave headphones and an iPod to nursing home residents with Alzheimer’s and played them their favourites songs from their past: they began accessing long-forgotten memories and regaining a sense of joy and identity. A day of filming turned into a documentary that spanned three years and followed Dan’s journey as he sought to change the way Alzheimer’s is treated in nursing homes around America.

But Alive Inside is more than just a documentary about people with Alzheimer’s unlocking memories through music. It tells the story of the nursing home, an enigmatic world that’s distant from most people’s minds. It critiques the medical model as a platform of care in these homes. It provides insights into the still largely unknown neuroscience behind the remarkable power of music. It holds a mirror to society and the way in which we treat our elders. It asks what will happen when this small cross-section of our population becomes the majority.

Poignant personal stories are interspersed with related narratives by contributors from an array of backgrounds, all executed with stunning sound and visuals that to some could appear contrived, but to me were fully complementary to the overall impact of the story. There were moments of laughter and moments of tears, both sad and happy. It laid bare the human condition and put into sharp focus everything that is so scary and beautiful about it. And there, too, was music, a natural elixir for the spirit, potent beyond words. For a while, the neuroscientific aspect of all of this didn’t really seem important; the only possible reaction was awe.

Having said that, it’s pretty neat to be involved in this important area of research, and the film puts into perspective just how much there is to uncover about the therapeutic potential of music. We are proud to be playing our own small part in the SMART Lab, where we’ve been looking at how music can be used to help better the lives of autistic children and people with Parkinson’s Disease.

The trailer for Alive Inside can be seen here.

Esztella Vezer

Why singing out of tune is not always so bad

Bob Dylan can't sing or can he?

When a singer hits the wrong note, whether at a karaoke bar or a concert, you can be sure that some portion of the audience can't help but cringe. Poor pitch accuracy is a salient indicator of an inexperienced performer. Yet there are other times when singing off pitch or bending the voice may actually improve an audience’s perception of the performance. 
In a study published last week in Frontiers in Psychology, we examined 12 professional male vocalists while they sang melodies with a range of different emotions. Vocalists with more years of acting experience sang the first note of their melody with greater pitch inaccuracy. More years of acting experience was also related to higher levels of jitter (fine-grained pitch perturbations). 

In a second experiment, listeners rated the same recordings for their emotional genuineness – whether they thought the performer was truly feeling the emotion they were singing. Vocalists with more acting experience were rated as more genuine (A), while recordings with greater pitch inaccuracy (B), more jitter, and a higher harmonics-to-noise ratio were rated as more genuine.

The level of singing training had no effect on audience perception. These results suggest that vocalists with more acting experience – but not singing training - may sacrifice pitch accuracy and certain aspects of voice quality to improve an audience’s emotional perception of the performance. So the next time you hear a performer sing out of tune, ask yourself whether they "can" or "can't" sing. Steven Livingstone and Frank RussoRead the full article at Frontiers (Open Access).

x