Play Music With Eye Movements and Facial Gestures

Play Music With Eye Movements and Facial Gestures

Musical interface enables people with physical disabilities to play any instrument

Emma Hutchings
  • 9 march 2016

Eye Conductor is a musical interface built by Andreas Refsgaard as a final project at the Copenhagen Institute of Interaction Design. It enables those with physical disabilities to play music using their eye movements and facial gestures. They are given the opportunity to play any instrument, build beats, sequence melodies or trigger musical effects.


Refsgaard carried out user research for his project, visiting several schools and communities for people with physical disabilities. While the people he met had various levels of physical ability, he found that music was a unifying interest for all of them.

A lot of people with physical disabilities lack the fine motor skills needed to produce music on traditional instruments. Refsgaard wanted to push the boundaries of interaction design by exploring how eye and face-tracking technologies could be used for creative purposes. He set out to create a solution that operated in the same domain as traditional instruments, giving people freedom but requiring them to practice.


Eye Conductor uses a $99 eye tracker and a regular webcam to detect a person’s gaze and selected facial movements. It translates eye gaze into musical notes or beats in a drum sequencer. Different movements can signal the adjustment of different musical effects, for example, raising your eyebrows can be used to transpose all played notes up one full octave, while opening your mouth can add a delay, reverb or filter effect to the instrument. The system is open and designed for inclusion with customization options, so thresholds for facial gestures can be adjusted to support the unique abilities of each user.

After a recent visit to a school in Denmark for youngsters with a range of physical limitations, Refsgaard says he realized that he should build a system that works across all platforms and sensor inputs with little hassle. He now plans to rewrite all the code in JavaScript and release a browser-based version of Eye Conductor later this year. You can learn more about this project in the video below:

Andreas Refsgaard

+Andreas Refsgaard
+Copenhagen Institute of Interaction Design
+Interaction Design
+Market Research
+musical interface

Capsule Is Reimagining The Pharmacy As A Patient-First Experience

Brand Development june 23, 2017
Gaming & Play june 23, 2017
No search results found.