Synaesthesia has a very practical use for one Dutch engineer.
We often think of sight and sound as two separate sensations. Though we perceive them simultaneously, it seems strange to try to relate these two important features of how we perceive the natural world. But that’s exactly what Dutch engineer Peter Meijer did in 1992 with vOICe, a groundbreaking system that converts shapes and patterns into musical notes and pitches.
This unique adaptation for the blind has now found the perfect home on Android with an additional system called EyeMusic. Though the system is at its most helpful when used with a head-mounted camera, allowing subjects to navigate around a room on sound cues alone, the phone-based system holds its own possibilities.
The visual language of vOICe sounds at first like a garbled mess, but subjects working with neuroscientist Amir Amedi and his colleagues at the Hebrew University of Jerusalem were able to use the system to spot a human figure with about 70 hours of training. The choices in creating the language sometimes seem arbitrary, but it also seems to play on all the shared words we have for musical and visual terms. For example, the pitches of vOICe’s tones get higher as the shapes they describe become higher in an image. Brightness, on the other hand, is signified through volume. EyeMusic elaborates on this system by translating colors into different musical instruments.
The app also has a talking color identifier, compass, face detector and GPS, as well as CamFind visual search and Google Goggles support. Its head-mounted functionality could bring Google Glass to the blind.
With 4500 pixels of resolution, the system has room for nuance and allows subjects to pick up on facial expressions and postures. To the delight of researchers, investigating the world in this way activated the visual cortexes of subjects – a testament to the adaptability of the human brain.
A self-proclaimed “cyborg,” Neil Harbisson was born with a rare condition that enables him to only see the world in black and white. However, an “eyeborg” device helps him “see” colors by translating the colors into sound. By turning the light frequencies into sound frequencies, Harbisson is able to perceive a full spectrum of color.