PhotoSynth > InterFACE

Categories: Software Prototypes/Non-Commercial Products

Submitted by:Zenon Olenski

24 May 2023

Elevator Pitch

PhotoSynth > InterFACE

Instant Musical Superpowers! The smile powered synthesizer - an augmented reality musical instrument played completely with your face!

Product Description

PhotoSynth > InterFACE

An extended-reality musical synthesizer - completely controlled by the face! Sometimes referred to as the Smile Powered Synthesizer =D

By piggybacking onto a person's own natural movements using motion-tracking in real-time, we reduce the complexity required to learn a new instrument to mere moments whilst still retaining full expression and precise control.

Simply open your mouth to begin - it is a lot like singing. The head becomes the primary interface and the mouth the main control - we can convert smiles into musical notes and winks into audio controls. By observing the features on the face over time, we know at any point the angle and orientation of the face as well as basic facial emotions and moods, and we can use these in order to trigger and modify sounds.

This virtual interface has no moving parts, requires no extra hardware, and can run on most modern computers and mobile phones. We can capitalise on a person's own expressions and movements to create unique personalised synthesized music but thanks to MIDI we can also control and manipulate almost any electronic musical instrument too.

Simply connect your MIDI instrument to instantly be able to play it with your head! Without any effort, the face is now an accessible monophonic MIDI controller, whilst with a friend in duet mode you can have polyphony too!

How It's Innovative

This is a zero-touch interface that piggy backs onto your natural movements to imbue them with new properties and skills. By recognising universally familiar movements, such as smiling and nodding, along with reading simple human emotions, we can form the basis of a new type of smart interface that can reduce the complexity of operation whilst still retaining full expression and control. Without any extra equipment or gear, your computer and studio can now be completely and simply operated by your face. The brief was simple but the end product requires much complexity in order to offer that simplicity :

● Requires no touch to be played
● Be suitable for all ages
● Have little to no learning curve
● Be accessible, inclusive and collaborative
● Be engaging, fun and a little bit magical

The scope of this project’s ability falls far beyond a musical instrument. This is as much a research project into perfecting natural-instinct interfaces for the future of human-machine interaction. Current state of the art motion capture devices are complicated, expensive and bespoke pieces of equipment - we intend to remove the need for any specialised equipment entirely and remove the cost so as to completely democratise accessible interface technology.

See MIDI Innovation In Action

Most Inspiring Use Cases

This is a collaborative project with a number of disabled artists and charities including accessible technology specialists, Drake Music Charity, who have been providing feedback and essential testing.

Ultimately it has dreams of becoming the most inclusive musical instrument and set a precedent for how interactions with other devices could be simplified in the future. Without any training or instructions this technology empowers anybody with a musical ability. We can use a person's own expressions and movements to control and manipulate hardware, software and data. Music is a great form of expression but also one of precision and depth - if this software is successful in it's endeavor it will be simple to then refine the technology to control practically anything. It will feature as an installation in the Accessible Musical Instrument Museum which is the world's first accessible technology site dedicated to representing and demonstrating musical equipment that anybody can play, but thanks to the internet this software can also be in every home and school and as it works on mobile devices, this can be used by anyone anywhere too. This technology has the capacity to provide people new capabilities and opportunities. It aims to bring more equality to skills and leverages technology to empower and include the most diverse and varied audience - giving confidence to the most underrepresented and unprivileged.

Expansion Plans

● Expand from duet to quartet - allowing up to 4 people to play simultaneously!
● Saving and loading of sessions as MIDI and real-time online jam collaborations.
● Emotion detection and recognition of facial expressions - play minor keys with a frown - major keys with a smile - add aggression with an angry glare... There are a lot of ways to communicate with a computer without saying or touching anything.
● Magic mapping to be added to allow MIDI commands to be assigned to any specific gesture or interaction by teaching the software by physical demonstration. ● Work with other labels, musicians and hardware manufacturers with the ultimate goal of having a wide library of sounds.
● Extra expression / input from other body parts including the hands and even the full skeleton. A movement could be mapped to control a signal, such as pitch bend / after touch or be used as an input method in itself : Set beats with fingers, set rhythms with dances, there are lots of interesting opportunities to explore. A hand in the shape of a mouth could easily allow one person to control up to 3 voices themselves.
● Collaborations with other musicians, artists and coders to expand the quantity and range of sounds and musicality.
● App for mobiles and version for installations with extra features and automation.


● Pro version with new sounds and features including saving session and exporting MIDI
● Sample sets from hardware manufacturers such as Korg and Roland to demonstrate the sounds of their instruments.
● Sounds from artists who want to offer their users a new way to interact with their music.
● Mobile App that would be available through the app store - this would offer the pro feature set and extra sounds with in-app purchasing to expand the library and capabilities.
● For use in schools, establishments and public places, varying licenses will be available including as ever, a free tier