Categories: Software Prototypes/Non-Commercial Products

Submitted by:Markus Ruh

24 May 2023

Elevator Pitch


MusiKraken is a modular MIDI construction kit for mobile devices. You can combine camera, touch, motion sensors and microphone input with effects to make music.

Product Description


MusiKraken is an experimental MIDI controller construction kit.

It runs on iOS and Android and is a completely modular system that allows you to combine the values of various sensors built into mobile devices with many MIDI effects. The MIDI events can be sent to other devices via WiFi, Bluetooth and USB cable, or to other apps, and it can host Audio Unit instruments on iOS.

The editor allows you to combine all modules by connecting the available ports. Orange MIDI ports send and receive MIDI events like Note-On and Note-Off, while green value ports send the values of the sensors.

Using the Value-to-MIDI module, you can convert any values to MIDI control change, note, pitch bend and channel pressure events. This way you are completely free in what your MIDI controller setup should be able to do. MusiKraken currently supports the following input sensors:

Touch: The Keyboard module is a typical musical keyboard, but it also allows you to control additional values while playing, like sliding on the keys and using touch pressure or touch radius (if supported). It supports MPE, has two different layouts, and lets you highlight one of 95 scales.

The Chords Pad has a button for each chord in a scale, and vertically has one for each inversion of these chords. While playing these chords, you can also control additional parameters by sliding or changing the touch pressure or radius. And the Touch Pad allows you to control up to five parameters simultaneously by moving your fingers on the screen.

Camera: Use the camera to track face-, hand- and body movements. Face tracking has parameters like head rotation and mouth opening amount. The hand tracking tracks the position of both hands and various hand gestures. And the body pose tracking detects the position of your head, hands and feet on the camera image. And on iOS devices with a TrueDepth sensor, the app can even use the distance to the hand (or other objects) as an input.

Motion Sensors: MusiKraken has a separate Accelerometer module to measure accelerations and a Motion Sensor module which combines the values of the Accelerometer, Gyroscope and Magnetometer to get the current device rotation.

Microphone: This module uses the microphone of the device to detect pitch and amplitude.

The effects in the app allow you do the following:
  • The Channel Switcher lets you route MIDI events to specific MIDI channels.
  • The Chord Splitter can split chords into separate notes and send each to a different channel to control multiple instruments simultaneously.
  • One effect let you transpose the notes and the Arpeggiator can arpeggiate through them.
  • The Beat effect lets you simulate percussion instruments with all input sensors, and the Threshold effect plays notes if a value goes over a specific threshold.
  • The LFO generates an LFO output value that can be controlled by other input values.
  • And the Envelope module generates an ADSR output value based on note input. As all these modules are independent and can be combined, this allows you to build your MIDI controller in any way you want.

How It's Innovative

You can adapt MusiKraken to whatever you are controlling with it. You do not need to adapt your music making style to the MIDI controller, you can instead change the MIDI controller until it fits you and the instrument that you are playing with it.

Because the setup in the app is completely modular, you can make combinations that were never before possible in a MIDI controller. MusiKraken also provides some input methods that do not exist in any other MIDI controller and makes it possible to combine multiple input methods to create new ones.

See MIDI Innovation In Action

Most Inspiring Use Cases

With MusiKraken, you can be very creative when combining the different modules. One way to use it is to ask yourself how your ideal MIDI controller would look like for controlling a specific virtual instrument: Would it make sense to play the notes with a touch interface, or would it feel more natural when using the motion sensors? Would it make sense to change the expression of the instrument by tracking my mouth or head movements?

The answer to these questions is usually very different if you are controlling a percussion instrument, a virtual string instrument, brass or woodwinds, or a synth. Or you can be completely open to new combinations and invent your own experimental MIDI controller. All modules that are currently included in MusiKraken can be used simultaneously, so the number of possible combinations are nearly unlimited.

Expansion Plans

Since its release, I have constantly added new modules to the app to increase its possibilities. And my TODO list is still growing. I usually work on multiple modules more or less simultaneously and finish the ones that make the most sense at the moment. So I have many half-finished features that might be released one day.

Currently I am adding support for external devices that can connect to the mobile devices and can be used as input: Apple Watch, Game Controllers, general Bluetooth devices and so on. Next I plan to add music theoretical modules like an experimental chord progression editor or an advanced pattern arpeggiator.

And another planned expansion path is a bit special: MusiKraken was created using the game engine Unity. The reason for this was that this way I can create as much as possible in a platform-independent way and combine it with native code to support all sensors of a device (and I already knew the disadvantages of it when planning the app).

But using Unity has two other advantages:

First, all platform-independent code also runs on other platforms. The app for example already would work on Windows and Mac.

And second, because of Unity there is a full 3D engine and a physics engine already in the app.

Both of which I haven't used yet. Controlling 3D graphics with MIDI and generating MIDI events using a physics engine will definitely increase the experimental possibilities of the app.

And another plan that I already have a prototype for: By creating a small connector software for Windows and Mac, I can not only make creating connections easier, it also opens up a lot of other possibilities: I can for example already control the mouse cursor with MIDI, or send commands to programs on the computer. As you can see, I still have a lot to do...


You can already buy MusiKraken on the Apple App Store and Google Play Store.