fbpx

Android Audio Plugin: MidiDeviceService

Categories: Software Prototypes/Non-Commercial Products

Submitted by:Atsushi Eno

10 May 2022

Elevator Pitch

Android Audio Plugin: MidiDeviceService

Import existing synth audio plugins to Android platform and use them as virtual MIDI devices

Product Description

Android Audio Plugin: MidiDeviceService

"Android Audio Plugin Framework" (AAP) project that brings in existing various desktop audio plugins to Android. "AAP MidiDeviceService" is a new applied usage of the framework, that exposes those instrument plugins as virtual MIDI devices and has MIDI 2.0 in mind from the ground. Android lacks commonly used Audio Plugin Framework. On Windows and other desktops, VSTs are popular. On Mac and iOS there is AudioUnit. On Linux LV2 is used (and VST3 recently). There is no such thing in Android. AAP is to fill this gap. As a software prototype, I have handful of LV2 and JUCE audio plugins ported to Android and successfully generated some simple MIDI notes as Android MidiDeviceServices in Android MIDI API, played by another virtual MIDI keyboard application. (It is still not really a product-ready stage as there is no way to control audio plugin parameters unless they are internally controlled by MIDI messages.) Since I am designing the entire framework from scratch, I could design what kind of content should go in and out. Support for MIDI 2.0 UMPs is also on the table (I once had it working, but still need to figure out whether it will be activated via MIDI CI interactions, or assign MIDI 2.0 plugin buffer channels etc.).

How It's Innovative

Android platform simply lacks commonly accepted audio plugin framework (mine is not yet either), so designing and implementing one comes with handful of challenges that most of existing frameworks didn't face (yet still, iOS AudioUnit and another plugin framework called AudioRoute SDK did). On mobile platforms like iOS and Android, we have to use separate app processes which makes simply impossible for host DAWs to load plugins from other plugin vendors in process. So far AAP makes use of Android binder framework and shared memory audio/MIDI buffers to achieve it (it has to be non-realtime yet). If you are a DAW vendor, you would most likely give up and provide closed in-app plugins. One thing that I believe none of other frameworks tries (and thus what's really "innovative" in AAP) is how we achieve plugin extension API like VST-MA, or those in LV2 and CLAP - It is fundamental to plugin frameworks like them as well as AAP to achieve framework stability. AAP is especially tricky because we cannot simply provide in-process extensibility e.g. we cannot simply invoke callback function pointers in the plugin or host process mutually. In business kind of aspect, adopting a new audio plugin framework is almost impossible for DAW and plugin developers, especially when the plugin framework is still a moving target, AAP comes with the idea that production stability would come with stable development tools and existing APIs, such as JUCE and LV2 (I believe CLAP audio plugin takes similar approach). Beyond that, exposing the sound as a virtual MIDI device is most likely stable. That's why I went forward and implemented MIDI device service support. Even if future AAP becomes incompatible at binary messaging level, the existing MidiDeviceServices would still work. (The idea on using VST plugins as virtual MIDI device happened on Windows before, but I'm using the idea to make audio plugin framework adopted.)

Most Inspiring Use Cases

I already have ported some popular JUCE plugins such as OB-Xd, dexed, Odin2, and Vital OSS, as well as couple of LV2 plugins such as sfizz, Guitarix, and mda-lv2 (note that they exist only as standalone GUI apps and AAP audio plugins without GUI integration). I believe that porting a bunch of more audio plugins and exposing them as MIDI devices is going to be fun. All those existing ports are published on GitHub too. My screenshot image would give you the idea - it is a screenshot of another MIDI keyboard application (that I wrote for my Kotlin library for MIDI 1.0 / 2.0). It is another standalone app (no AAP involved) and I could successfully load those plugins as virtual MIDI devices.

Expansion Plans

AAP as a framework still has a lot of milestone tasks to do. I'm focusing on brushing up framework core parts, but many pieces such as support for complicated audio bus, more comprehensive MIDI-CI and UMP support, presets and midnams etc. - they are all logged as GitHub issues. One big missing piece is GUI integration support, which had been drafted for years and still not worked on yet: showing native plugin GUI on DAW (i.e. another application) is simply impossible, so we have to seek for other approaches to support it. The most realistic idea for now is WebView UI for each plugin i.e. GUI is implemented in HTML/JS, and there are some frameworks that support Web UI like iPlug2, or juce_emscripten, so there would be some foundation technology. I would rather focus on how plugin foundation (parameters etc.) and UI should get connected.

Commercialization

One fundamental principle in AAP is that it aims to be a liberal framework that no one has to pay for AAP even for commercial uses. So, commercialization on the framework, including MIDI device support, is not going to happen, as long as I am the primary developer. But it is an essential question that I haven't figured out how to answer yet, as I don't work for any company for now and I should either work at some company or launch a business to get income, at some stage. It is important for the project on sustainability. It is technically possible to release JUCE integration under a dual license with GPL and commectial just like JUCE, but I would rather find that potential market like those vendors would need myself to help porting, which would result in a lot more income than just licensing (which I don't find I can live just on it anyways). That is not really a good business idea, as I want to make things as simple as possible so that people don't need me to adopt it. To be honest I had been refrained from announcing the project a lot, because plugin framework like this has to provide stability and AAP is not quite stable yet. I'm changing my mind on that because stability can be achieved through the public API like this MIDI device service, which is rather a recent addition to the framework. This entry reflects the idea of making it more public and reachable. Potential wise, there have been couple of inquiries on AAP from commercial and non-commercial Android audio and music app developers that want to expand their capability. I would be able to provide better pluggability for those folks at some stage. But if I do it now, that would involve too tight integration into their apps, and would result in annoying individual plugin app distribution that would kill openness of AAP. Inquiries and seeks for potential use of the plugin framework is still highly welcomed, of course.