Minecraft & MIDI Connections
Since late 2020, I’ve been using Minecraft as an immersive musical instrument. However, my original project, void loop () only made use of in-game sounds. Custom sounds had to be added via a resource pack, and the use of custom samples was limited. In May 2022, I began looking for ways to expand the expressivity of the game. Through the use of the Mineflayer Javascript library, I’ve managed to make robust data connections in and out of the game. In-game chat can trigger MIDI events, report entity locations, and report basically any other game mechanics. Two open source projects are available for the nerdy/savvy few who may want to deploy this. mcMIDI.js - which creates a general MIDI connection between Minecraft and the MIDI device of your choice. Currently, this offers one-way communication from Minecraft. mcM4L.js - which makes a more powerful connection between Minecraft and Ableton Live. This application offers two-way communications. Minecraft can trigger MIDI notes & MIDI CCs, tap into Ableton's Max for Live API, and connect to anything Ableton can. Ableton can thusly send messages back into the world to trigger world events. Numerous performances (my solo projects and many collaborations with my students at UMass Lowell) have already included these communication methods.
Given the world's most expressive video game and one of the most hackable Digital Audio Workstations, it seemed a travesty that the two had not been connected. The data connections exploited in my projects offer an infinite set of artistic and expressive options for working with music and visuals. The backstory outlining various innovations and developments: My work on these projects began in summer of 2020, when we received word that UMass Lowell would be fully remote for the 2020-2021 school year. I decided that my electronic ensemble would be required to actually perform via Twitch. I learned a modicum of javascript and C++ to create some applications and Arduino-compatible devices for better interactions between the Twitch audience and the performers. While not many students in the group adopted my methods, I did manage to develop a live performance (October 31, 2020) collaboration between one student (controlling cameras and visuals remotely) and myself (playing guitar/live looping). In November 2020, I witnessed my daughter playing Minecraft online with one of her friends. Their in-game interactions seemed to have less lag than what most of us were accustomed to on Zoom. It felt like something my ensemble could exploit - creating virtual spaces for performing instead of the floating boxes videos that became so popular. I spent a little time exploring Minecraft myself in December 2020, and quickly decided to include it as an option for students in the electronic ensemble. In January 2021, half of my ensemble was signed up to develop collaborative Minecraft projects. By April 1, 2023, my programming chops enabled me to implement a Twitch chat bot that could relay various commands to control my Minecraft character. I performed at UMass Lowell's Recital Hour entirely in Minecraft in a series of pieces called void loop (). Most of my in-game sounds were custom samples in a resource pack, and the audio was piped through Ableton Live for live looping and other antics. Work on my methods halted until May 2022 when I began developing a 24/7 Twitch stream with Minecraft/musical installations. The stream has been visited by literally dozens of viewers; it offers them a chance to control game elements (solve mazes, walk through an ambient landscape, fly through the world) while musical elements adapt to audience decisions. The work I did on this installation prompted me to revisit some of the 2021 projects, this time with the goal of achieving enormous expressivity. In July 2022, I presented "Minecraft as my Musical Instrument" at the HOPE (Hackers on Planet Earth) conference. I received much constructive feedback and was pointed in so many great directions for further developing these ideas. From August 2022 through today, I have been building on to my methods, creating a huge library of game/audio mechanics. The developments outlined above prompted me to propose an official Video Game Ensemble at UMass Lowell. It was met with nearly universal approval, and we have just completed our first year in existence. In addition to frequent Minecraft projects, my students and I have worked with games like Super Mario Maker 2, Beat Saber, OSU!, Mario Kart Live: Home Circuit, and Wii Tennis as live performative instruments. While not many games offer the chance to exploit data connections like Minecraft, I'm excited to see what other games might provide similar musical connectivity.
