SoundCraft Operational!

12 May 2013

I’m really excited to announce that a recent project, SoundCraft, has made its way into the New Interfaces for Musical Expression (NIME) conference! SoundCraft combines a bunch of my favorite things: hanging out with Spencer and Ge who are co-authors of the project, playing StarCraft 2, writing code, making music, and playing with new languages/systems. SoundCraft enables real-time data gathering from a StarCraft 2 game, allowing for a novel interpretation of the game. Or, as I’ve been telling people - we’re making music while playing StarCraft! Our paper, SoundCraft: Transducing StarCraft 2 is going to be published in the conference proceedings and our proposal for a performance using SoundCraft, titled GG Music, was also accepted! In a most fitting way, the conference is being held in the Mecca of eSports, South Korea!

The idea for SoundCraft originated on StarCraft 2 day at Smule in July 2012, during a chat with Tom Lieber about potentially interesting computer music projects for NIME. After some quick research on the StarCraft custom map editor, I had a map writing out the amount of minerals and gas per second for each player to an XML file. I asked Tom how he’d parse an XML file, so it should come as no shock that the XML parser is written in Ruby. With the Nokogiri gem, parsing XML was painless and with the Open Sound Control (OSC) gem, data was ready to be received by your favorite OSC-compatible language. With a ChucK patch - featuring a saw wave and pulse oscillator with frequencies mapped to a player’s mineral and vespene amounts - a proof of concept for the system was completed.

Aside from the strong urge to play StarCraft while working on a StarCraft-based project, the most difficult part of the project was creating the custom map, the biggest unknown in the entire loop. The programmer in me would have loved to just get an API, and while the map-making community seems to have pieced together Blizzard's API and found a way to load code into a map, documentation was pretty sparse. Fortunately, I was able to procure a lot of help from the map-making community over at the SC2 Mapster forums, including clutch support from Ryan Schutter whose GameHeart custom maps are used by huge tournaments like the GSL Team League and DreamHack! After plenty of pain, googling, and tweaks, we had a custom map that was writing out a ridiculous amount of information including:

• The player’s race (i.e. Terran, Protoss, or Zerg)
• Composition of the standing army and units lost
• Units and structures being produced and their current build progress
• Various metrics about economic health (e.g. mineral and vespene gas collection rate, harvesters built/lost)
• Total number of enemy units destroyed and enemy structures razed

Spencer took the lead on creating GG Music, the first live performance piece powered by SoundCraft. The map was Bel'shir Vestige, I played Protoss, Spencer played Terran, and Ge was the observer who would direct what the audience sees while we played. The world premiere of GG Music was at CCRMA's Music and Games Concert in late April. Aside from a little meta-technical difficulty with the Blizzard Updater update getting stuck, the performance went off without a hitch! Here is a video of the performance, recorded and edited by CCRMA's David Kerr:

The three main gameplay elements sonified here are:

1. Workers returning resources trigger pings generated via FM synthesis
2. Unit composition - Protoss units produce rhythmic/percussive sounds; Terran units produce uniformly rhythmic/melodic/arpeggiated sounds
3. Dying units create a jarring ping generated by passing an impulse through a low-pass which envelopes a harsh high-pitched sound

We’re excited to head to NIME to present SoundCraft, give another performance of GG Music, rendezvous with our computer music friends, eat a lot of good food, go to some professional StarCraft tournaments (we are going to the GSL finals!), and of course play more StarCraft!

Feel free to check out the source code for SoundCraft and GG Music on GitHub. If you’d like any more information on SoundCraft, feel free to send me a tweet! GG!