Blastanova

Finally! I’m at the point where I have a demo-able application. Sambaverse is the first phase of my master plan for Blastanova.

Quite some time ago, I got interested in creating music based Flash games. I created a prototype, and it worked pretty well – but it was severely lacking in one aspect. That aspect was the ability to look ahead in music, to know what’s coming up 5 seconds from now or even 5 milliseconds from now. Or to be able to look back at what happened in the music before. Or to be a little smarter about logical song breaks.

While Flash has some decent realtime audio processing capabilities, realtime just wasn’t good enough – my music games needed mystical, physic powers…to be able to see into the future and know how the musician who composed the piece thinks.

In real terms, I needed a tool that was smart enough to load an MP3, take a good stab at automatically detecting beats, break, loud sections, and different sections of the song – like verses and choruses.

So, I set out to create Sambaverse. A person using this application loads up an MP3. Right away the gears start turning – an audio waveform and song navigator is brought up in a few seconds. You can browse around, and listen to snippets. You can properly visualize your song. That’s not very special though….you can do that in any audio editor. This is why I made some music visualization modes…

using the application
At the upper right of the application is the “Playback Visualization” menu. You can choose one or multiple options here. You have the ability to view the song beats, look where the quiet periods of the song are, where the most intense portions of the song are, and find logical breaks (sections) in the music like where a chorus ends and the bridge begins.

Unfortunately, automatically finding logical breaks in the song was a little tough, and it will never be perfect. I’d describe my break detection as a good start, I have quite a ways to go. This is why I’ve added two drop down options. The first is the “custom song sections”. This mirrors the automatically detected segments, until you drag them around to place them where you want. You can fully customize and refine these segments.

Likewise, I added a song exclamation overlay. Using this mode will place little stars on the timeline. The purpose of these stars is to signify special musical events that occur very rarely. Like when that singer does that one scream in the song “YAHHHHHH!” or there’s a sudden gong sound effect…just something wacky that occurs in the song against the norm.

So you have a few view modes at your disposal. Even more than that, pressing the orange “Analyzation Settings” button will bring up the settings window. Here you can change how quiet your quiet breaks need to be or how long they are in order for them to be identified as quiet breaks.

Theres a few settings for every view mode. I’ll be tweaking these as I go along to see what works best, and probably change names and defaults.

Finally, you can save this data as XML. Actually – I’m not really saving it yet….just posting it to a webpage, and you can save it if you want. Whatever. I know I’m psyched to play around with my new app, tweak some songs, and make some cool games.

Under the Hood

I learned a lotta cool stuff making this. The biggest thing of course was how to manipulate and use audio for my bidding in Flash. I used Pixel Bender to get a huge speed increase when loading the sound initially. That was a lot of fun once I figured out how to get it working.

Although a little annoying at the time, I was using the latest releases of Flex and Swiz. As I’ve been working on this app for a little while, I had to go through and upgrade from Flex Gumbo to Beta 1 to Beta 2 to RC to Release. Stuff changed every time. Likewise, I had an annoying evening transitioning from Swiz 0.64 to 1.0.

But I’m happy I did this. Flex skinning is awesome. Swiz allowed some nice shortcuts and organization in my application. I even tapped into Flash Catalyst for the initial application design. Catalyst seemed a little limiting after the initial graphics dump. After that for quicky and dirty graphics, I just cracked open Illustrator and exported to FXG which I then copied as text into my MXML skins.

future improvements

I’m going to take a break from this application for a bit and focus on some games. I’m sure I’ll tweak as I go along and I find limitations that I need to solve as I’m working on the games. One huge feature I’d like to add when I get back to this is to be able to run a Fast-Fourier Transform on the audio and drop frequencies I don’t care about. This way, I can zero in on the drums or the bass guitar and only analyze those. In rock music especially, the frequencies are all over the place at different tempos and can make it a little hard to find a rhythm.

I’d also like to make it so you can load in a previously saved XML file to get all your work back for a particular song.

Lastly I’d like to get a sample game/animation you can preview as you work to visualize the beats better than the Flash lights I have.

So that’s Sambaverse! Check it out! (but be nice, it’s only an alpha) .

No, I didn’t accomplish the mythical Flash autotune in time for my “Audio Manipulation in Flash and Flex” presentation for NCDevCon this Sunday.  But I don’t see why it’s not possible in the least

I didn’t accomplish it, not because of Flash, but because of my lack of digital audio experience.

About a month ago, I was thinking what effect I could dazzle my attendees with, and ONE thought popped into my head:  AUTOTUNE!!!  This led me down the rabbit hole of crazy amounts of math, signal processing theory, code optimization, and more.  I even started reading a free PDF 600 page book by Stephen Smith http://www.dspguide.com/.

I’ve learned about what Fourier Transforms REALLY are, that they are really useful beyond getting frequency data, how to do low pass filters, and tons of other stuff.

I really didn’t think it would be this involved.  This is like an entire field of expertise.  I honestly thought I could steal some algorithm online and be good to go.  Nope, I’m swimming with using the FFT, then back with an iFFT, convolving signals together, and all this crazy stuff.  It’s pretty awesome though, the things you can do and learn from audio signals, and any signal in general.  I’ll definitely be finishing that 600 page book (probably 3-4 times over so I understand it).

Anyway, I was able to accomplish pitch shifting in a couple different ways, and riding a voice on top of another tone (which sounds really close to autotune if you could just get rid of that damn tone!)

I didn’t even think that one of the things that autotuning did was to detect the frequency of a sung note, and step it up or down to the correct frequency.  I thought at the beginning that when T-Pain did his thing, he just sung whatever, and the software would push the voice to whatever the producer wanted.

With Flash, I’ve seen the Audio Processing Library for Flash detect notes in a sound, and as I’ve said, I can now pitch shift!  In truth, the real Autotune by Antares Audio Technologies is said to use a “phase vocoder”, which I’m still not up on my theory enough to know what it is.  It’s probably a combination of smart pitch-shifting coupled with a flange like effect to go all robot sounding.

I finally downloaded 10.1 and got my microphone working – I recorded in from the microphone, and played back via the sound buffer (all through the sample data event).  I pitchshifted first by just speeding up the tempo so things got all high pitched and fast.  But then I grabbed a PitchShifting class by Stephan M. Bernsee  that was ported to Actionscript from C# by Arnaud Gatouillat.  Using this was VERY processor intensive.  In fact, in debug mode, my entire computer was overheating, and it was a crapshoot whether the sound would actually come out right.  And then of course the Windows sound buffer kept doing weird snapping/popping noises every so often until I restarted.  However, running as a release build seems to work just fine.  I now know why Andre Michelle’s AudioTool has a warning against using the debug player.

But all in all, it was a great learning experience.  I’m embedding the demo I’ll be showing at NCDevCon on Sunday.  And of course I’ll keep learning to become a DSP master, and someday get my Autotune working (hopefully someone beats me to it, and shares the code).

Autotune Attempt – You Need Flash 10.1 and a Microphone

It’s been a busy past month just getting a new job and a new car – I felt like I didn’t have time for anything.  Well, it just the Friday after Thanksgiving today, and I feel like I have all the time in the world this coming weekend, so I’m starting to get excited about my latest pet project.

I’ve been busy early this fall with writing “reactive music games”.  If you read my blog, which you probably don’t, you’d know that I couldn’t shut up about music games all summer.  I even did a little research into the different types.  The reactive game, is where music affects the things that happens in your game environment – but you don’t have to play along to (like Guitar Hero or Rockband).

There’s actually no popular music games I can think of that ARE reactive.  But I like to think of reactive games really…. like an interactive music video.  I even watched a bunch of Spike Jonz videos before I started sketching.  I have 10-15 games to do in the short term, and most are designed to be pretty easy to develop and generic – ESPECIALLY since this summer I’ve been working on min-framework in Flash AS3 that listens to beats in music.

The first game, I’ve done is basically an alpha, and if the music doesn’t load, just hit refresh (sometimes my security sandbox seems to go a little wacky).  I’ve codenamed it “Popfly” in my sketches, cause the balls kinda pop and fly out with the beats.

I’ve tried to pay special attention to things that I’ve noticed I didn’t like in other games.  Specifically, trying to tie visual events to music better.  Balls pop out, and when they reach their peak, its not very tied to music because it’s a split second after the beat.  This is why I put a subtle glow burst at the bottom when the balls do pop out.  I also gave the side gutters some ambient glow – just to have something additional on-screen that’s tied to the music in the game.  I also made some bigger balls pop up when the volume gets high – and it nicely adds a little crazy mosh effect when the music goes crazy.

I can’t wait to get through some more games, and see where this takes me, and hopefully way down the line, do a nice in-depth game with some characters.

But for now – here’s the PopFly game