Blastanova

March 24, 2010

Pixel Bender Roundup

Filed under: Uncategorized — admin @ 11:46 pm

Hey everyone,

I just gave a Pixel Bender presentation at our local Adobe User Group yesterday.  Here’s a roundup of the blog posts I wrote as I was collecting my thoughts:

First I wrote about the problems I wanted to solve, and why Pixel Bender seemed like the right choice:

http://www.blastanova.com/blog/2010/03/19/pixel-bending-for-speed-in-flash-and-flex/

Next, an overview and intro of Pixel Bender:

http://www.blastanova.com/blog/2010/03/24/intro-to-pixel-bender/

And then, a post about using Pixel Bender for data instead of images:

http://www.blastanova.com/blog/2010/03/24/using-pixel-bender-to-process-lots-of-data/

Finally, some speed tests along with a sample application I wrote to run the speed tests:

http://www.blastanova.com/blog/2010/03/24/pixel-bender-speed-tests/

Happy Pixel Bending!

Pixel Bender Speed Tests

Filed under: Uncategorized — admin @ 11:38 pm

This is my last in a series of blog posts on Pixel Bender.  I’ll probably be doing one more PxB post when I upgrade to Swiz 1.0 and we’re talking about PxB processors with Swiz, but until then….

This post will be focusing mostly on speed in Pixel Bender and doing similar operations in normal Actionscript.

Pixel Bender is good on speed, pixel by pixel, but there are some operations that make PxB a little less efficient.  In my tests – these efficiency deficiencies are made worse with specific operations.

The worst is when you sample the nearest pixel.  In PxB, sampling one pixel is fine.  But the more pixels you sample as you evaluate one pixel, the worse this problem is exasperated.

Why sample more than one pixel?  Well, a blur effect is a great example.  When you blur, you’re probably averaging the center pixel and the 8 surrounding pixels.  Or downsampling an image…you’d be sampling a whole mess of surrounding pixels and reducing to 1 pixel.  In my audio visualization application, I’m not reasonably going to view 2.5 million pixels.  No, I’m going to reduce down to something that can be displayed – maybe a few thousand, depending on how far I zoom.

In any of these scenarios, I’m sampling more than one pixel.  The more pixels I sample, the worse the performance gets.  In my experiments, it can get to the point where PxB is no better than just doing things in Actionscript.  In these cases, at least Pixel Bender has the whole multi-threaded feature going for it, so your UI performance won’t suffer.

Another horrible aspect of sampling more than one pixel is that PxB for Flash doesn’t support loops and functions.  So, if your kernel needs to sample 100 neighboring pixels, you have to write the 100 lines of code to go with it.

For audio processing, sampling surrounding pixels is the whole point.  The key is finding the right amount of pixels to sample where you get a good average, but still have your Pixel Bender shader running lean and mean.

With that said here’s a sample application which demos performing math operations in Pixel Bender vs performing the same operation in Actionscript 3.

Pixel Bender Speed Test Application

Keep in mind that we keep all the loading of the MP3, and the extraction of the byte array at the very start of the application.  These actually take a fairly long time (almost a full second to extract the audio), but are a necessary evil whether you use Pixel Bender or you don’t use Pixel Bender.  Lets look at the results:

Simple Copy Operation

Take data, and return the same data (but make sure to copy each and every value to a byte array one at a time)

Flash AS3: 1.715 seconds

PxB: 0.475 seconds

Power of 2 Operation (Square a Value)

Take each value, square it, and copy into a new byte array

Flash AS3: 2.81 seconds

PxB: 0.825 seconds

Multiply by 12.54 (a random number I picked for this test case)

Multiply each number by a value and copy to a new byte array

Flash AS3: 1.747 seconds

PxB: 0.495 seconds

Do a Complex Math Operation

Take each value, multiply by 12.54, divide by 100, take the sin of that, and then take the square root of the result.

Flash As3: 3.53 seconds

PxB: 0.713 seconds

Average 10, 20, 50, and 100 values

This is where Pixel Bender should start getting less effective, as we are sampling more and more surrounding pixels.

Average 10 – Flash AS3: 3.727 seconds

Average 10 – PxB: 0.878 seconds

Average 20 – Flash AS3: 3.954 seconds

Average 20 – PxB: 1.153 seconds

Average 50 – Flash AS3: 3.613 seconds

Average 50 – PxB: 2.047 seconds

Average 100 – Flash AS3: 3.636 seconds

Average 100 – PxB: 3.469 seconds

As you can see, Pixel Bender does VERY well until you start sampling more and more neighbors.  As we reach 100 neighboring pixels sampled,  there really is no speed advantage to Pixel Bender.  Fortunately though, we’re still talking about an operation on a different thread that won’t slow down the UI.  So even if it does take 3 and a half seconds, its 3 and a half seconds of time that our UI is as snappy as ever.

Remember before when I talked about extracting your audio?  Well, you don’t have to grab 60 seconds all at once and lock up your UI for a full second.  You could grab less, and process the audio in increments.  With PxB, you can just keep running shader job after shader job.  You’ll have to determine what’s best for your UI vs how fast you want the operation to complete.

Another thing that I’ve hidden (though I didn’t mean to)  in this demo application is the time it takes to create the Shader.  I’ve found that it takes around one second to create a Shader.  This means that before you utilize a Pixel Bender kernel, you must take 1 second of unresponsive UI time to have a shader be created.  Creating a ShaderJob, on the other hand takes about a millisecond.  This means that you could spend 1 additional second at the beginning of your application to initialize the shader.  That’s one second of time that you didn’t have to spend running your AS3 only code.  However, the more ShaderJobs you have to split up the work, the more this 1 second becomes a moot point.

An additional note of interest is the shaderJob.start method.  By default, the parameter you pass in here is “false”.  This boolean indicates whether to “waitForCompletion”.  If you don’t wait for completion, then you are using this event asynchronously – meaning you assign an event complete listener and wait for the operation to complete while your code does other things.  If you specify the “waitForCompletion” flag as true, then all Flash operations will halt until  your shader job finishes, and you don’t have to set an event complete listener because the next line of your AS3 code will just execute after the job completes.  I’ve read that this can give you a slight speed increase, but I haven’t had this experience.  Plus it will lock up your Flash code execution until it completes, meaning your UI will be unresponsive.

And that completes my series of posts on Pixel Bender for now.  Kevin Goldsmith at Adobe has been doing some pretty cool stuff as well.  He does audio processing live, or rather each time Flash reaches into the audio buffer to get the next sound samples.  So he actually uses Pixel Bender to alter audio AS YOU PLAY IT.  So that’s very cool, check it out here.

There is much more to this Pixel Bender stuff.  Find it online, and especially read up on the Pixel Bender Twiiter account.  For now though, I’m done.  Happy Bending!

Using Pixel Bender to Process Lots of Data

Filed under: Uncategorized — Tags: , , , , , — admin @ 11:38 pm

In my previous post, I talked about using Pixel Bender in the way it was intended.  By this, I mean to input an image, change the pixels in some way, and output an image.  As you might be aware, this can happen very quickly and be used to create some great run time visual effects in Flash and Flex.

What piqued MY interest in PxB though, was the fact that the you didn’t have to send it an image to process.  It could be any ByteArray object.  My interest was a little bit more specific – a ByteArray extracted from a flash.media.Sound object.  At 44,100 samples per second, with typical MP3 files being 180-200 seconds, we’re talking about pushing 10 million samples.

My project involves visualizing the waveform of a song, so I want to process all of these samples at once, and as quickly as I can.

So, lets look at how you would normally apply a PxB shader.  The following example is taken from Mike Chamber’s blog:

import flash.filters.*;
import flash.utils.ByteArray;		 		
 
//the file that contains the binary bytes of the PixelBender filter
[Embed("testfilter.pbj", mimeType="application/octet-stream")]
private var TestFilter:Class;		 	
 
private function onApplicationComplete():void {
//Pass the loaded filter to the Shader as a ByteArray
var shader:Shader = new Shader(new TestFilter() as ByteArray);
shader.data.amount.value = [100];
var filter:ShaderFilter = new ShaderFilter(shader); 	
 
//add the filter to the image
im.filters = [filter];
}

So, Mike’s got the right idea here. He creates a shader, makes a shader filter out of it, and then applies it as a filter to the image.

We’re going to be doing something slightly different. Instead of a ShaderFilter, we’ll be creating a ShaderJob. And we’ll also need to tell our ShaderJob certain things that our ShaderFilter knew automatically. Specifically, we’re talking about the height, width, and input source of the image. After setting these properties you add an event listener to the job, and tell it to start.

var output:ByteArray = new ByteArray();
data.position = 0;
 
var channels:int = 4;
var width:int = 2000;
var height:int = data.length / width / (channels*channels);
 
shader.data.src.width = width;
shader.data.src.height = height;
shader.data.src.input = data;
 
var shaderJob:ShaderJob = new ShaderJob(shader, output, width, height);
shaderJob.addEventListener(Event.COMPLETE, onComplete, false, 0, true);
shaderJob.start();

OK, I’ll back up…it’s a lot to start with. Let’s talk about the input data first – our byte array.  Because I’m talking specifically here about a byte array extracted from sound, I know that my byte array is a long list of 4 byte floating point numbers.  This particular list of numbers alternate from left channel to right channel, but that’s neither here nor there until we get into the Pixel Bender kernel.

So, now we’re talking the language of Pixel Bender – a byte array of 4 byte floats.  Now, we’re typically we’re dealing with 4 of these 4 byte floats (formerly known as red, green, blue, and alpha).  I believe that PxB is SUPPOSED to accept image1, image2, image3, or image4 type inputs, however I’ve read there are bugs associated with doing anything less than image3 as the input.  Even if we wanted to take advantage of an image2 – think about it.  PxB is designed to process a pixel as fast as it can,  Why not take advantage and create the largest pixel we can?  Little decisions like this multiply in big ways when you start talking about your entire data set, and you could be talking about saving hundreds of milliseconds for these little decisions.

So, each pixel is 16 bytes.  In my sample code, I’m calling out 4 “channels”.  This is an important variable when thinking about our height and width.

Let’s talk a little about height and width now.  As you know, an image has a height and width.  But how does this apply to a one dimensional list of items for PxB?

It all comes down to speed and optimization.  You could certainly give your data input a height of 1 and a width of 8192 (the height and width limit) and treat your data as a one-dimensional list.  It’s certainly easier to picture that way, but then you wouldn’t be taking advantage of PxB’s built in speed and optimization.

Pixel Bender is multi-threaded, multi-core, and GPU enabled.  While that last part doesn’t hold true for Flash, the first two parts are important.  Not only can PxB itself operate on a different thread and CPU core, so can each row of data.  So, PxB can be processing many rows of pixels all at the same time.  If you only do one row per job (a height of 1), you’re throwing away most of the speed benefits of Pixel Bender!  The data itself will wrap around to the next row, just like a text field wraps text to the next row.

How does you know what width and height to give your data?  Well, with a possible width and height of 8,192, you’re giving yourself a possible 67 million pixels to work with in one job.  Each one of these pixels holds 4 channels.  Which means, in one job you can process 268 million floating point numbers.

Earlier I had talked about each MP3 file being around 10 million samples.  I’m personally conducting my own speed tests, but I’ve found that processing 60 seconds of an MP3 file at a time is a little better overall for speed.  So we’re only talking about 2.5 million samples that I would personally process at a time.  So, now the question becomes: How can we reasonably distribute our data as an image.  Well, if we max out our width, we may not be producing many rows.  As  a result, speed may be reduced because we’re not utilizing multiple threads as efficiently as we could.  Based on my input data, I’ve found that a width of 2000 works well for me based on the specifics of my application.  Your mileage may vary, so feel free to try different things.  Just remember, you’ll want to take advantage of many rows for PxB optimization.

And then of course, to get the height, you can work that out by calculating this from your width and the overall length of your byte array.

After all has been said and done, you start the job, set the event listener and wait for the job to complete.

Next up, I’ll be posting some speed tests!

Intro to Pixel Bender

Filed under: flash/flex,music video games — Tags: , , , — admin @ 11:37 pm

So this is my intro to using Pixel Bender – if you don’t know what it is or why to use it check out the documentation or my first post.

Actually the documentation is a good place to start.  Go to http://www.adobe.com/devnet/pixelbender/ before you begin.  Go on….I’ll wait…

Once there you’ll want to download the Pixel Bender Toolkit.  It’s a simple and light program – not subject to the lengthy installs of other Adobe software.  Don’t bother downloading the PDF documentation, there are links in the help menu to these documents once you run the toolkit.

The best part of the documentation – which actually turns into the most depressing part once you learn it, is that you ignore around half of it if you’re developing Flash shaders.  Most of the more advanced functionality only applies to Photoshop and After Effects shaders.

So crack open the toolkit!  What to do now?  Well first, go to the file menu and choose “new kernel”.  Kernels are basically the “programs” you’re creating that compile to shaders.  A new kernel will look like this:

kernel NewFilter
<   namespace : "Your Namespace";     vendor : "Your Vendor";     version : 1;     description : "your description"; >
{
    input image4 src;
    output pixel4 dst;
 
    void
    evaluatePixel()
    {
        dst = sampleNearest(src,outCoord());
    }
}

Don’t worry about that top part – it’s just noting the author of the script.

So the first thing to worry about are the two variables at the top. There’s “image4 src” and “pixel4 dst”. You might guess that you’re defining a source image and a destination image. But what’s up with the funny syntax?

Well, first of all, Pixel Bender is one of those languages where the data type is in front of the variable. So you have a variable “src” of type image4, and a variable “dst” of type pixel4. Image and pixel datatypes might make sense, but the “4″ is what threw me off at first, but don’t worry it makes sense.

PxB mainly deals in floating point numbers. And no automatic type conversion! Doing float var = 2 is no good, but doing float var = 2.0 is OK. There are basically 4 types of floating point numbers: float, float2, float3, and float4. A float4 is basically an array of 4 floating point numbers. Example: float4 myfloat = float4(2.0, 2.0, 2.0, 2.0);

It only goes up to 4. Once you realize that the main point of PxB is to manipulate pixels, you being to see why. Red + Blue + Green + Alpha = 4 channels and an array of 4 floating point numbers.

I’ve found that I can use pixels and floats interchangeably (maybe I’m wrong). Images are reserved for an entire image comprised of pixel 4′s/float 4′s. Pixel Bender also supports integers and booleans (each with 4 or less values).

OK that was my rant on variables. Lets move onto “evaluatePixels”. This is THE method that everything PxB revolves around. In fact, in Flash, you can’t even create other methods in your kernel (PS and AE allow this though).

Every PxB kernel is designed to do one thing and one thing only. Take in a source image, go pixel by pixel, and create an output image pixel by pixel from the “evaluatePixels” method.

That’s easy enough to understand – but what about that wonky syntax they start you off with?

dst = sampleNearest(src,outCoord());

So, lets work from the inner to the outer. Starting with outCoord(). This method gets the current coordinates that pixel bender is analyzing at that moment. Hint: it’s a float2, containing both X and Y values.

That was the easy part – the hard part is “sampleNearest()”. For us Flash folks, this introduces you to the wholly confusing notion of sampling on half pixels and pixel ratios and other such nonsense. But then you realize that its Flash, and all pixels are square, and sampling a pixel samples the entirety of the pixel. At this point you realize that sampleNearest is just how PxB works – but it’s entirely unnecessary for Flash.

So in other image editing software (and apparently this holds true especially for video), pixels don’t have to be square. Pixels can have a different height and width, giving them an aspect ratio, which you can actually check for in PxB.

But then there’s PxB…it will scan each pixel in the image as IF THEY WERE square. So you end up with coordinates that could be x:4.56, y:1.567. When you do “sampleNearest”, you’re sampling the nearest pixel to these fractional values to end up with nice locked-in PxB world coordinates like x:4, y:2. You can also call “sampleLinear” which takes the average of the surrounding pixels when you ask for something on a half pixel.

Betcha feel smart now, don’t you? Well forget everything you just learned. If you are doing things in Flash, all pixels are square, and all pixels match to the PxB world coordinate system perfectly. So “sampleNearest” is just something you have to do to get the red, green, blue, and alpha values of the pixel.

So – in the end…you’re just taking the pixel you’ve come to, evaluating the 4 channels, and dumping those right back into the destination pixels. In other words, you’re doing nothing.

At this point though, it becomes easy to start manipulating an image. Go to the file menu again, and load an image. PxB has some sample ones to use, like this one:

Now change dst = sampleNearest(src,outCoord()); to:

dst = sampleNearest(src,outCoord()) * float4(0.25, 1.0, 1.0, 1.0);

Now click “run”

Congratulations! You just went into every pixel and turned the red down to 25%.

How bout a weird cross-hatch type effect?

dst = sampleNearest(src,outCoord()) * float4(sin(outCoord()[0] * 4.0), cos(outCoord()[1] * 4.0), sin(outCoord()[0] * 4.0), 1.0);

Play around, try different things. If you break anything, Pixel Bender will give you red error messages of varying usefulness on the right side.

The hardest thing to get used to is always typing numbers with decimals and usually performing operations not with one set of numbers but with a set of 4. If you run into any trouble, keep asking yourself these two questions:

  1. Am I performing a mathematical operation on two different data types?  Float2 * Float4 = Error!
  2. Am I performing a mathematical operation on a float using a number with no “point zero” on the end?  Float * 2 = Error!  Float * 2.0 = Good!

So that’s the basics of PxB!  You can manipulate surrounding pixels if you like by performing operations on surrounding pixels.  Just add or subtract X and/or Y to your outCoord(), and sample that pixel.  Combine and average surrounding pixels to get a blur effect for example.

Here’s an example of taking a big image, and downsampling the image to a tiny corner in the upper left of the destination:

        float4 colorAccumulator = float4(0.0,0.0,0.0,0.0);
        float4 avg;
        colorAccumulator += sampleNearest(src, outCoord() * float2(9.0, 9.0) + float2(-1.0, -1.0));
        colorAccumulator += sampleNearest(src, outCoord() * float2(9.0, 9.0) + float2(0.0, -1.0));
        colorAccumulator += sampleNearest(src, outCoord() * float2(9.0, 9.0) + float2(1.0, -1.0));
        colorAccumulator += sampleNearest(src, outCoord() * float2(9.0, 9.0) + float2(-1.0, 0.0));
        colorAccumulator += sampleNearest(src, outCoord() * float2(9.0, 9.0));
        colorAccumulator += sampleNearest(src, outCoord() * float2(9.0, 9.0) + float2(1.0, 0.0));
        colorAccumulator += sampleNearest(src, outCoord() * float2(9.0, 9.0) + float2(-1.0, 1.0));
        colorAccumulator += sampleNearest(src, outCoord() * float2(9.0, 9.0) + float2(0.0, 1.0));
        colorAccumulator += sampleNearest(src, outCoord() * float2(9.0, 9.0) + float2(1.0, 1.0));
 
        dst = colorAccumulator/9.0;

My next post will be about taking Pixel Bender and using it for non-image data processing. Stay tuned!

March 19, 2010

Pixel Bending for Speed in Flash and Flex

Filed under: flash/flex,music video games — admin @ 1:17 am

Lately, as I’ve been progressing with a Flex 4 based application which is largely an audio visualizer, I’ve felt the pain of slow response times from my UI.

The slow response is due to the fact that I’m loading a 3 minute MP3, extracting the audio to a byte array, and then processing the data in that byte array.  Consider a 180 second song with 44,100 samples per second.  That’s 7.9 million numbers to process as fast as I can.

This has caused me to do some complicated things.  The best of my efforts entailed processing the audio in segments on every enterframe handler.  I’d tell flash to process a reasonable amount of data, stop, and then process more the next round until everything was finished.  If I remember correctly, this took around 20-25 seconds or so, and STILL made my UI very sluggish as I was processing the data.  And of course this was in my development environment – in real life, a user would be waiting even more time while an MP3 loads.

So, I had two basic problems.  The first was that data takes too long to process.  I can hide this potentially with a good user experience – after all people expect that loading a large file can take some time.  But this leads to my second problem – Actionscript runs on one thread.  Only one thing can happen at a time.  If I’m processing audio, I can’t be updating/refreshing my UI.  And the whole application becomes unresponsive.  The more responsive I make my UI, the less data I process in a frame – but this just makes the whole darn thing take longer to process. Worse yet, if I process the whole thing in one go, Flash can timeout from inactivity at 15 seconds.

Basically it boils down to running it all at once, and potentially doing this:

or this….

Fortunately, there are two such Flash technologies that had the promise of helping me out.  First there’s Alchemy.  Alchemy is an Adobe research project that compiles C++ code to a Flash SWC.  Supposedly it can run code up to 10x faster than code compiled with Actionscript.

But, I haven’t touched C++ in a while, and still had the problem of an unresponsive UI for the time it takes to process the audio (whatever that time is).  It was tempting, but I had my sites set on trying Pixel Bender.

Pixel Bender is a cross-product Adobe technology.  It runs in Flash, Photoshop, and After Effects.  Basically it allows developers to write their own image filters.  In Flash, you can apply this image filter to images, animations, video, components…..and well anything that displays on the stage.

The best part?  It can run in different threads, on different CPU cores, and on your GPU.  Well, actually, strike that last part….you can’t run it in Flash on your GPU, but Photoshop and After Effects are cool.

This means that you can run a Pixel Bender shader on an image, and your UI doesn’t slow down.  Maybe you wrote something insanely complicated, well…then your PxB shader will suffer performance, but your UI won’t!

It doesn’t stop there – you don’t even have to have an image as your…erm…source image.  Yes, PxB assumes red, green, blue, and alpha channels.  But you can easily lie to PxB, and have it assume that your custom data is RGBA data.

This brings me full circle back to my problem.  I have something that supposedly processes data quickly, and on a different thread.  Hooray!  In fact, there’s a few projects going on that use PxB for audio processing already.

Come to the March RDAUG meeting on Tuesday to find out more!  I’ll be presenting, and quickly talking about what I covered here, but also diving into the nitty gritty.  You know….how to actually use this stuff in your work.

I’ll also be following up this blog post with a second part next week covering what I went over in my presentation.

February 9, 2010

Vacation from Flash

Filed under: flash/flex — admin @ 9:14 pm

I just came back from a long weekend trip to Orlando, Florida.  Just prior to that I had been working to launch this website – complete with a jQuery driven photo album, Joomla based CMS, and WordPress based blog.

This past weekend, waiting in the long Disney World lines, I had plenty of time to read the tweets coming in as the Adobe development teams and the Flash developers I know responded to Flash bashing and HTML 5 superiority in the wake of the Apple iPad announcement and the impending release of Flash Player 10.1, which Adobe is pushing for mobile devices.

I’m not going to add anything to this argument, but my current opinion stands that everything has it’s time and place.  People use Flash horribly just like they’ll use Javascript and HTML 5 horribly, and likewise each will produce some great things.

While most folks are comparing the end result when you create with the different technologies, what was eye opening in creating my website is the comparison in the development process between Flash, Flex, HTML, CSS, and Javascript.

Javascript and Actionscript are the most obvious comparisons.  Both have pretty much the same syntaxes.  Both are ECMA based, however Javascript has a few less features, that to me….can slow development and can lead to poorly written and badly maintained code.  Please note that I said CAN slow and CAN lead to!

Let’s go back in time a little.  Flash 5 and 6 were great at the time,  but didn’t encourage any good programming standards.  Basically you’d write scripts on each frame of your animation.  You could also (and many people did) go into each object and animation and write scripts on those as well.  In the end, you could have a huge unorganized mess which was impossible to comprehend or troubleshoot.

If Flash was your first development environment, over time, you’d start to see that you’ve created big unorganized messes and start to think about how to better maintain your projects.  However, there was absolutely nothing in Flash that encouraged this behavior.

Flash 6 and Actionscript 2 started rolling out with new enhancements to the ECMA standards.  This included the ability to organize your code in classes, and really rounded out an object oriented approach to development.  You still weren’t forced to acknowledge OO practices in your work, but the option was definitely there and some of the more experienced programmers could utilize them.

With Flash 9 and Actionscript 3, developers who tried to do anything substantial were forced to start acknowledging OO practices.  This approach led to Adobe facing some complaints from the casual Flash developer, but also led to the larger community adopting some good (or better) practices around object oriented development.

I think this very much relates to the state of Javascript today.  Someone coming from a more robust language like Java or C++ could pick up both Flash 5 or Javascript today.  They could absolutely introduce a code structure to mimic  an organized object-oriented approach.  However, the reason they can do this is because they have the experience to know how to better organize code, and they know right off the bat what features are missing, and can concoct ways around them.

Contrast this to someone just learning Javascript or Flash 5 as their first development language.  They just wouldn’t know any better than to create lots of inline code, not realizing that classes or methods even exist.  As this person gets more and more experience under their belts, they would see how their approach lacks maintainability and strive for something better.  In the interim however, this leads to horrible, mangled, and unmaintainable scripts.

Actionscript has increased by leaps and bounds over the years, but Javascript still has not.  Great code can and will be developed in each environment – but I just don’t think that Javascript encourages any good practices.

Keep in mind that I’m well aware that the exact same thing can be said about Java/C++ versus Actionscript with Actionscript on the low end of the totem pole here.

The other thing to point out is variable typing.  Having to call a variable a string, number, etc when first declaring it seemed like a nuisance, but turned out to speed up development quite a bit.  This is an important thing that Javascript lacks.

Someone, a year back, turned me onto Aptana for use when developing with HTML/CSS/Javascript.  Aptana is fabulous.  I get to use my experience with Eclipse and Flex/Flash, and just switch to something that looks exactly the same to develop my non-Flash web pages with.  Code hinting is huge for me.  I don’t constantly have to look up syntax, and can start typing and have Aptana suggest what to do next.  This is incredibly helpful for CSS, since I don’t remember all the properties and need constant help.

With Javascript, on the other hand, code hinting is hindered a great deal.  If I don’t declare a variable type, Aptana can’t suggest what properties and methods I can use on an object very accurately.  It actually does a great job guessing (especially with jQuery), but it can only go so far if it doesn’t know what kinds of objects you’re working with.

I suppose you could call CSS a win on the non-Flash side.  CSS, to me, is simply a joy to work with given the right tools.  Aptana, as I said before, does well with code-hinting for CSS.  As a Flex developer who knew enough CSS to style text in HTML and style my Flex components, really getting into CSS was a little daunting.  It’s a lot more powerful and robust than I would have thought.   Flex doesn’t give you the ability to control layout and positioning with CSS, nor does Flex 3 allow inheritance and chaining styles together (Flex 4 does though!).  It can also be a little confusing in Flex to know which properties to set for what you want to do.

All in all, I think I enjoy style and layout in HTML/CSS more.  I’m still a newbie, but it’s my shiny new toy.  Unfortunately, my shiny new toy is very tarnished by having to maintain compatibility in Internet Explorer.  I recently tried and failed to use a glow effect for some text that looked great in Firefox and Chrome, but looked awful because of IE’s clear type.  And then of course, there’s no real practical way to use vector graphics (IE doesn’t natively support them).

For pure programming power, I’ll choose Flex and Flash any day of the week.

I look forward to getting more experience with Javascript and CSS.  It’ll definitely help when I’m evaluating a project.  I’m sure I lean too far on the Flash side right now, and I look forward to being able to better approach a project from any perspective.

February 3, 2010

Retiring yellow5labs.com, launching blastanova.com

Filed under: Uncategorized — admin @ 7:24 pm

Oh so long ago (2001 maybe?), I started yellow5labs.com.  I had been recently laid off, and wanted to set myself up with an online identity to get contract work and look all flashy so people would hire me.

The name yellow5labs was inspired from Yellow #5 food dye.  You know, that stuff…that you put into twinkies, and doritos, and all sorts of good stuff to give something plain that extra pizazz.  I was kinda thinking I would sell it as I was bringing that pizazz to my interactive work.  I still think it’s a pretty cool name, except it’s horribly impractical.

Every time I have to give my email address, I have to say “yellow…..five…..no, the number five, don’t spell it out, labs.com…..no, I don’t have 5 Golden Labrador retrievers…..no…FIVE….don’t spell it out…”

It got a little tiresome.  The site is still up, but I’ll probably just let the domain lapse this summer.

Anyway, in with Blastanova.  If you hear it, I think you’ll know how it’s spelled, and that should save me a lot of grief.  Why blastanova though?  Well, lately I’m both fascinated with the idea of music based video games, and sad that I don’t like music as much as I used to.  I think a lot of it has to do with the fact that I’m not forced to use the CD player in my car anymore, and I have an unlimited amount of music.  So songs are almost disposable now, and no real connections are made with the artists, because you aren’t listening over an over again to the same disc.

I think games rhythm games (like Rock band most recently), really get you listening to the songs over and over again, and really establishing a connection between you and the music.

So – “blast” as in blasting stuff in video games…..and “blastanova” as in bossanova, a music that literally translates to “new beat” and derived from Samba which itself is known for it’s rhythm.  I dig.

In the short term, I’ll be blogging on new media, developing for the web, Adobe’s Flash & Flex, and bands I’m listening to.

In the long term, I’m developing an application to scan songs and their rhythm and structure, and soon thereafter, I’ll be applying the data to games I develop.

So that’s the plan.

July 14, 2009

Composing Music with the Flash ByteArray

Filed under: flash/flex,music video games — admin @ 11:43 pm

Demo here, view source for code

(view source for code, and who knows what bugs there are)

I’m getting into unfamiliar territory these days with exploring sound in Flash.

Flash 10 gives you the ability to take a sound object, and extract the entire thing into the raw data….a byte array.  How do you interpret this data? How can you process it into something meaningful?  Well I don’t know.

What I do know is that to get a firmer grasp on how I can utilize this, I need to understand what the sound object is at a basic level.  Probably the worst way to understand it is to take an entire song (mp3) file and let my eyes glaze over at the stream of numbers coming from the sound.extract feature.

No – the best way, I thought, would be to compose my own notes and chords, and work up from there.  To do this, I needed to understand what a note is, and what a sound is at a basic level.

My first thought was to picture sound as the nifty looking sound visualization that comes with music players these days…you know, those dancing bars:

Turns out that this is the worst way to imagine sound for this purpose.  My first question was….”OK, how do I get the value of the low frequency?”, “how about something in the midrange?”, “what about the highend?”.

Yes you can imagine sound this way, but only after processing your sound data with a Fourier Transform.  If you know how to do this, stop reading this right now, cause you’re way smarter than I am at this point in my sound exploration.

The BEST way to imagine sound is to picture a sine wave:

If you look at how tall the wave is, that is how loud the sound is, or the amplitude.   How close the peaks and valleys are in this picture is the frequency.

Frequency, at this very basic level can’t be thought of as high pitched, low pitched, etc – it’s only how far apart these peaks and valleys are.  Now think of this as a side-scrolling wave that goes on forever.  If you were to scroll at a constant speed, each peak would hit that vertical center line at a constant rate.  This rate would be the frequency.  If it goes faster, the frequency is higher – slower, the frequency is lower.  And, of course, this directly relates to how you hear it.  The more peaks and valleys at a given time, the higher the tone you hear.

Now think about how this relates to our Flash sound object’s byte array.  We could actually draw a sine wave with numbers.  This is nothing new to programmers.  Folks use trigonometry all the time.  But, I personally, never thought to use it for sound.

Try this AS3 code:

for (var i:int=0; i < 100; i++ ) {
var sample:Number;
sample = Math.sin(i * 2 * Math.PI) * 50;
}

This will basically plot a picture like the one above – a sinewave.

If we made this into a sound, it almost, but not quite be a tone.  However, if we pop over to google, we can actually look up the frequency, of say…..a middle C note, or a middle A.

http://www.phy.mtu.edu/~suits/notefreqs.html

There’s one more piece to this puzzle though, and that’s sampling rate.  A digital audio file could have all the sinewaves, peaks and valleys in the world, but if your audio playback is super slow, it’s gonna sound like garbage.  That’s why we tell our audio software to read our sound at a certain speed.

If you’ve heard the term 44.1 kbps when folks talk about sample rate – you can look back at our little for loop that drew the sine wave, and realize that you need more than the 100 points of data as we drew, you need 44.1k, or 44,100 PER SECOND.

So let’s rewrite that code:

for (var i:int=0; i < durationinseconds * 44100; i++ ) {
var sample:Number;
sample = Math.sin((i) * 2*Math.PI/44100 * 440) * volume;
}

OK!  So now, we’re creating a one second middle A (4th octave).  Our sample rate in Flash is always going to be 44100 if coming from a flash sound object (though that’s not true for any audio file).  We learned from our handy/note frequency chart when looking on google that 440hz is a middle A.

As a side note, lets say we want a different octave.  To go lower, half the frequency to get 220.  To go lower, half that.  Higher?  Double it.

Let’s think about the voice now.  The nice sine wave will give a nice, round tone.  For a dirtier tone that sounds like it has sharp edges….well our sine wave needs to have sharp edges – which is actually a square wave.  Our nice round peaks and valleys would be just straight corners.

To change the tone in the code, try this:

sample = Math.sin((i) * 44100 * 2 * Math.PI * frequency) > 0 ? volume : -volume;

As for more voices, well, I haven’t tried it, but a real world instrument would have a hard strike and then some falloff.  So our nice sine wave would be less round at the start of each peak, but comes back down way slower.

Here’s my final code (and keep in mind that I’m writing the sample 2 times, one for the left channel and one for the right):

returnBytes:ByteArray = new ByteArray();
for (var i:int=0; i < _duration * 44100; i++ ) {
var sample:Number;
if (_voice==VOICE_SQUARE) {
sample = Math.sin((i) * 44100 * 2 * Math.PI * this.frequency) > 0 ? _amplitude : -_amplitude;
} else {
sample = Math.sin((i) * 44100 * 2 * Math.PI * this.frequency) * _amplitude;
}
returnBytes.writeFloat(sample);
returnBytes.writeFloat(sample);
}

Now, what do we do with that byte array?  Well, as of Flash 10, our sound object, has a sample data event.  When this event is called, when it needs new bytes, it’ll call out to your custom sample data method.

What I do – and what you may choose to do, or not choose to do, is make my whole byte array first, and then read sequential amounts of data into the byte array each time it’s called:

soundBytes.position = 0;
dynamicSound = new Sound();
dynamicSound.addEventListener(SampleDataEvent.SAMPLE_DATA, addSoundBytesToSound, false, 0, true);
soundchannel = dynamicSound.play();

private function addSoundBytesToSound(event:SampleDataEvent):void
{
var bytes:ByteArray = new ByteArray();
soundBytes.readBytes(bytes, 0, Math.min(soundBytes.bytesAvailable, 8 * 8192));
event.data.writeBytes(bytes, 0, bytes.length);
}

To explain, about the 8 * 8192….

8192 is the maximum amount of samples you can use for each sample data event.  However….each sample is a left and a right 4 byte float.  So that’s 8…..time 8192.

There’s tons of cool stuff you can do with this.  If you don’t believe me, look up Andre Michelle – he’s THE MAN when it comes to this stuff.

June 27, 2009

Working with the Flash Sound Object as a ByteArray

Filed under: flash/flex — admin @ 4:24 pm

Last year I was experimenting quite a bit with Flash’s computeSpectrum functionality.   With computeSpectrum, I could take a snapshot of a playing sound and get volumes for various frequencies of the snapshot so I can visualize the sound.

Now, I wanted to take it a step further and visualize the sound before it’s even played.  Of course doing this with computeSpectrum is impossible because the sound has to be played first.

Luckily, in Flash Player 10, Adobe introduced sound.extract();  This can take the sound object and turn it into a byteArray.  I came across an example by Thibault Imbert and of course the example worked very well to visualize a sound spectrum from a byte array – though I still wasn’t sure what each byte represented in the Flash Sound object.  And even though, the spectrum looked OK,  a comment from Schell explained that this is a false waveform and to look at his blog post.

The blog post provides some great working code – but there was quite a bit I didn’t understand yet.

So, I attacked the problem from another angle today.  I thought – OK, I don’t really understand the bytes that are coming OUT of the sound object, but maybe if I play with shoving bytes INTO a sound object to create dynamic sounds, I can get a better handle on tings.

The real struggle came because I used Christopher Martin-Sperry’s code to create a sound object from the raw byte array provided by an MP3 file.  I used this before really looking into how all this stuff works.  The code works fantastically – but put my brain on the wrong track.

I was on the wrong track because MP3′s and Flash’s sound Object work a wee bit differently from each other.  In an MP3 file, there is a header to give some info about the file.  Also, the MP3 file is broken into “frames”.  Each frame has a header to give information about the frame (bitrate and other things).

This doesn’t map so well to Flash’s sound object.  First of all, the bitrate of a Flash sound object looks like it doesn’t vary.  It seems to be 44.1 kilobytes per second all the time.  And it’s always in stereo.  It looks like the source of the audio will always get upconverted or downconverted to this 44.1KBps stereo format.

Not only that, but there’s no concept of frames.  I thought I’d need to know how to read a frame header, or at least know how long each frame is to know what type of data I’m getting.

But that doesn’t appear to be the case.  The byteArray extracted from a sound object appears to be an alternating left/right channel stream.

How does the stream relate to time? I felt a little dumb when I figured this one out – because it’s a little obvious.  It generated a few tones before having this little gem of an epiphany.

Well, if the Flash sound object is 44.1KBps, that’s 44,100 bytes per second.  So when reading the stream, we read 44,100 bytes before a second is up. Right?

Well, not so right it would seem.  You have to consider both channels.  You’d read each channel 44,100 times to make up a second, making one second 88,200….something.

What is that something?  Well my computer sciece teaching has failed me, but I can speculate.  Adobe provides a morse code generation sample on their developer network.

In it, there is a comment when they are writing the samples to the sound stream that they are writing 8192 samples to the stream.  However, the length of the byte array they are actually writing is 8192 x 8, explaining that each sample is two 4-byte floating point numbers.

So perhaps when reading floats with ByteArray.readFloat, you’re actually reading both left and right floats from the same byte.  If I’m right, then you’re still using 44,100 bytes per second, but each byte represents two floating point numbers.

However, it works out, I’ve noticed that when creating six 1 second tones, by writing two floats at a time, 44,100 times,  it clocks in at 6 seconds.  So infer whatever you wish from that – I know I have.

My next step is to figure out how to pull frequency data at a particular point in time.

February 18, 2009

theWB.com

Filed under: projects — admin @ 7:29 pm

Just writing a short post to tout theWB.com which went live today after a makeover.  My new team and I at Digitalsmiths produced the video players and the clip search tool.  After just cancelling cable, I can’t argue with even better video sites putting out great content.  I’m always a fan of Buffy and Angel which they have plenty of.  Everyone should also definitely check out the short WB shows “Joni and Susanna” and “Children’s Hospital” starring Rob Cordry.  Someday soon I’ll definitely check out “Babylon 5″, but I don’t know if I have enough geek cred yet.

http://www.theWB.com

« Newer Posts — Older Posts »

Powered by WordPress