VFX / Sound Driven Motion, Bake and Unbake

Here is a follow up to my previous experimental post (Perfume) that is more or less related to sound, motion, and animation.

The result of previous experiment. Just array of disc spawn and scaled based on Audio.

CREATIVE AUTOMATION
Recently, I am into topics where we can drive or automate or trigger certain actions.

It is not really new, but we do have more technology these days that allow us to further develop whatever ideas that we may already have in the past but stuck. Things like: Kinect Motion Capture, Augmented Reality and AR Glass, Mobile Devices, 3D Printers, 3D Scanner, Arduino, etc. They are all available.

It is as if nowadays, we are applying more and more "Game Environment" automation into "Real World".

Those areas are really interesting to me. I just wish I know more people that actually working on such projects and I don't have to rely on my brain alone. I will look around if maybe University around here have some space where I can research and learn more about all these.

And of course, since this blog is about Blender, I want to see some kind of real-time setup using Blender happening on my own personal computer.

I remember many years ago, I researching on how to do this kind of audio-driven animation using Maya (MIDI Driven). It was a dumb effort. There was actually a clever MEL script that kind of does it, but in the end the script is discontinued or kind of limited.

And then there is Houdini. I can see that Houdini CHOPs is actually perfect of Audio Driven Animation. Andrew Lowell released a book few years ago to cover this:
http://www.andrew-lowell-productions.com/andrew-lowell-productions/resources.html

I have not get my head around Houdini to be able to confidently create anything. But I have a bit more time now, I will look into it further soon.

What I like about Blender is that Blender is somewhat open to creative projects like this. And somewhat I got to know clever artists out there to help me with my random questions.

Anyways, a bit off topic, let's get back to Blender and see how Blender handles audio-motion.

RESEARCH ON BLENDER AUDIO-MOTION
A quick browsing around the Blender Artist forum for AUDIO RELATED post, there are apparently heaps already. I really have to do a little bit more research and testing all the tools to actually write a useful post.

batFINGER Speaker Tools Script Thread
http://blenderartists.org/forum/showthread.php?236729-Speaker-Tools-(Audio-Visualiser-Driver-Lipsync-)

MIDI Driven Thread
http://blenderartists.org/forum/showthread.php?182443-MIDI-driven-animations

There are also this from Japan Blender artist/developer:
Blender Real-time Lipsync Tool from Microphone
https://vimeo.com/38157187

I did try that Real-Time Lipsync Tool, but got stuck. Well, maybe I need to spend a bit more time on it. That Real-Time Lipsync Tool requires a few 3rd party softwares outside Blender. It feels complicated, but maybe not that complicated once understood. It has interesting potential.

Not long ago, I also stumbled a few good video tutorials on Blender audio driven animations (was pointed by Pavel yesterday). I shall look into them again soon.
http://cgcookie.com/blender/2011/06/02/building-an-audio-visualizer-in-python/
http://www.blendernation.com/2012/01/17/audio-driven-animation/


MAKE SOMETHING ANYWAY
So, I guess I am a total newbie in this audio-motion area. Exactly like when I started to learn Blender, it took me a while to filter all the information and trying to make sense of things (-__-)b

Anyways, let first review things that we already know.

1. BAKING SOUND TO F-CURVES

It cannot be any easier than this. This is actually one of the feature that gives me a big surprise when I first learned Blender 2.5x.

Firstly you start with some kind of keyframe on any object. Specify what parameter you like to bake sound into and create a keyframe there (so that you can later bake sound into it).

Inside Graph Editor, all you need to do is to Bake Sound to F-Curves. Point into audio WAV file (or MP3 audio file will work as well).

On the file import panel you will see Bake Sound to F-Curves option on the left side bar:

Pay attention to the Lowest and Highest frequency.

The resulting F-Curves:

NOTE: I like to do this to an Empty so I could then set it up to drive something else. Maybe one could try on a Bone or Armature as well and let it drive something?

2. UNBAKING F-CURVES
This one awesome script comes from Liero called UNBAKE. With unbake add-on script, you can basically convert those "baked" F-Curves into:
1. 3D Curve Path

Also read:
http://blenderartists.org/forum/showthread.php?244692-waveform-data-to-3D-object

2. Editable F-Curves

And even more, this "Unbake" script from Liero has extra options, especially for "unBake to 3D", if you check Use Radius, you will get a straight curve but with Extrusion.

Do that above and apply Build Modifier on top, and you have something that similar to my previous experiment. It's all procedural.

Now, we know that Blender Curve is special. It can store Radius on each points. And if you parent a Mesh into it to Follow Path, the mesh will understand the Radius and will Scale up and down based on that radius. Therefore .... you can have something like this: (that's particle emitted).

How cool is the UnBake Script? Tell me about it, I did not even know this is possible.

WHAT CAN WE DO WITH IT
Alright, we got some interesting raw data to work on. What can we do with it? That is exactly the question I am still trying to answer while writing this....

Normalize? New Min Max?
I think we need to NORMALIZE this curve first. I don't know quite how to do that in Blender yet. But the idea is to turn this F-Curves into 0 to 1 value and map into new min and max value that is more usable.

You could clamp the value using Limits Modifier. I am not quite sure yet with Normalize.

Further Modification of F-Curve

Since Baked Audio is now an F-Curve, you can further modify the curve using F-Curve Modifier. Lots of interesting Modifiers there, such as Stepped, which turn the curve into something that looks like a random city buildings.

Using that Unbake Curve As Motion Path
Right away, you can simple use that path as a Motion Path. Maybe parent an object and let it emit Particles or Smoke. Or perhaps parent a Brush object to paint on Dynamic Paint Canvas to create something that looks like a graph. I am sure there are many cool motion graphics music video done that way. Here is one:
https://vimeo.com/6045312

I want to do some smoke, but it's just at the moment I ran out of HDD on my computer. And smoke requires some GBs and also time consuming baking and testing.

Array Modifier and Curve Modifier?
This is Liero's suggestion, I don't understand how to manipulate object so that it affects the "Scale" or "Array" objects. But I am getting something like this by using that Curve generated from Sound.

Array and Sound
Array (Copy) Counts seems to have a nice potential. Let see if we could drive the value of Counts using the data from sound wave:

Yes, it is quite easy to drive array copy with Audio Bake using Drivers.

That setup can give a very simple type of Sound Visulization without any scripting.

If you can somewhat offset and animate the Curve, you can have something that is more interesting. Kind of like a moving mountain of arrays. The long way to do this is to use Drivers.

Displace Modifier
Just connect the wave into the Strength of Displace Modifier and you get something that looks interesting.

Dynamic Paint
Sound Driven Painting.... it might be interesting.

Alright, I stop there for now. I think Sound Driven Animation/Motion is interesting but I need to do a bit more research on sound visualization and all kind of sound to visual experiments that have been done before.

I think if know Blender a little better in the future, it is possible to create Interactive Game that uses Motion to Drive or Trigger Animation using Blender Game Engine (BGE). I wish someone could point me to it.