Revolution, Part 1: Audio

A friend of mine has been working very hard on a project which involves two revolutionary technical innovations. I am not usually the kind to care for revolutions, but I honestly believe that both of these are really, really significant. I’m actually hoping to join up and help make these things into a proper product, in fact. These are things you have not seen in games, and he has it working in prototype form on a mobile platform. That’s unusual in this industry.

Here, listen to this Youtube video while reading. You need headphones for this, which is why the tech is launching on the iPhone first. The video is not ours, but it is a very good demonstration. Just listen and read. Remember, it doesn’t work without headphones — if you don’t have any, it might actually be worth reading this only after you’ve found some.

The underlying principal is binaural recording. (This is not the revolutionary bit, and has been around for quite a long time.) Games have had 3D audio for ages, so that is in itself nothing to get excited about. Current 3D audio basically works by modifying channel volumes for playback of a mono sound in order to simulate a 3D space. It works alright if you have a 5.1 setup, but it’s not terribly effective in stereo and in general the effect is a bit weak. Binaural recording, however, is a method of recording sounds with a pair of microphones and an actual head model that attempts to produce a stereo sound that simulates what our ears hear. You need headphones because of the recording methodology, and if you’re listening to the video I linked, you’re probably spazzing out right now.

There is a catch to all this, which is that nobody can synthesize it. The sound is recorded by physically placing it relative to the head, so you can’t go back later and place it at an arbitrary location. (Some people have pointed out that there are processors and algorithms that try, but they are expensive and don’t really work well.) That’s essentially why it’s never showed up in games –although headphones-only isn’t a thrilling restriction, either. Still enjoying the barber?

Here is his binaural recording. (It’s 8 seconds, just pause the barber.) There’s one key difference, though. That’s not a binaural recording of a sound being moved in front of a recording head. It is done in real-time. (This is the revolutionary bit.) This friend of mine has figured out how to do it. The original implementation worked very well but required a lot of memory and processing power. But the current system is efficient enough to fit on the iPhone. I’ve seen and heard the demo, working in real time off an iPod Touch. It works well enough to make your skin crawl, like those scissors are probably doing right now if you’re still listening to the barber.

I can’t really say too much about how he’s pulled it off, because we think it’s kind of a big deal. You’ll see iPhone game releases with the technology later this year, and hopefully by early to mid next year we’ll be licensing an actual SDK for whatever platform you might care to use. We’re fairly confident this is technology people will want, and hey, it wouldn’t hurt to forward this post around the office.


2 thoughts on “Revolution, Part 1: Audio

  1. I’m definitely not an expert in audio programming, but a thing has been bugging me since I’ve read your post – aren’t audio engines like FMOD doing the same thing? I mean, they have a 3D sound source setup, they have a listener, they compute sound volumes, panning, etc. for each speaker – what’s revolutionary about your friends approach?

  2. Well, there’s a few components to it. If you look at what happens to a sound wave as it hits our head, there’s a very complex series of reflections and refractions as it enters the ear canals. Traditional 3D audio doesn’t attempt to simulate this; what it does is to define angles for each speaker in your setup, and determine channel volumes based on those angles and the distance of the sound. The approach behind surround sound is to, as much as possible, create the sound at that location and let our brains’ localization hardware do the work.

    Binaural audio instead subtracts your internal localization completely, sending audio directly into your ear canals. That’s why it only works with headphones. There have been attempts in the past to model binaural effects mathematically, in the form of Head Related Transfer Functions (HRTFs). These don’t really have great success in doing it, though, and these audio clips are usually generated by physically putting microphones into a dummy head and recording a sound.

    Our innovation is that now, we can position the sound anywhere we want in real time, and get the full binaural effect.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s