benbritten.com

Hey All,

I have been fiddling with openAL on the iPhone for awhile now, and have discovered a few things. If you havent already read it, (and want a basic overview of openAL on the iPhone) have a look at the openAL tutorial I posted a while back.

Today I want to talk a bit about how the Audio Sessions system interacts with the openAL stuff on the iPhone, and how to get it all working in your app. It took me a looong time to figure this out and get all the little bits in the right place at the right time. Many thanks to the contributors at the apple iphone dev forums, couldn't have done it without them. In any case, hopefully this will save someone out there some time trying to figure this all out.

First off: WTF are Audio Sessions? From the docs:

"Audio Session Services is a C interface for managing an application’s audio behavior in the context of other applications."

So basically, if you want your app to play nice with other sound making things on the iPhone, you will need to deal with Audio Sessions. The biggest reason to do this is so that you can play your iPod music and have it mix with the sound that your app is generating.

So, lets get right into it.

First off, you need to let the Audio Session Service know that your app exists and give it a way to talk to you via a method callback:

 
OSStatus result = AudioSessionInitialize(NULL, NULL, interruptionListenerCallback, self);
 

The first two params that I am passing as NULL specify which run loop and what run loop mode you want to use when calling the callback method. passing NULL basically says: "Use the main run loop, and the default run mode", which for our purposes here is going to be fine.

the third param is the callback method pointer, more on that in a moment.

and lastly, the final param is the object that is the target of the callback message.

OK, now we have let the Audio Sessions know that our app exists, the next thing to do is tell it what kind of sound we will be generating. We do this with an audio category.

There are a handful of valid categories:
kAudioSessionCategory_UserInterfaceSoundEffects
kAudioSessionCategory_AmbientSound
kAudioSessionCategory_MediaPlayback
kAudioSessionCategory_LiveAudio
kAudioSessionCategory_RecordAudio
kAudioSessionCategory_PlayAndRecord

If any of these sound like your app, then look em up in the docs and double check. I am going to talk about the two that I use for most everything: kAudioSessionCategory_AmbientSound and kAudioSessionCategory_MediaPlayback.

kAudioSessionCategory_AmbientSound: This one is for 'longer' sound playback, but works fine for short sounds as well. It is what i refer to as the 'game category'; good for games. good to play longer looping background sounds (like a rockin soundtrack, or atmospheric noises) as well as short effects (like gunshots or interface clicks) This category will also allow the iPod music to mix with your app sounds.

kAudioSessionCategory_MediaPlayback: this is good if you have your own music and do NOT want to allow the ipod to play while your app is active. Basically this one tells the iPhone that you want your app to be the only source of sound. Good for things that generate music, or where it doesnt make sense to allow for iPod sound.

OK, so how do we tell the audio session about our category?

 
UInt32 category = kAudioSessionCategory_AmbientSound;
OSStatus result = AudioSessionSetProperty(kAudioSessionProperty_AudioCategory, sizeof(category), &category);
 

Pretty easy really, just set the property and you are done.

OK, with just those few lines of code your app will play nice with other apps and allow (or not) iPod music to mix with your sound.

But there is one more thing we need to do: handle sound interruptions. Like an alarm or phone call. (note: if you get a phone call and answer it, then your app will be dismissed. However if you get a phone call an ignore it, then your app will get the interruption call. Of course, on the Touch you dont get calls but you can still get alarm clock interruptions etc..)

OK, so we need to define a callback method. this needs to be some ugly C code, so all you Cocoa-only types, just hold your nose and cut and paste :-)

 
void interruptionListenerCallback (void   *inUserData, UInt32    interruptionState ) {
 
        // you could do this with a cast below, but I will keep it here to make it clearer
	YourSoundControlObject *controller = (YourSoundControlObject *) inUserData;
 
	if (interruptionState == kAudioSessionBeginInterruption) {
		[controller _haltOpenALSession];
	} else if (interruptionState == kAudioSessionEndInterruption) {
		[controller _resumeOpenALSession];
	}
}
 

What is going on here? Pretty simple really; the inUserData is a pointer to your object (or whatever obejct you put in for the last param above when you called AudioSessionInitialize()), and interruption state is an enum so you know if you are being interrupted or resumed.

I like to get right out of the ugly C code and right back into the lovely land of Cocoa by calling some cocoa methods on my controller object. (though you could just call the openAL and audio session functions from right there, but it is ugly)

So, how do we halt and resume openAL?

first thing, let the audio sessions know that you are shutting down

 
// Deactivate the current audio session
AudioSessionSetActive(NO);
 

Next, shut down openAL in a nice way so you can keep your context intact:

// set the current context to NULL will 'shutdown' openAL
alcMakeContextCurrent(NULL);
// now suspend your context to 'pause' your sound world
alcSuspendContext(mContext);

easy.

And to come back from the above halting?

Basically the same thing in reverse, but with one added twist: you have to re-register your audio session category:

 
// Reset audio session
UInt32 category = kAudioSessionCategory_AmbientSound;
AudioSessionSetProperty ( kAudioSessionProperty_AudioCategory, sizeof (category), &category );
 
// Reactivate the current audio session
AudioSessionSetActive(YES);
 
// Restore open al context
alcMakeContextCurrent(mContext);
// 'unpause' my context
alcProcessContext(mContext);
 

And that is it. That will allow your openAL app to play nice with the rest of the system and recover from interruptions.

edit: renato reminds me in the comments that you should always be checking your error codes. Otherwise openAL can choke on some small issue and it wont work until that code has been cleared.

I tend to put one of these after every call to openAL: like so:

 
alSourceQueueBuffers(sourceID, 1, &bufferID);
ALEnum err = alGetError();
if (err != 0) NSlog(@"Error Calling alSourceQueueBuffers: %d",err);
 
alSourcef(sourceID, AL_PITCH, pitch);
err = alGetError();
if (err != 0) NSlog(@"Error Calling alSourcef with AL_PITCH: %d",err);
 

and so on.

Cheers!
-ben

Hey All,

my last post (which was a zillion years ago it seems) was about openAL. It has gotten quite a bit of traffic in the past few months, so apparently openAL on the iPhone is something people are interested in. So i am going to post this which is a few more tidbits that I have discovered over the intervening months about openAL.

First off, i have noticed that the vast majority of people are taking apple's sample code 'SoundEngine' and plopping it directly into their code, and then spending countless hours and days trying to get it to work just right for them.

Here is my advice:

Dont.

Look, I love the apple engineers, I love the work they do, but sample code is just that; a sample. It is meant to show you all the various different ways you can do something. the SoundEngine code does way more than you should ever really need. Your best bet is to start from scratch, figure out what your sound needs are and build you own sound controller object. If you do this you will be in a much better place to debug (because you will know how it all works) and you will be able to add only the code your app needs, and less code means less bugs.

What you say? What about code re-use?! I shouldnt reinvent the wheel, I dont fall into the 'not invented here' trap!! Heresy!

to that I say: you are re-using code. you are re-using the openAL codebase. but it turns out at the level that you are gluing openAL into your app, you should strive to make your own implementation geared towards what you need.

This is especially true for the iPhone. If there is any code in your app that you can get rid of by pre-formatting your sounds then you should be doing that. There is absolutely no reason that your sounds should be in more than 2 formats (ie an uncompressed format for your short sounds and a compressed format for anything longer) if you are checking the type and format of your sound files at runtime you are doing it wrong. Pick a sample frequency and format (stereo/mono) that works with all your sounds and make them all the same. This will simplify your code immensely and simpler code == less bugs == better performance.

Anyway, that was a bit of a rant. But I see so many posts to the apple dev forums that are like: "I am using SoundEngine from the crashLanding app and i am having this problem..."

Really, openAL is not that hard. I spent about 2 hours with the programmer guide and I got my simple sound controller working and wrote a tutorial since i was such an expert by then. That and the resulting code was about a tenth the size and complexity as soundEngine and I have had very few problems with it (and the problems I have had were easy to track down and debug because there just is not that much code there.

Anyhow, I really did mean to have some technical advice in here, and not just rant, however this is getting long, so it may have to wait until another post. :-)

-b

Hey all,

Now that the NDA is lifted, and we can start talking about the iPhone code out in the open, i thought it might be nice to talk about some of the problems I have encountered in my forays into the iPhone world and how I went about fixing them.

Currently I am working on an iPhone game. It is all openGLES based and uses openAL for sound. I think I am gonna talk about openAL today.

For now I am only going to be talking about sounds that are less than 30 seconds, so sound effects and short loops. Before you can think about playing sounds on the iPhone they need to be in the right format (or they should be in the right format, many of the audio toolbox methods will handle multiple formats, but if you put the sound in the right format to start, then the iPhone wont have to do it at play time).

So, pop open terminal and type this:

 
/usr/bin/afconvert -f caff -d LEI16@44100 inputSoundFile.aiff outputSoundFile.caf
 

what the hell does that do? you ask. it puts the file into a nice Little-Endian 16-bit 44,100 sample rate format. (generally saved with a .caf extension)

OK! now we have a nice .caf file in the proper format, we are ready to do something.

There are lots of ways to play sound on the iPhone, there is the 'easy' way, and then there are a few 'hard ways'.. I am gonna touch on the easy way quickly and then move onto the openAL 'hard way'.

the quickest (and easiest) way to make the iPhone spit out some sound is to use the audio system services:

 
NSString* path = [[NSBundle mainBundle] pathForResource:@"soundEffect1" ofType:@"caf"];
NSURL * afUrl = [NSURL fileURLWithPath:path];
UInt32 soundID;
AudioServicesCreateSystemSoundID((CFURLRef)afUrl,&soundID);
AudioServicesPlaySystemSound (soundID);
 

this works well for making your interface buttons click and simple UI interaction stuff. However, it is absolutely shite for anything more complicated than that (think: a game). It doest always play right away, and if you are trying to match up specific frame of your game with specific sound effects, then this method is basically useless. (I actually implemented my whole sound engine using the above style of code, then i got onto the phone and every time a sound played, it was either late by many frames or the whole thing would pause and wait for the audio toolbox to load the sound into the buffer, it sucked.

For better control of the sound, you will require either openAL or audioUnits or the audioQueue.

I decided to go with openAL so that my sound code could be kinda sorta portable, and by learning how to use openAL I would be able to use those skills on some other platform besides the iPhone. (and since I am a code-mercenary, i figured that having openAL experience was more marketable than audioQueue experience) (that and I already have familiarity with openGL, and openAL is very similar, and the audio units and audio queue code is kinda ugly)

So, this will be a super quick tutorial on openAL and the absolute bare minimum you need to do to accomplish static sound generated from openAL.

OpenAL is really quite straight forward. there are 3 main entities: the Listener, the Source, and the Buffer.

The Listener is you. Any sound the listener can 'hear' comes out the speakers. openAL allows you to specify where the listener is in relation to the sources, but for this example we dont care, we are going to bare minimum static sound, so just keep in mind that there is a concept of 'listener' and that you could move this object around if you wanted to do more complicated stuff, but I wont go into it in this post.

The Source: basically this is analogous to a speaker. it generates sound which the listener can 'hear'. like the listener, you can move the sources around and get groovy positional effects. However, for this example we wont be doing that.

The buffer: basically this is the sound that will be played. the buffer holds the raw audio data.

there are two other very important objects: the device and the context.
the device is the actual bit of hardware that will be playing the sound, and the context is the current 'session' that all these sounds are going to be played in (you can think of it as the room that all the sources and the listener is in. Or it is the air that the sound is played through, or whatever.. it is the context.)

How does this all work: (this is the bare minimum)

1) get the device
2) make a context with the device
3) put some data into a buffer
4) attach the buffer to a source
5) play the source

that is it! The above presumes that your implementation of openAL has decent defaults for the listener and if you dont specify any listener or source positions then this will all work dandy. (it works just dandy on the iPhone in any case)

so, lets look at some code:

 
// define these somewhere, like in your .h file
ALCcontext* mContext;
ALCdevice* mDevice;
 
// start up openAL
-(void)initOpenAL
{
	// Initialization
	mDevice = alcOpenDevice(NULL); // select the "preferred device"
	if (mDevice) {
		// use the device to make a context
		mContext=alcCreateContext(mDevice,NULL);
		// set my context to the currently active one
		alcMakeContextCurrent(mContext);
	}
}
 

Pretty straight forward really. get the 'default' device. then use it to build a context! done.

Next: put data into a buffer, this is a bit more complicated:

First: you need to open the file in a nice audio-friendly way

 
// get the full path of the file
NSString* fileName = [[NSBundle mainBundle] pathForResource:@"neatoEffect" ofType:@"caf"];
// first, open the file
AudioFileID fileID = [self openAudioFile:fileName];
 

wait! what is that: openAudioFile: method?
here it is:

 
// open the audio file
// returns a big audio ID struct
-(AudioFileID)openAudioFile:(NSString*)filePath
{
	AudioFileID outAFID;
	// use the NSURl instead of a cfurlref cuz it is easier
	NSURL * afUrl = [NSURL fileURLWithPath:filePath];
 
	// do some platform specific stuff..
#if TARGET_OS_IPHONE
	OSStatus result = AudioFileOpenURL((CFURLRef)afUrl, kAudioFileReadPermission, 0, &outAFID);
#else
	OSStatus result = AudioFileOpenURL((CFURLRef)afUrl, fsRdPerm, 0, &outAFID);
#endif
	if (result != 0) NSLog(@"cannot openf file: %@",filePath);
	return outAFID;
}
 

this is pretty simple: we get the file path from the main bundle, then send it off to this handy method which checks the platform and uses the audio toolkit method: AudioFileOpenURL() to generate an AudioFileID.

What's next? Oh yes: get the actual audio data out of the file. To do this we need to figure out how much data is in the file:

 
// find out how big the actual audio data is
UInt32 fileSize = [self audioFileSize:fileID];
 

another handy method is needed:

 
// find the audio portion of the file
// return the size in bytes
-(UInt32)audioFileSize:(AudioFileID)fileDescriptor
{
	UInt64 outDataSize = 0;
	UInt32 thePropSize = sizeof(UInt64);
	OSStatus result = AudioFileGetProperty(fileDescriptor, kAudioFilePropertyAudioDataByteCount, &thePropSize, &outDataSize);
	if(result != 0) NSLog(@"cannot find file size");
	return (UInt32)outDataSize;
}
 

This uses the esoteric method: AudioFileGetProperty() to figure out how much sound data there is in the file and jams it into the outDataSize variable. groovy, next!

Now we are ready to copy the data from the file into an openAL buffer:

 
 
// this is where the audio data will live for the moment
unsigned char * outData = malloc(fileSize);
 
// this where we actually get the bytes from the file and put them
// into the data buffer
OSStatus result = noErr;
result = AudioFileReadBytes(fileID, false, 0, &fileSize, outData);
AudioFileClose(fileID); //close the file
 
if (result != 0) NSLog(@"cannot load effect: %@",fileName);
 
NSUInteger bufferID;
// grab a buffer ID from openAL
alGenBuffers(1, &bufferID);
 
// jam the audio data into the new buffer
alBufferData(bufferID,AL_FORMAT_STEREO16,outData,fileSize,44100); 
 
// save the buffer so I can release it later
[bufferStorageArray addObject:[NSNumber numberWithUnsignedInteger:bufferID]];
 

OK, lots went on here (well, not really). made some room for the data, used the AudioFileReadBytes() function from the audio toolkit to read the bytes from the file into the awaiting block of memory. The next bit is slightly more interesting. We call alGenBuffers() to make us a valid bufferID, then we call alBufferData() to load the awaiting data blob into the openAL buffer.

Here I have just hardcoded the format and the frequency. If you use the afconvert command at the top of the post to generate your audio files, then you will know what their format and sample rate are. However, if you want to be able to do any kind of audio format or frequency, then you will need to build some methods similar to audioFileSize: but using kAudioFilePropertyDataFormat to get the format, then convert it to the proper AL_FORMAT, and something even more byzantine to figure out the frequency. I am lazy so i just make sure my files are formatted properly.

Next I put the number into a nice NSArray for later reference. you can do with that ID whatever you want.

OK, now we have a buffer! neato. Time to hook it to the source.

 
NSUInteger sourceID;
 
// grab a source ID from openAL
alGenSources(1, &sourceID); 
 
// attach the buffer to the source
alSourcei(sourceID, AL_BUFFER, bufferID);
// set some basic source prefs
alSourcef(sourceID, AL_PITCH, 1.0f);
alSourcef(sourceID, AL_GAIN, 1.0f);
if (loops) alSourcei(sourceID, AL_LOOPING, AL_TRUE);
 
// store this for future use
[soundDictionary setObject:[NSNumber numberWithUnsignedInt:sourceID] forKey:@"neatoSound"];	
 
// clean up the buffer
if (outData)
{
	free(outData);
	outData = NULL;
}
 

Much like the buffer, we need to get a valid sourceID from openAL. Once we have that we can connect the source and the buffer. finally we will throw in a few basic buffer settings just to make sure it is all set up right. If we want it to loop, then we need to set the AL_LOOPING to true, if not, the default is not to loop, so ignore it. Then I store the ID into a nice dictionary do I can call it out by name.

lastly, clean up our temporary memory.

So close now! everything is all ready to go, now we just need to play the damn thing:

 
// the main method: grab the sound ID from the library
// and start the source playing
- (void)playSound:(NSString*)soundKey
{
	NSNumber * numVal = [soundDictionary objectForKey:soundKey];
	if (numVal == nil) return;
	NSUInteger sourceID = [numVal unsignedIntValue];
	alSourcePlay(sourceID);
}
 

that's it. alSourcePlay().. easy. If the sound doesnt loop, it will stop of it's own accord when it is all done. If it is looping, or you want to stop it early:

 
- (void)stopSound:(NSString*)soundKey
{
	NSNumber * numVal = [soundDictionary objectForKey:soundKey];
	if (numVal == nil) return;
	NSUInteger sourceID = [numVal unsignedIntValue];
	alSourceStop(sourceID);
}
 

That is basically the quickest and simplest way to get sound out of the iPhone using openAL. (that I can figure out anyway).

Lastly, when you are done with everything, be nice and clean up:

 
-(void)cleanUpOpenAL:(id)sender
{
	// delete the sources
	for (NSNumber * sourceNumber in [soundDictionary allValues]) {
		NSUInteger sourceID = [sourceNumber unsignedIntegerValue];
		alDeleteSources(1, &sourceID);
	}
	[soundDictionary removeAllObjects];
 
	// delete the buffers
	for (NSNumber * bufferNumber in bufferStorageArray) {
		NSUInteger bufferID = [bufferNumber unsignedIntegerValue];
		alDeleteBuffers(1, &bufferID);
	}
	[bufferStorageArray removeAllObjects];
 
	// destroy the context
	alcDestroyContext(mContext);
	// close the device
	alcCloseDevice(mDevice);
}
 

One note: in a real implementation you will probably have more than one source (I have a source for each buffer, but I only have about 8 sounds, so this is not a problem). There is an upper limit on the number of sources you can have. I dont know the actual number on the iphone, but it is probably something like 16 or 32. The way to deal with this is to load all your buffers, then dynamically assign those buffers the the next available source that isnt already playing something else.

Groovy, hopefully this will be helpful to someone. I had a bit of a hard time finding a good basic sample to get myself started so I made this one by going through the openAL programmers guide and just doing the very minimum.

Cheers!
-b

EDIT: clever reader Nathan points out that I forgot to include:
AudioFileClose(fileID);
in my sample code! Whoops! good catch, it is now fixed in the tutorial :-)

EDIT: this page continues to be the most visited page on my site, so: yay! Unfortunately, if you get here via google, then it can be hard to find other good articles about OpenAL on my site. So, to be servicey, if you read this article, you might also like:

• More OpenAL tidbits for iPhone
• restarting openAL after application interruption on the iPhone
• Lots and Lots of sounds in openAL
• alBufferDataStatic: why you should avoid it
• Streaming in OpenAL