Now that the NDA is lifted, and we can start talking about the iPhone code out in the open, i thought it might be nice to talk about some of the problems I have encountered in my forays into the iPhone world and how I went about fixing them.

Currently I am working on an iPhone game. It is all openGLES based and uses openAL for sound. I think I am gonna talk about openAL today.

For now I am only going to be talking about sounds that are less than 30 seconds, so sound effects and short loops. Before you can think about playing sounds on the iPhone they need to be in the right format (or they should be in the right format, many of the audio toolbox methods will handle multiple formats, but if you put the sound in the right format to start, then the iPhone wont have to do it at play time).

So, pop open terminal and type this:

/usr/bin/afconvert -f caff -d LEI16@44100 inputSoundFile.aiff outputSoundFile.caf

what the hell does that do? you ask. it puts the file into a nice Little-Endian 16-bit 44,100 sample rate format. (generally saved with a .caf extension)

OK! now we have a nice .caf file in the proper format, we are ready to do something.

There are lots of ways to play sound on the iPhone, there is the ‘easy’ way, and then there are a few ‘hard ways’.. I am gonna touch on the easy way quickly and then move onto the openAL ‘hard way’.

the quickest (and easiest) way to make the iPhone spit out some sound is to use the audio system services:

NSString* path = [[NSBundle mainBundle] pathForResource:@”soundEffect1″ ofType:@”caf”];
NSURL * afUrl = [NSURL fileURLWithPath:path];
UInt32 soundID;
AudioServicesCreateSystemSoundID((CFURLRef)afUrl,&soundID);
AudioServicesPlaySystemSound (soundID);

this works well for making your interface buttons click and simple UI interaction stuff. However, it is absolutely shite for anything more complicated than that (think: a game). It doest always play right away, and if you are trying to match up specific frame of your game with specific sound effects, then this method is basically useless. (I actually implemented my whole sound engine using the above style of code, then i got onto the phone and every time a sound played, it was either late by many frames or the whole thing would pause and wait for the audio toolbox to load the sound into the buffer, it sucked.

For better control of the sound, you will require either openAL or audioUnits or the audioQueue.

I decided to go with openAL so that my sound code could be kinda sorta portable, and by learning how to use openAL I would be able to use those skills on some other platform besides the iPhone. (and since I am a code-mercenary, i figured that having openAL experience was more marketable than audioQueue experience) (that and I already have familiarity with openGL, and openAL is very similar, and the audio units and audio queue code is kinda ugly)

So, this will be a super quick tutorial on openAL and the absolute bare minimum you need to do to accomplish static sound generated from openAL.

OpenAL is really quite straight forward. there are 3 main entities: the Listener, the Source, and the Buffer.

The Listener is you. Any sound the listener can ‘hear’ comes out the speakers. openAL allows you to specify where the listener is in relation to the sources, but for this example we dont care, we are going to bare minimum static sound, so just keep in mind that there is a concept of ‘listener’ and that you could move this object around if you wanted to do more complicated stuff, but I wont go into it in this post.

The Source: basically this is analogous to a speaker. it generates sound which the listener can ‘hear’. like the listener, you can move the sources around and get groovy positional effects. However, for this example we wont be doing that.

The buffer: basically this is the sound that will be played. the buffer holds the raw audio data.

there are two other very important objects: the device and the context.
the device is the actual bit of hardware that will be playing the sound, and the context is the current ’session’ that all these sounds are going to be played in (you can think of it as the room that all the sources and the listener is in. Or it is the air that the sound is played through, or whatever.. it is the context.)

How does this all work: (this is the bare minimum)

1) get the device
2) make a context with the device
3) put some data into a buffer
4) attach the buffer to a source
5) play the source

that is it! The above presumes that your implementation of openAL has decent defaults for the listener and if you dont specify any listener or source positions then this will all work dandy. (it works just dandy on the iPhone in any case)

so, lets look at some code:

// define these somewhere, like in your .h file
ALCcontext* mContext;
ALCdevice* mDevice;

// start up openAL
-(void)initOpenAL
{
	// Initialization
	mDevice = alcOpenDevice(NULL); // select the "preferred device"
	if (mDevice) {
		// use the device to make a context
		mContext=alcCreateContext(mDevice,NULL);
		// set my context to the currently active one
		alcMakeContextCurrent(mContext);
	}
}

Pretty straight forward really. get the ‘default’ device. then use it to build a context! done.

Next: put data into a buffer, this is a bit more complicated:

First: you need to open the file in a nice audio-friendly way

// get the full path of the file
NSString* fileName = [[NSBundle mainBundle] pathForResource:@”neatoEffect” ofType:@”caf”];
// first, open the file
AudioFileID fileID = [self openAudioFile:fileName];

wait! what is that: openAudioFile: method?
here it is:

// open the audio file
// returns a big audio ID struct
-(AudioFileID)openAudioFile:(NSString*)filePath
{
	AudioFileID outAFID;
	// use the NSURl instead of a cfurlref cuz it is easier
	NSURL * afUrl = [NSURL fileURLWithPath:filePath];

	// do some platform specific stuff..
#if TARGET_OS_IPHONE
	OSStatus result = AudioFileOpenURL((CFURLRef)afUrl, kAudioFileReadPermission, 0, &outAFID);
#else
	OSStatus result = AudioFileOpenURL((CFURLRef)afUrl, fsRdPerm, 0, &outAFID);
#endif
	if (result != 0) NSLog(@”cannot openf file: %@”,filePath);
	return outAFID;
}

this is pretty simple: we get the file path from the main bundle, then send it off to this handy method which checks the platform and uses the audio toolkit method: AudioFileOpenURL() to generate an AudioFileID.

What’s next? Oh yes: get the actual audio data out of the file. To do this we need to figure out how much data is in the file:

// find out how big the actual audio data is
UInt32 fileSize = [self audioFileSize:fileID];

another handy method is needed:

// find the audio portion of the file
// return the size in bytes
-(UInt32)audioFileSize:(AudioFileID)fileDescriptor
{
	UInt64 outDataSize = 0;
	UInt32 thePropSize = sizeof(UInt64);
	OSStatus result = AudioFileGetProperty(fileDescriptor, kAudioFilePropertyAudioDataByteCount, &thePropSize, &outDataSize);
	if(result != 0) NSLog(@"cannot find file size");
	return (UInt32)outDataSize;
}

This uses the esoteric method: AudioFileGetProperty() to figure out how much sound data there is in the file and jams it into the outDataSize variable. groovy, next!

Now we are ready to copy the data from the file into an openAL buffer:

// this is where the audio data will live for the moment
unsigned char * outData = malloc(fileSize);

// this where we actually get the bytes from the file and put them
// into the data buffer
OSStatus result = noErr;
result = AudioFileReadBytes(fileID, false, 0, &fileSize, outData);

if (result != 0) NSLog(@"cannot load effect: %@",fileName);

NSUInteger bufferID;
// grab a buffer ID from openAL
alGenBuffers(1, &bufferID);

// jam the audio data into the new buffer
alBufferData(bufferID,AL_FORMAT_STEREO16,outData,fileSize,44100); 

// save the buffer so I can release it later
[bufferStorageArray addObject:[NSNumber numberWithUnsignedInteger:bufferID]];

OK, lots went on here (well, not really). made some room for the data, used the AudioFileReadBytes() function from the audio toolkit to read the bytes from the file into the awaiting block of memory. The next bit is slightly more interesting. We call alGenBuffers() to make us a valid bufferID, then we call alBufferData() to load the awaiting data blob into the openAL buffer.

Here I have just hardcoded the format and the frequency. If you use the afconvert command at the top of the post to generate your audio files, then you will know what their format and sample rate are. However, if you want to be able to do any kind of audio format or frequency, then you will need to build some methods similar to audioFileSize: but using kAudioFilePropertyDataFormat to get the format, then convert it to the proper AL_FORMAT, and something even more byzantine to figure out the frequency. I am lazy so i just make sure my files are formatted properly.

Next I put the number into a nice NSArray for later reference. you can do with that ID whatever you want.

OK, now we have a buffer! neato. Time to hook it to the source.

NSUInteger sourceID;

// grab a source ID from openAL
alGenSources(1, &sourceID); 

// attach the buffer to the source
alSourcei(sourceID, AL_BUFFER, bufferID);
// set some basic source prefs
alSourcef(sourceID, AL_PITCH, 1.0f);
alSourcef(sourceID, AL_GAIN, 1.0f);
if (loops) alSourcei(sourceID, AL_LOOPING, AL_TRUE);

// store this for future use
[soundDictionary setObject:[NSNumber numberWithUnsignedInt:sourceID] forKey:@”neatoSound”];	

// clean up the buffer
if (outData)
{
	free(outData);
	outData = NULL;
}

Much like the buffer, we need to get a valid sourceID from openAL. Once we have that we can connect the source and the buffer. finally we will throw in a few basic buffer settings just to make sure it is all set up right. If we want it to loop, then we need to set the AL_LOOPING to true, if not, the default is not to loop, so ignore it. Then I store the ID into a nice dictionary do I can call it out by name.

lastly, clean up our temporary memory.

So close now! everything is all ready to go, now we just need to play the damn thing:

// the main method: grab the sound ID from the library
// and start the source playing
- (void)playSound:(NSString*)soundKey
{
	NSNumber * numVal = [soundDictionary objectForKey:soundKey];
	if (numVal == nil) return;
	NSUInteger sourceID = [numVal unsignedIntValue];
	alSourcePlay(sourceID);
}

that’s it. alSourcePlay().. easy. If the sound doesnt loop, it will stop of it’s own accord when it is all done. If it is looping, or you want to stop it early:

- (void)stopSound:(NSString*)soundKey
{
	NSNumber * numVal = [soundDictionary objectForKey:soundKey];
	if (numVal == nil) return;
	NSUInteger sourceID = [numVal unsignedIntValue];
	alSourceStop(sourceID);
}

That is basically the quickest and simplest way to get sound out of the iPhone using openAL. (that I can figure out anyway).

Lastly, when you are done with everything, be nice and clean up:

-(void)cleanUpOpenAL:(id)sender
{
	// delete the sources
	for (NSNumber * sourceNumber in [soundDictionary allValues]) {
		NSUInteger sourceID = [sourceNumber unsignedIntegerValue];
		alDeleteSources(1, &sourceID);
	}
	[soundDictionary removeAllObjects];

	// delete the buffers
	for (NSNumber * bufferNumber in bufferStorageArray) {
		NSUInteger bufferID = [bufferNumber unsignedIntegerValue];
		alDeleteBuffers(1, &bufferID);
	}
	[bufferStorageArray removeAllObjects];

	// destroy the context
	alcDestroyContext(mContext);
	// close the device
	alcCloseDevice(mDevice);
}

One note: in a real implementation you will probably have more than one source (I have a source for each buffer, but I only have about 8 sounds, so this is not a problem). There is an upper limit on the number of sources you can have. I dont know the actual number on the iphone, but it is probably something like 16 or 32. The way to deal with this is to load all your buffers, then dynamically assign those buffers the the next available source that isnt already playing something else.

Groovy, hopefully this will be helpful to someone. I had a bit of a hard time finding a good basic sample to get myself started so I made this one by going through the openAL programmers guide and just doing the very minimum.

Cheers!
-b

update your stuff to r158. (there is a new binary on the opentouch code site)

NOTE: Don’t forget that BBTouch requires openCV.framework now. so be sure to grab that as well and stick it into your ~/Library/Frameworks folder. (you may have to make this folder).. It is actually a private framework, and I have been playing around with trying to get it to be part of the BBTouch codebase, but googlecode is not playing nice with it (it claims to be locked, which is possible since I got it from someone else who had compiled it. So anyway, i haven’t had the time to go in and see why SVN thinks it is locked and so it is still not technically part of BBTouch, so you have to install it manually, my apologies)

But! if you do not install it, then BBTouch will brash on load. so if you re having crash on load issues, start with openCV. (BTW I uploaded the version that I am testing against to the opentouch downloads page, so you can get it there. (the link to the right that says “Fairly recent binaries”).

Anyhow, I am in the process of trying to find the time to get around to thinking about updating the documentation to include the above warning about openCV as well as all the new filtery stuff. But I havent had the chance, so tell your friends.

Anyhow this is all running on my prototype table which is a diffused illumination setup, the tracker is BBTouch and it is all using TUIO to communicate. The tech is all core animation using CALayers. there is lots of mipmapping going on to keep the performance up even when there are tons of images, and it works quite nicely even on my macbook pro. BBtouch never tops 50% processor and xPrexxo is similar.

Some of you may be aware of the monome project: http://monome.org/ I think these guys have a great deal in common with the Multi-touch community :-)

Anyhow, there is a cool iphone port of the monome stuff called Haplome: http://toddtreece.com/haplome/ it allows you to pretend you have a monome button array on your iPhone. fun stuff.

Anyhow, the reason this is topical to my lame site is that he is using BBOSC to help talk to the other OSC apps thet can play nice with the monome stuff. So there you go!

Thanks Todd for a cool project!

Here it is:
http://code.google.com/p/bbtuio/

the most recent BBTUIOTest is there as I mentioned, and a binary of the same is also on the downloads page.

It is pretty straightforward to use, just load it up, set the port and hit ’start listening’. If you want to run it full screen, then just pick a screen and hit ‘go fullscreen’. to get out of fullscreen either hit ’stop fullscreen’ (if you can see the original window) or make sure the fullscreen window is in front ( by clicking it with the mouse ) and hit ‘ESC’. Quitting the app will also get you out of fullscreen.

the ‘fake mouse events’ tries to use the first recognized TUIO cursor as a mouse. If it ‘locks up’ while faking mouse events then it probably fakes a mouse down and then never faked a mouse-up. if you simply wait long enough this should resolve itself, but if you are in a hurry; command - option -esc will open the force-quit dialog. The act of opening this dialog will break you out of any mouse events that the TUIOTest got you into. (you dont actually need to force-quit anything)

Groovy!
-b

I just commited a few more files, bringing BBTouch up to r157. I had been running 152 all week with no trouble, but Sandor managed to find a handful of bugs (of course, right when he is trying to use it at a show :-) so I made some changes and hopefully it is a bit better.

The biggest changes between 152 and 157 are that I fixed some very minor UI issues where the TUIO settings fields were not being properly disabled so you could change them out from under the OSC objects (causing some odd behavior). But the big one was that in some circumstances BBTouch would boot up and the filter settings (and the dark blobs pref) would be basically disconnected from the detector objects. no matter how much you changed them they didnt really affect anything. In any case, with 157 that is all fixed.

That is all for now :-)

I should be posting the newest BBTUIOTest.app soon, once I get it all back together.

cheers!
-b

Anyhow, r152 is up now with working TUIO again. (like, 3 lines had to change, i suck!)

here is the new binary:
bbtouchr152app

I just checked in a big chunk of new code and a new interface for BBTouch. (now up to r151)

I moved some of the interface stuff around, I put the TUIO stuff and the Screen Selector into sheets to unclutter the main interface a bit. I also added a button to open the filter config window. as well as the new config library :-) more on that in a sec.

I added a whole new configuration control system. Now you can save and load configurations on the fly very easily through the config library.

It is pretty straight forward: just get all the settings to where you like them, then open the library, hit “+” and it will add a new configuration (double click to edit the name). each configuration holds every value on the interface as well as both the camera mesh and the projection mesh. So now you can configure all the various surfaces/situations you might need and just switch them easily on the fly.

Thanks again to Sandor for funding all these new changes (as well as the other code that I am working on that will be posted next week; a few demo apps like a cocoa based lightbox (all TUIO of course, so you can use it even if you dont want to use BBTouch). Also, Sandor and I are rushing to get these apps working by Monday, so if you want to help out, load up bbtouch and break the hell out of it and let me know what bugs you find.

Here is a binary of r151:
bbtouchr151app (check more recent posts for newer binaries)

also the code is up on the repo, link in the sidebar to the right.

I will probably be changing the BBTouch code a few times this week, so check back for the most recent stuff. (i will always commit the code to the google repo too, so that will always be updated)

Cheers!
-b

I just finished committing the last of the new filter code to the repository. (r150) No binary yet (I am at a conference, waiting for the keynote to start, so no time now :-) so you will have to compile your own. You will also need the openCV.framework
(found here: http://www.ubaa.net/shared/processing/opencv/)

(I plan to add it properly to the project as a private framework, but didnt do it when i added it first for some reason, and now I keep forgetting)

anyhow, the only thing to know really is that in the View Menu, there is a new option : Filter Config… This will allow you to adjust the filter parameters for your setup. It is kinda optimized for DI stuff right now (because that is what I have) but I will add more FTIR friendly filters later (if someone tells me what they need) Oh, and dont put even numbers (or zero) in for the kernel values, or it will crash. I need to put a constraint on that, but haven’t got to it. You can put any number in for the Threshold Mean Offset tho. (negative numbers usually work best)

Let me know any bugs you find!

CHeers!
-b

Here is a youtube video (my first :-) of the initial testing of the newer/better/more filtery-er BBTouch along with the BBTUIOTest app (now with more lightboxyness).

Apologies for the supreme craptacularness of the video quality. I don’t have a camcorder, so I just pointed my laptop’s built-in iSight at the table and used iMovie to capture. (while also running BBTouch in the background)

Anyhow, the BBTouch setup was like so:

BBTouch tracking, generating TUIO events, sending them via wifi to yet another machine running BBTUIOTest which is running the projector, and is running the lightbox code.

I know what you are all thinking: since when does BBTUIOTest have a lightbox feature?? and the answer to that would be since a few days ago when I added it. (no code up yet, it is still pretty raw)

I think it is also important to note that all this development comes with much thanks from Sandor’s development budget. I have been working with him on an MT app that he designed (more on that later :-) and now we are moving to the next step and getting BBTouch and some nice (yet to be made :-) cocoa/core image demo apps up to snuff so that he can use them for his own nefarious purposes :-) So! Everyone needs to buy Sandor a beer and thank him for his generous funding of BBTouch and Multitouch code development, and fastracking BBTouch improvements that I may never have gotten to otherwise! (thus saving me at least temporarily from some other, far less interesting jobs :-)

SO! BBTouch, now with more filters: It is still in a very raw state, and the code wont be stable for a few more days at the very least, but here is a sneak peek:

I know, everyone is wondering why I didnt add filtering sooner. Well, it is like this: I like to keep things as simple and minimal as possible (especially things that are generally very complex, like MT stuff). And before, when BBTouch was merely a pet project and I could optimize the blob detection algorithm to my heart’s content, that is what i did. I think it paid off in the end, because i think that just the blob detector pre new filtery goodness was about 85% there. All it was doing was a background subtraction and a simple threshold, and I got really good coverage across about 85% of my surface. However, I was getting the bane of all DI surfaces: false hits from you palm or curled fingers on the light areas and no hits for any blobs in the dark areas. These filters help with that immensely.

Now I have added basically two more filters: a High Pass and an Adaptive Threshold. (and a noise reducer for the highpass, so I guess technically 3 filters.) (all using the openCV image filters. more on that later too)

The High pass filter made all the difference. For anyone doing DI type setups, make sure that you are using a higpass, it will make your life so much easier.

Anyhow, enough for now, back to work for me!

Cheers!
-b