People were sitting on the floor and the room was about 35 degrees (that is 35 degrees C, for those of you in the states, that is approximately a million degrees fahrenheit). But all in all it was pretty awesome.

For anyone just tuning in, the talk started out with a quick tour of the unity interface, then I basically dove right into building a game prototype from scratch, trying to get it done in just over an hour. It was a bit of a frenzy of code and components and general Unity awesomeness. Well, I ended up only getting through the creation of a plane that you could fly around and shoot stuff and crash into things and exploding. That is as far as I got :-) I did not make ti to enemy turrets, or procedural city generation. However, as I mentioned earlier, there were so many good questions that I probably covered more actual material than I had originally planned.

(This is the prototype game using the frogames buildings, roads, textures and the plane)

Anyhow. Since the talk was all about prototyping, I was using the totally fantastic and incredably inexpensive protopack from frogames. These guys have a handful of unity compatible content packs that are absolutely worth every cent in terms of time saved and making it so easy to just grab a few prefabs, throw them into a scene attach a few scripts and you have your game idea outlined and ready to tweak.

However, I want to be able to give away the project, but I cannot give away the protopack… sooo I went in and stripped out the proto pack stuff and replaced it with primitives. It doesnt look as nice, but it still does the job, and you can get to the scripts and have a play with all the stuff I talked about today.

(this is the non-awesome looking primitives-only version of the project, which you can download)

So anyway, here is the Freeplay project sans protopack stuff.

SO: here is my advice: go and buy the protopack, it is only like $15 or something stupidly cheap. Then go and build your prototype game.

Let me know if you have questions or whatever!

Cheers!
-B

If you havent checked them out and enjoy some choose-your-own-adventure style fantasy games, then definitely do so. However, that is not what this post is about. For the books, we have an intro bumper video that plays the first time you open the app. Our first book: Assassin in Orlandes had a landscape oriented menu screen and so we made the video a normal landscape type video. However, the book reading bit was portrait orientation. In our in-house testing we didnt go back and forth between the two orientations all that often, but (as is often the case) the users used the app differently and were constantly going back and forth and so the orientation switch was a bit annoying.
For the second book: The Siege of the Necromancer, we switched to a fully portrait oriented app, with the exception of the intro video (it was already made when we decided to change the orientation, and being a poor indie studio we didnt have the cash to re-do the animations in a vertical aspect. However, most people watch the bumper once or twice then never see it again, so that is OK. )

Now with book three, Slaves of Rema, we are finally moving to a fully portrait oriented video. This poses some interesting problems.

Since I am the developer and we are indies, I am also the video codec guy, and often the video editor as well. The animations are outsourced and the other GFX are done in house, but it falls to me to put it all together and get it ready for the iPhone.

What is the problem you ask? Well, on the iPhone you can only play video in landscape and you can really only play landscape formatted videos. On the iPad you can play videos in portrait, but they get all scrunched down. If you want full screen vertical oriented videos then you really need to make your videos sideways.

This can be a bit of a pain to do. One way to do it is to take all of you grafics and rotate them before inserting them into final cut or similar. This means that you also need to tell the guy doing the animations that you need them to be done sideways and all of that. This is a fine way to go about it, but it can make working on the video a pain in the ass. Who wants to work on a sideways video?

Instead you can make the video upright and rotate it after you have done all the editing. I find this to be the best for our workflow. That way all the graphics guys and animation guys are all working with the right orientation in mind and everything looks good. The hard bit is editing a video that is 320 x 480. Final cut, which I love, is total shite at doing anything that is not meant for TV or HD resolutions. What I use is a combination of Screenflow and Quicktime pro.

wait what?

Screenflow is a screencasting program, why would you use it for editing? Well, mostly because I already use it for screencasting, so I have it laying around. And it does a fine job of doing basic editing. Most importantly it allows you to export in any crazy-ass aspect ratio you want, and the export options are excellent. This allows me to edit the video in the aspect it will be seen at, ie portrait.

So, what I do is edit my video in Screenflow, (in the case of the new books these videos are at 640 x 960, so I can do high rez versions for the iPhone4 and the iPad, but it all works the same for 320 x 480, just be sure your final export size is the size you are shooting for)

I export the video at full rez and lossless from Screenflow (make sure that your rez is 320 x 480 or 640 x 960).

next thing is to open it in Quicktime Pro. (note, the quicktime that comes with your mac these days is not qt pro, it is just qt player. QT pro is $30 ish and is an indispensable tool to have in your arsenal if you do any amount of video work. (if you have final cut or FCX, you should already have QT pro).

Once in QT Pro, go to window->show movie properties. Select the video track, and look at the Visual Settings tab.

All you need to do here is hit one of the rotate buttons. I generally thing that rotating left (the rightmost button) is the way to go. That way the home button on the phone will be facing downward when the phone is is standard portrait orientation and the video is playing.

At this point your video should look like the one above that is tipped sideways.

Now for the fun part, the compression.

This may take a little while, and if you have a slow machine it may take a long while. The idea here is that you will probably need to try a half dozen different settings before you get one that looks good and is the size you want.

First off, see if the built-in settings work OK for your video:

Choose File->Export. Then select the Movie to iPhone setting and save it off. This will generate an .m4v file.

Once it is done compressing, open it back up and watch it. If your video still looks fine then you are done. This seems to wrok well for about 90% of the videos I run through it.
However I find that the default ‘iPhone’ settings sometimes produce terrible video artifacting for certain types of video so I have to do it the hard way. (also, if you are trying to really squeeze every last meg out of your app packages, then the you can get similar quality with half (or less) the size by tweaking the settings yourself.

In this case you need to change the export options and select ‘Movie to MPEG-4′

Have a look at all the fields on the MPEG-4 options.

Make sure that the video size is set to what you want, make sure the video format is set to H.264. After that the most important thing on this screen is the Data Rate. but before we get to that, hit the ‘Video Options’ button, and select baseline.

NOTE: if you dont do this, the video will simply not play on your device, not at all.

Ok, back the the main MPEG-4 screen and the data rate. This is where you will want to change the value and make a few different videos until you are happy with the quality and the size. I generally start at 1200 or 1500 kbits/sec (this usually produces a fairly nice quality video, but it is generally pretty beefy in terms of size.) then I go down by 250 until the quality degrades to a point where I would never use it. From there you should have a handful fo videos fo different qualities/sizes to choose from. You can tweak the data rate down as accurate as you want. If the video quality is almost good enough, then just bump up the data rate by 25 or 50 and see what you get.

cheers!
-Ben

Here is how this works:

Step 1: write an original article on your blog about a useful thing (like OpenAL) (we will call this the ‘original post’ and the ‘original blogger’)
Step 2: someone else likes your article and cuts and pastes it into their blog, they change a few words here and there, but mostly it is left identical (errors and everything) (let’s call him ‘copy blogger’, and we will call this the ‘copy post’)
Step 3: wait a while
Step 4: somebody notices that ‘copy post’ is very similar to ‘original post’ and tells ‘Copy Blogger’ that someone has copied his post! Mon Dieu!
Step 5: Copy Blogger emails Original Blogger and tells him that if he is going to be copying his (plagiarised) posts then Original Blogger should have the common courtesy to provide a linkback to the ‘Copy post’ from the ‘Original post’!
Step 6: Original Blogger laughs and laughs.

If this is all too abstract, then I will simplify it: I wrote a post about OpenAL on the iPhone in 2008, right after the NDA was lifted. This single article is in the top 5 for traffic for my site. Lots of people have seen it. Presumably there are lots of OpenAL codebases out there that have snippets of code from that article, which is great!

That article has a bunch of code snippets and instructions on how to build your own simple OpenAL sound player. (but does not provide a working version, you have to do that yourself :-)

Someone (whom shall remain nameless) took that article and copied it into his website, and claimed authorship of it. This was in early 2009. Now to his credit, he did change a few words here and there, and he took the code snippets and put them into a single file and filled in the gaps. This is exactly what I had hoped people would do! (with the exception of claiming authorship of my words, that is kinda dickish)

I actually noticed that this had happened and made a fleeting mention of it at the top of my post about ‘lots of sounds’ in OpenAL. But to be honest, I didnt really care all that much about it. Hey, it’s the internet, people steal anything that isn’t bolted down, and they sometimes steal that stuff too.

However, this morning, I get an email from our random internet plagiarist telling me: (and I quote) “just want to notice that someone told me it seems your article has alot in common with my own article postet on [RADACTED]. If your article is related to that, you should post a linkback or something like that. ”

He wasn’t a dick about it, he was pretty cool. He is probably a stand-up kinda guy.

To be honest, it has been a looooong time since I noticed that he had copied me, and I had basically forgotten about it. I dutifully went to his site and had a look (because I was curious, and had forgotten about the whole thing, and frankly OpenAL isn’t that complicated, and there are only so many ways to do it, so really most articles on OpenAL on the iphone could be considered ‘similar’ on many ways, and if his site was good, then I would link to it anyway) and when I saw my own words staring back at me I remembered the whole thing from last year and I laughed and laughed.

Then I sent him a kinda shitty reply, sorry about that internet plagiarist dude, I probably should have waited till I had breakfast before replying to your email.

At the end of the day, I post stuff here so that people hopefully get something out of it. If you want to copy all the code here and all the txt here and post it on your site, well, that is perfectly possible, and not illegal. But taking ownership of my words is a bit of a douchebag move. At least have the common courtesy to re-word it.

Cheers!
-Ben

• a basic intro to OpenAL on the iPhone
• A rant about OpenAL and sample code
• how to properly handle interruptions in openAL on the iPhone
• how to play lots of sounds all at once in OpenAL

One thing that people keep asking me is how to playback gigantic files in OpenAL. Things like songs or audio books. (This post has been slowly growing over the past few months and I am finally going to put the final bits in and post it!)

The answer to this problem (in case you didn’t read the title of the post) is streaming.

First off, lets define what I really mean when I say streaming. In all of the above code samples, the sound buffers were fully resident in memory. This works great for smaller sounds like button click sounds and swords clanging sounds for games and whatnot. However, for big files like full quality songs and really anything over about a dozen seconds long, putting it all into memory (especially on the iPhone) becomes a bit of a problem.

To solve this we read the data off of the disk in chunks, filling small buffers one at a time and then playing them back one after the other. All the while, in the background we are removing the old buffers and replacing them with new buffers so that we only ever have a few seconds of sound in memory at one time.

Here is a crappy diagram that I spent way too much time on, and it is still crappy:

OK, at the top we have the ‘standard’ way to do thing (the way described in all those posts linked above). Basically:

• Step1: Load the sound data from a file into an OpenAL Buffer
• Step2: Connect the OpenAL Buffer to an OpenAL source with something like: alSourcei(sourceID, AL_BUFFER, bufferID);
• step 3: Start source playing with with alSourcePlay(sourceID);

Pretty simple.

Streaming is a bit more complicated, but still pretty simple (in concept). The first three steps are really very similar to the ‘standard’ way above.

• Step A: Load in a few chunks of the big file into all the waiting OpenAL buffers.
• Step B: Queue up all the newly filled buffers into a single OpenAL source via alSourceQueueBuffers(sourceID, 1, &bufferID);
• Step C: Start the source playing with alSourcePlay(sourceID); This will begin to consume the queued up buffers.
• Step D: In a background thread, check to see if there are any used buffers, and if so then load the next chunk of the sound file into the used buffer
• Step E: Queue up this newly filled buffer to the source with alSourceQueueBuffers(sourceID, 1, &bufferID); As long as there are still bufferes queued up, the source will continue to play them

So there it is, five easy steps. Lets look at them one at a time:

Step A: Load in a few chunks of the big file into all the waiting OpenAL buffers.

Ok first thing we need to do is to prepare a place for all this state data to live. This will be kinda like preloading the streaming file, execept that we dont load any sound into memory just yet, we are just going to get ready to do that.

// this queues up the specified file for streaming
-(NSMutableDictionary*)initializeStreamFromFile:(NSString*)fileName format:(ALenum)format freq:(ALsizei)freq
{
	// first, open the file
	AudioFileID fileID = [self _openAudioFile:fileName];

	// find out how big the actual audio data is
	UInt32 fileSize = [self _audioFileSize:fileID];

First thing, we open the file. This doesnt load it into memory, it just gives us a handle to some data that we need. Specifically the file size. We will need this to calculate the number of buffers we need to play the whole file.

	UInt32 bufferSize = OPENAL_STREAMING_BUFFER_SIZE;
	UInt32 bufferIndex = 0;

	// ok, now we build a data record for this streaming file
	// before, with straight sounds this is just a soundID
	// but with the streaming sound, we need more info
	NSMutableDictionary * record = [NSMutableDictionary dictionary];
	[record setObject:fileName forKey:@"fileName"];
	[record setObject:[NSNumber numberWithUnsignedInteger:fileSize] forKey:@"fileSize"];
	[record setObject:[NSNumber numberWithUnsignedInteger:bufferSize] forKey:@"bufferSize"];
	[record setObject:[NSNumber numberWithUnsignedInteger:bufferIndex] forKey:@"bufferIndex"];
	[record setObject:[NSNumber numberWithInteger:format] forKey:@"format"];
	[record setObject:[NSNumber numberWithInteger:freq] forKey:@"freq"];
	[record setObject:[NSNumber numberWithBool:NO] forKey:@"isPlaying"];

Next we are going to make a streaming sound record. This dictionary will hold all of the state for this stream.

The OPENAL_STREAMING_BUFFER_SIZE is how many bytes of data we want in each buffer chunk. I have had good luck with 48000, this is about a second at high quality so I have plenty of time to grab the next chunk.

The bufferIndex is the current buffer that is full. For now that is zero of course.

Format and Freq we will use when we eventually setup our source. and isPlaying will let us know if we need to be refilling the buffers or not.

	// this will hold our buffer IDs
	NSMutableArray * bufferList = [NSMutableArray array];
	int i;
	for (i = 0; i < 3; i++) {
		NSUInteger bufferID;
		// grab a buffer ID from openAL
		alGenBuffers(1, &bufferID);

		[bufferList addObject:[NSNumber numberWithUnsignedInteger:bufferID]];
	}	

	[record setObject:bufferList forKey:@"bufferList"];

This is where the actual buffer references will live. In theory you can get away with just two buffers. One to play and one to fill while the other is playing. However If you get any stitch in your background thread, or do any heavy lifting on the processor that delays that refill, then you will get some nasty skipping. Instead I tend to use three buffers. This takes up more memory, but affords me a bit more robust playback.

	// close the file
	AudioFileClose(fileID);

	return record;
}

Finally close our audio file and return the record. Take that record returned from the above method and store it into a big NSMutableDictionary called soundLibrary with some key, like the name of the sound. (or in whatever data structure you want, that is how I do it, and that is how the sample code works)

We are still in Step A (well, pre step A even). We haven't loaded any buffers, but we are ready when that time comes.

Once you are ready to play your sound, we move to:

Steps A, B and C in one fell swoop

OK, lets define a play streaming sound method that does A, B and C for us. (and kick off Step D)

We would call this method when we want to actually begin playing the streaming sound, the key is whatever key you used to store the record into the sound library.

This method returns the sourceID so that the calling object can use it to stop the sound.

- (NSUInteger)playStream:(NSString*)soundKey gain:(ALfloat)gain pitch:(ALfloat)pitch loops:(BOOL)loops
{
	// if we are not active, then dont do anything
	if (!active) return 0;

	ALenum err = alGetError(); // clear error code

Just some housekeeping. I have a big boolean called 'active' that shuts off all sounds. Makes it nice and easy. Second we clear out the OpenAL errors so that anything in there will be a result of what we do in this method.

	// generally the 'play sound method' whoudl be called for all sounds
	// however if someone did call this one in error, it is nice to be able to handle it
	if ([[soundLibrary objectForKey:soundKey] isKindOfClass:[NSNumber class]]) {
		return [self playSound:soundKey gain:1.0 pitch:1.0 loops:loops];
	}

If you followed along on the other tutorials you would know that previously I had simply stored the NSNumber value for the buffer in the soundLibrary. This is a way to handle that if the streaming method gets called with the wrong key (ie a non-streaming sound) It just punts to the standard playSound method. Similarly in my playsound method I check to see if the record is an NSDictionary, and if so then it calls this method. That way you can just use the playSounds method with either streaming sounds or regular sounds and it all works dandy.

	// get our keyed sound record
	NSMutableDictionary * record = [soundLibrary objectForKey:soundKey];

	// first off, check to see if this sound is already playing
	if ([[record objectForKey:@"isPlaying"] boolValue]) return 0;

Ok, we grab the record and start to go through the state. If the sound is already playing then we get out early.

	// first, find the buffer we want to play
	NSArray * bufferList = [record objectForKey:@"bufferList"];

	// now find an available source
	NSUInteger sourceID = [self nextAvailableSource];
	alSourcei(sourceID, AL_BUFFER, 0);

	// reset the buffer index to 0
	[record setObject:[NSNumber numberWithUnsignedInteger:0] forKey:@"bufferIndex"];

Now we are going to move into Step A proper. Grab the buffer list, and get an available source. the Method nextAvailableSource simply goes through a big list of premade sources and finds one that is not being used currrently. I think I went over that in the 'lots of sounds' tutorial linked above.

Then we reset the bufferindex to 0. This is basically setting the playhead to the beginning of the sound. next, we fill the buffers

	// queue up the first 3 buffers on the source
	for (NSNumber * bufferNumber in bufferList) {
		NSUInteger bufferID = [bufferNumber unsignedIntegerValue];
		[self loadNextStreamingBufferForSound:soundKey intoBuffer:bufferID];
		alSourceQueueBuffers(sourceID, 1, &bufferID);
		err = alGetError();
		if (err != 0) [self _error:err note:@"Error alSourceQueueBuffers!"];
	}

Ok, this is pretty simple looking but there is the one magic method: loadNextStreamingBufferForSound: intoBuffer: I will get to this in a minute, but basically it grabs a chunk of the audio file based on the bufferIndex and loads it into the buffer. then it increments the bufferIndex so that the next time I call this method I will get the next chunk.
We load a chunk into every buffer in the buffer list (which in our case will be three buffers)
And here is the important part: (this would be the Step B part of the diagram)

alSourceQueueBuffers(sourceID, 1, &bufferID);

This is the magic OpenAL function call that makes this source a streaming source instead of a single buffer source. Basically it will continue to play as long as there are buffers queued up.

	// set the pitch and gain of the source
	alSourcef(sourceID, AL_PITCH, pitch);
	err = alGetError();
	if (err != 0) [self _error:err note:@"Error AL_PITCH!"];
	alSourcef(sourceID, AL_GAIN, gain);
	err = alGetError();
	if (err != 0) [self _error:err note:@"Error AL_GAIN!"];
	// streams should not be looping
	// we will handle that in the buffer refill code
	alSourcei(sourceID, AL_LOOPING, AL_FALSE);
	err = alGetError();
	if (err != 0) [self _error:err note:@"Error AL_LOOPING!"];

With our buffers loaded and queued on the source, we just need to set up the source with all the properties that were passed in. This is exactly like you would do it for a single buffer sound.

	// everything is queued, start the buffer playing
	alSourcePlay(sourceID);
	// check to see if there are any errors
	err = alGetError();
	if (err != 0) {
		[self _error:err note:@"Error Playing Stream!"];
		return 0;
	}

Ok, finally we move to Step C: and we start the source playing. From this point on, we are on the clock to keep the buffers filled up.

	// set up some state
	[record setObject:[NSNumber numberWithBool:YES] forKey:@"isPlaying"];
	[record setObject:[NSNumber numberWithBool:loops] forKey:@"loops"];
	[record setObject:[NSNumber numberWithUnsignedInteger:sourceID] forKey:@"sourceID"];

	// kick off the refill methods
	[NSThread detachNewThreadSelector:@selector(rotateBufferThread:) toTarget:self withObject:soundKey];
	return sourceID;
}

This last bit sets up the state we need to keep the buffers full, and kicks off a new thread to run in the background to keep our buffers full.

OK, before we move onto Step D lets have a look at our buffer loader method

loadNextStreamingBufferForSound: intoBuffer:

This is roughly equivalent to the method that you would use to load an entire file into a buffer for standard sound playback, only we are only going to be grabbing a small bit of the file. Luckily for us this is a pretty common thing you would want to do, so mostly all we have to worry about is keeping the state set properly.

// this takes the stream record, figures out where we are in the file
// and loads the next chunk into the specified buffer
-(BOOL)loadNextStreamingBufferForSound:(NSString*)key intoBuffer:(NSUInteger)bufferID
{
	// check some escape conditions
	if ([soundLibrary objectForKey:key] == nil) return NO;
	if (![[soundLibrary objectForKey:key] isKindOfClass:[NSDictionary class]]) return NO;

First off just some simple checks to make sure I am not trying to load a non-existent sound file, or a non-streaming file.

	// get the record
	NSMutableDictionary * record = [soundLibrary objectForKey:key];

	// open the file
	AudioFileID fileID = [self _openAudioFile:[record objectForKey:@"fileName"]];

	// now we need to calculate where we are in the file
	UInt32 fileSize = [[record objectForKey:@"fileSize"] unsignedIntegerValue];
	UInt32 bufferSize = [[record objectForKey:@"bufferSize"] unsignedIntegerValue];
	UInt32 bufferIndex = [[record objectForKey:@"bufferIndex"] unsignedIntegerValue];;

Grab the record that has all of our state information, and set up all of our variables.

	// how many chunks does the file have total?
	NSInteger totalChunks = fileSize/bufferSize;

	// are we past the end? if so get out
	if (bufferIndex > totalChunks) return NO;

	// this is where we need to start reading from the file
	NSUInteger startOffset = bufferIndex * bufferSize;

	// are we in the last chunk? it might not be the same size as all the others
	if (bufferIndex == totalChunks) {
		NSInteger leftOverBytes = fileSize - (bufferSize * totalChunks);
		bufferSize = leftOverBytes;
	}

Here we are just using our state info to figure out where in the file to look and how big of a chunk to take. If we are at the last chunk, then it may not be a full sized chunk, so we have to take that into account and change our data size.

	// this is where the audio data will live for the moment
	unsigned char * outData = malloc(bufferSize);

	// this where we actually get the bytes from the file and put them
	// into the data buffer
	UInt32 bytesToRead = bufferSize;
	OSStatus result = noErr;
	result = AudioFileReadBytes(fileID, false, startOffset, &bytesToRead, outData);
	if (result != 0) NSLog(@"cannot load stream: %@",[record objectForKey:@"fileName"]);

	// if we are past the end, and no bytes were read, then no need to Q a buffer
        // this should not happen if the math above is correct, but to be sae we will add it
	if (bytesToRead == 0) {
            free(outData);
            return NO; // no more file!
        }

Ok, here we do the actual meat of the method. We alloc some memory, and then use the AudioFileReadBytes() function to grab our desired slice of data from the big file. This loads our chunk of sound data into the outData memory. At this point we will proceed exactly like we would with a single-buffer sound.

	ALsizei freq = [[record objectForKey:@"freq"] intValue];
	ALenum format = [[record objectForKey:@"format"] intValue];

	// jam the audio data into the supplied buffer
	alBufferData(bufferID,format,outData,bytesToRead,freq);

Load out sound into our OpenAL buffer. easy.

	// clean up the buffer
	if (outData)
	{
		free(outData);
		outData = NULL;
	}

	AudioFileClose(fileID);

Do some cleanup.

	// increment the index so that next time we get the next chunk
	bufferIndex ++;
	// are we looping? if so then flip back to 0
	if ((bufferIndex > totalChunks) && ([[record objectForKey:@"loops"] boolValue])) {
		bufferIndex = 0;
	}
	[record setObject:[NSNumber numberWithUnsignedInteger:bufferIndex] forKey:@"bufferIndex"];
	return YES;
}

Finally we increment the bufferIndex so that the net time we call this method we get the next chunk of data in the sequence. If we are looping we reset the index to 0 at the end.

So, that is steps A, B, C, and the beginning of D.

Lets look more closely at our background thread now.

Step D (and E): The background thread to refill our buffers

OK, you may recall, like ten pages ago, that we kicked off a thread to refill the buffers in the background. Lets look at that:

-(void)rotateBufferThread:(NSString*)soundKey
{
	NSAutoreleasePool * apool = [[NSAutoreleasePool alloc] init];
	BOOL stillPlaying = YES;
	while (stillPlaying) {
		stillPlaying = [self rotateBufferForStreamingSound:soundKey];
		if (interrupted) 	{
			// slow down our thread during interruptions
			[NSThread sleepForTimeInterval:kBufferRefreshDelay * 3];
		} else {
			// normal thread delay
			[NSThread sleepForTimeInterval:kBufferRefreshDelay];
		}
	}
	[apool release];
}

This is a pretty simple method. Remember we are in a new thread, so we need to set up our own pool. Once we are no longer playing, then the thread ends. We make one call basically to rotateBufferForStreamingSound:. Finally if we are interrupted then we dont need to be refilling but our thread will still run (until we are terminated if that happens). To be good citizens we will decrease the amount of time we are checking the thread.

Otherwise we just come back in kBufferRefreshDelay seconds. Setting this number can be a bit tricky. If you set it too close to the actual time it takes to play your individual buffers (as you would think) then if it lags at all then you will fall behind and never be able to catch up. You want it to be more than once during every chunk, incase you need to load more than one chunk because of a slow thread. However, run it too often and you are wasting cycles. I have mine set to 0.25 seconds based on the 48000 byte buffer. I dont even remember why i came to these numbers and they might be terrible. but they do work. Feel free to tune them to your heart's desire.

Next up, the actual buffer rotator:

// this checks to see if there is a buffer that has been used up.
// if it finds one then it loads the next bit of the sound into that buffer
// and puts it into the back of the queue
-(BOOL)rotateBufferForStreamingSound:(NSString*)soundKey
{
	// make sure we arent trying to stream a normal sound
	if (![[soundLibrary objectForKey:soundKey] isKindOfClass:[NSDictionary class]]) return NO;
	if (interrupted) return YES; // we are still 'playing' but we arent loading new buffers

	// get the keyed record
	NSMutableDictionary * record = [soundLibrary objectForKey:soundKey];
	NSUInteger sourceID = [[record objectForKey:@"sourceID"] unsignedIntegerValue];	

First some defensive programming, if we are getting called with the wrong key then get out, if we are interrupted then we are not loading any new buffers, so get out (but return YES because we want to keep the thread alive)
Then we grab our ubiquitous record and start to fill in some variables.

	// check to see if we are stopped
	NSInteger sourceState;
	alGetSourcei(sourceID, AL_SOURCE_STATE, &sourceState);
	if (sourceState != AL_PLAYING) {
		[record setObject:[NSNumber numberWithBool:NO] forKey:@"isPlaying"];
		return NO; // we are stopped, do not load any more buffers
	}

First up: check to see if this source that we are meant to be loading buffers into is stopped. If it is, then we dont need to load any new buffers, and we want to return NO so that the thread finishes as well.

	// get the processed buffer count
	NSInteger buffersProcessed = 0;
	alGetSourcei(sourceID, AL_BUFFERS_PROCESSED, &buffersProcessed);

	// check to see if we have a buffer to deQ
	if (buffersProcessed > 0) {
		// great! deQ a buffer and re-fill it
		NSUInteger bufferID;
		// remove the buffer form the source
		alSourceUnqueueBuffers(sourceID, 1, &bufferID);
		// fill the buffer up and reQ!
		// if we cant fill it up then we are finished
		// in which case we dont need to re-Q
		// return NO if we dont have mroe buffers to Q
		if (![self loadNextStreamingBufferForSound:soundKey intoBuffer:bufferID]) return NO;
		// Q the loaded buffer
		alSourceQueueBuffers(sourceID, 1, &bufferID);
	}

Next up, the big event: we see how many buffers the source has processed since our last check. For every buffer that has been processed we should fill in a new one and queue it up.
Before we can queue up a new buffer, we need to dequeue the old one. We do this with a call to alSourceUnqueueBuffers(). This fills in our bufferID variable, we can use this buffer now for whatever we want. In this case we want to fill it up with the next chunk of sound data and then put it at the end of the queue.

We call our loadNextStreamingBufferForSound:intoBuffer: method and if it returns NO, then we are all done and we can get out.
If it returns YES then we can queue up the newly filled buffer.

	return YES;
}

Finally, if we have made it this far then it was a successful buffer load and we return a YES to keep our loop going.

That is about it. The way this is all built, once you cann alSourceStop() on your streaming sound ID, everything sorta cleans itself up. The buffers stop rotating, and the thread stops. You still have buffers in memory tho, so if you want to clean those up too, be sure to add that code.

To Sum Up

OK, so there are the five-ish steps to streaming sound glory with OpenAL.

Obviously I have left out all the other code you need to get this running, have a look at the articles linked at the top, they have most of the rest of it.

Also I should mention that all of this code comes from an earlier, simpler version of what I generally use as my 'sound engine' nowadays. (basically I have been working on finishing this tutorial for awhile now and my working code has evolved since then :-)
This is not to say the sample code here is necessarily inferior, in fact my current code does almost exactly the same things, only there is a bit more optimised state handling and some other things that make it faster/easier, but not as easy to explain in an already very long post. So feel free to take this code and make it better in your own way :-) (for instance, instead of using dictionaries for state, I now have some proper sound classes that hold all that state for me etc..)

Also, i should say that if this code crashes your machine, or bricks your phone or makes your cat lose all it's hair, it is not my fault, you have been warned.

Cheers!
-B

Thanks to anyone who stopped by the Unity3d booth at GDC and said howdy to Sandor and I and checked out the multi-touch table: the FishTish.

As promised, the updated uniTUIO scripts have been made available on the xtuio.com site!

They have actually been up for quite some time now, but I ahve been (as usual) either too busy or too lazy to get a post up, so here it is:

Note: they have been pretty much fully redesigned to take advantage of the ‘iPhone Input’ style of touch detection.

What does this mean? Well, the older scripts used a more centralized raycasting and distribution method. There was one main event handler that took each touch point and checked to see if it collided with any registered ‘touchable’ objects. This is a perfectly good design, and it works very well when you have lots and lots of points (in other words, you are only raycasting once per touch per frame)

However, we decided to go away from that and create a fake iPhoneInput class that you can access from the desktop version of Unity.

Why?

Well, the most compelling reason was so that we could quickly and easily port iPhone apps to the touch table. Once I had the faux iPhoneInput class that was being populated with TUIO generated touch information, then it was trivial to take a few of the iPhone apps (and most all of the Unity iPhone tutorial projects) and simply drop them in and have them ‘just work’.

In fact, funny story: On the first day of the GDC expo, I had another Unity/iPhone dev come up and he was excited about the FishTish and thought that his app: iPottery would work really well on it. I grabbed the code from him with the intention of dropping in our scripts and installing it as one of the demo apps on the FishTish. Well, that day was quite a long one and I did not have the energy to actually do anything with the script until the next morning. I spent approximates 15 minutes with a totally unfamiliar code base (iPottery scripts are all in JS and the uniTUIO stuff is all in C#, so that also speaks to how simple it is to use them :-) and I had it working no problems. I spent most of my time deleting the onGUI stuff (onGUI doesnt work with uniTUIO, soory!) and adding some guiText/guiTexture based buttons instead.

For the next two days of the expo we showed off iPottery along with all the other demo apps and it worked great!

Ok, enough of my tangent, how does this actually work?

Well, grab the sample project from the link above, there are two scenes, one called ‘basic touch’ and one called ‘buttons’. Buttons actually contains the entirety of ‘basic touch’. (it just has more buttons in it)

Effectively what the new scripts provide is am iPhoneInput class that you can use in your TUIO enabled MT apps just like you would if you were writing an app for your iPhone or iPad. I have provided a dozen or so scripts that show some examples of how to use the iPhoneInput to do things like basic drag/scale/rotate and build buttons using either GUIText/GUITextures or 3d objects.

I havent had time to put together a video yet, but the code is pretty easy. The only thing that you will need is to make sure that you have an active BBiPhoneInputManager script running somewhere in your scene.

I use the BBTouchManagerStarter script to check to see if there is already an input manager and if not then make one. This is good because if you switch scenes, the input manager does not get destroyed. In an actual deployment, you can just instantiate one in your opening scene and be done with i, but during testing when you want to be able to just look at the scene you are in, this can become cumbersome, so I use the BBTouchManagerStarter in all my scenes instead.

Cheers!
-Ben

Sorry this has taken so long. I had it sitting in my to-do pile forever. And I had figured out the problem, just not fixed it.

Thanks to Morgan at bluecrash.com for sending me an email to kick me in the ass and spend the ten minutes to fix it.

Anyhow,

If you are wondering what the problem was it was that I had forgotten to lock the CVPixelBaseAddress before trying to access the CV frame pixel data. (when grabbing frames off the QTCapture frame buffer)

On Leopard this worked fine (for some reason) but was wrong. Here is the proper way to go about it:

// Lock the base addy for the pixels
// or it will return nil
CVPixelBufferLockBaseAddress(videoFrame,0);
unsigned char * srcBytes = CVPixelBufferGetBaseAddress(videoFrame);
.
.
.
// dont forget to unlock it when you are done
CVPixelBufferUnlockBaseAddress(videoFrame,0);

And that seemed to be the problem on snow leopard. I know, lame and easy. So again my apologies for not just getting it done sooner.

Also, I do plan to release the code for xTouch, but it needs a refactor and a clean first, so that will have to wait. You can get the binary (works on leopard and snow leopard) here:

xTouch.zip

Cheers!
-B

In any case, I managed to stay up very very late a few nights ago and went through the Molecules codebase and extracted the essence into what we are calling the uniTUIO community edition.

If you have not heard of this before, the basic idea behind uniTUIO is to have a nice set of scripts to allow you to easily implement TUIO based multi touch input into any Unity3d application.

If you dont have Unity3d or dont know what it is, then you really should go to www.unity3d.com and get a copy. It is crossplatform and the basic version is free (and it really isn’t very basic, I built a couple complete games using that version (back when it was called the indie version)).

Anyhow, the home of uniTUIO is at the xTUIO.com website. Ultimately I will be publishing posts like this one up there, but for now, since it is still early days and not that many people know about xTUIO.com I will post this here and have sandor crosspost it there.

Currently the uniTUIO CE is comprised of a unity project which has a couple basic scenes and a couple dozen scripts. Also, I made a couple videos showing how to use the various scenes.

There are two basic methods to get at the TUIO event information.

The first one, which is used by the BBCrosshairController and BB3dCrosshairController scripts, is a simple polling method. These scripts have access to all of the touch information for the entire surface and can act accordingly.

These scripts call up to a singleton script: BBTouchEventManager. It is the touch event manager that actually takes the TUIO information and converts it into a BBEvent object, which basically means that it converts the information from TUIO form into a Unity friendly format.

If you are building a ‘traditional’ sort of unity game where you have a central character that you are controlling, then using this polling method makes sense. This is basically the same thing as using the Input class in unity or the iPhoneInput class in Unity iPhone.

One example of this is the particles scene which uses a script called the BB3dCrosshairController which polls for all available events and spawns a particle system under any touch events. using this method you can replicate all the fun ‘fire from the fingertips’ sort of MT demos in about 3 minutes. (or longer if you are like me and love to just tweak the particle systems for ever and ever to get them ‘just right’. Note: the particle system that I did for the examples is not ‘just right’, i just threw some values at it so that you would have an idea of how to do it yourself)

However, in many multi-touch applications you tend to have dozens and dozens of objects in the scene that all need to be aware of any touch events that are interacting with them. In this case it does not make good design sense to have each and every object grab a copy of all the events and try to figure out if any of them apply to that specific object. Instead we have a central distribution point that checks the incoming events against the scene full of objects and sends out event messages as appropriate.

This is where the BBTouchable scripts come in. The BBTouchEventManager takes the raw TUIO events and raycasts through the scene looking for any objects that are both in a special ‘touchableObjects’ layer and have a BBTouchable script attached to them.

The BBTouchable script provides some overrideable abstract touch input handling methods. For the most part, when doing MT apps that deal with lots and lots of different objects that need touch inputs, you will be subclassing BBTouchable to get your custom functionality.

In the uniTUIO scripts I have made a few example subclasses of BBTouchable. one is called BBBasicTouchManipulation, and it provides any object with the basic single touch to drag/double touch to scale and rotate gestures that are very common in MT apps.

The second one, BBTouchableButton shows how you can make a button that lives in 3d space and reacts to touch events.

You should have everything you need to get started with Unity and TUIO now. If you have questions, dont hesitate to comment here or on the xtuio site, or email me directly, or pm me at the unity forums, or whatever.

Cheers!
-b

Since we announced uniTUIO a few months ago, and entered a uniTUIO-enabled app into the Unite09 contest, there has been a flood of emails coming in asking me if we are releasing the uniTUIO source.

So! Here is the answer:

Yes!

But I have to find a few free hours to rub together to get the code into a useable and clean form. We set out with the goal of having a nice framework of scripts to make using TUIO inputs in UNity3d trivially simple. I think we succeeded.

uniTUIO was built during (and for) the Molecules project (which was really a half-dozen different media presentation applications). As the project deadline began to loom, making the Molecules apps became a bigger priority than preserving the purity and re-usability of the uniTUIO framework. As a result, I had to cut many corners in order to meet the deadline (which pretty much always happens) so the uniTUIO stuff is riddled with Molecules specific code which is neither generically useful, or particularly re-usable in any easy way.

So, I need to go back in and refactor the uniTUIO stuff to make it not suck basically.

However, I will explain to you what the uniTUIO stuff includes, so that you can at least not get your hopes up too much :-)

First off, at the low level, it starts with the c# TUIO/OSC implementation by Martin Kaltenbrunner, which everyone can get ahold of here: http://reactivision.sourceforge.net/.

On top of that is pretty much where uniTUIO starts. It is a collection of about a dozen scripts. There are a few static singleton ‘manager’ style objects that start up the TUIO scripts and listen to the events as they come in. These convert the TUIO data stream into some portable ‘event’ objects that include the local Unity environment information. (in other words they convert from TUIO screen-space to Unity viewport space).

Also there is an event manager that does raycasting into the unity scene and picks out any objects that are in the right layer (a ‘touchable’ layer) and passes the events onto those objects.

There are a few more scripts that are meant to be attached to touchable objects.
One is a high level touch event handler that accepts touch events from the event manager and applies them to it’s gameObject in a generic fashion.
Inherited from that object is a basic touch manipulation script that provides simple gesture support to handle the basics like dragging, rotating and scaling a gameObject.

That is what uniTUIO includes. It is meant to be a simple starting place for you to be able to build on for your own MT projects in unity.

So anyway, we will announce it when we release it (hopefully sometime this month) so keep an eye on the RSS feeds. Cheers!
-B

ps: for those of you who are very impatient, have a look at this thread on the unity forums. User Jorgen posted a simple unity TUIO implementation project, which is a good place to get started.

This is pretty huge news for the game development community. It is, as they say (and forgive the pun) a Game Changer.

If you don’t already know, Unity is a very powerful tool for rapidly building games. (it is billed as a 3d game engine, but it also works quite well for 2d games) One of the many features of Unity is the ability to take the game project you have built, and with the click of the mouse, build a version for the mac, a version for the pc and a version that runs in a web browser. This is excellent for distributing your game but it has other side benefits as well. I collaborate with people all over the world, and we use the in browser games to share our ideas quickly and easily, not to mention it is much easier to get your testers to test out your prototypes if all they have to do is open a browser window.

Before now, the indie version was not very expensive, less than $200 or so, but now that it is free, I think we will see a flood of new and interesting games come out in the coming months.

What does this mean for indie game development?

Well, the obvious thing is that we will see a ton more web based titles, probably lots of crap, but there will always be a few gems in there.

I think that Flash development will start to see a big challenge from the Unity guys. I know a ton of flash guys who have been eyeing unity for awhile now and this will most likely push them over the edge. dont get me wrong, flash is a great tool, and it is great for making games, but it is not a game-building-tool. Unity is designed from the ground up to make games. Here is a good post to read if you a flash dev who is on the fence: http://diamondtearz.org/2009/01/14/10-reasons-for-flash-developers-to-learn-unity-3d/

The even bigger news for me as an iPhone developer is that I think many many more people will start to use unity to build games for the iPhone.

Now, Unity iPhone Basic (the cheapest iPhone license) is still about $400. And some of the zero budget indie developers might have to think twice about that kind of money (it is totally worth it, trust me) But now you can download the Unity Free and prototype your game, or release a web version or just get to know unity and then be ready to port it over to the iPhone version. We are doing that right now with ‘Mole’ and I will be blogging about what it takes to move a prototype level game from the desktop version of unity into a working iPhone version.

I will also be interested to see how this effects the other iPhone game-focused APIs like cocoas2d. Cocoas2d and the like are still free (so still $400 cheaper than using unity to develop your game) and if you are building a 2d game it makes a lot of sense to start with cocoas2d, but I can tell you from my personal experience that building a 2d game with unity is very easy and if your time is worth anything to you then the $400 investment to upgrade to iPhone Basic is well worth the money. Also, I cant stress this enough, you can now prototype your game in Unity for free. Even if you then decide to use cocoas2d to build the deployment version (a few reasons you might want to do this, more on that in a second) you can still do a rapid prototype with Unity to figure out all your gameplay mechanics and decide if the game is fun or not.

So, Unity is great, but what are the downsides?

Well, in terms of desktop or web development, there aren’t any real downsides. Go get a copy and start making games.

In terms of iPhone development there are a few caveats.

First off, as i have mentioned, it is not free. But $400 is very cheap for what you get.

The biggest issue with using Unity Basic (and to some extent this applies to the pro version as well) to develop iPhone games is that your app size will not be under 10M (which is the size limit for apps to be able to be downloaded over the cell network). For the most part this is not a big deal, most games with any amount of depth tend to be bigger than 10M no matter the development tools used.

However, the casual games that you are trying to sell for $0.99 and you want to be an impulse buy, and so you want them to be less than 10M, these are nigh impossible to make with Unity. Why is that? Well, the unity engine is very capable, comes with all sorts of great things like built in physics, scripting, shaders, and lots of other goodies, but all that comes at the cost of size. If you spring for iPhone Advanced then you can strip out parts of the engine you are not using, and possibly get under the 10M mark, but this is a very hard thing to do.

My advice: even if you are looking to build small casual games for the iPhone, I would still suggest you get the now free Unity and prototype with it. Then once your game is working fall back on one of the other frameworks like cocoas2d to build it. However, if you have even an inkling that your game will exceed the 10M limit, then just get the iPhone version.

Cheers!
-B

http://www.snowferno.com/2009/09/23/playhaven-unity-and-snowferno/

Check it out if that sounds like something you might want to do.

Cheers!
-B