I have been trying to write a blog post about texture atlases forever now, but I keep getting distracted and then I forget I have a draft up here waiting for me.
SO! I am going to be doing a series of posts about some basic openGL|ES stuff on the iPhone and how to make your cocoa-based core animation application (or game) perform better. Ultimately I will tell you that you should switch to openGL, but for now I am going to keep it simple and talk about core animation layers and texture atlasing. (and for now, we will only be talking about 2d stuff at 1:1 rez, so no mip mapping of the atlases for those of you who are reading ahead, as that gets more complicated than we need right now)
Why do I want to share this information with you? Well, the first game I did on the iPhone: Snowdude was a big ole 2D sprit animation game. I originally did the entire thing in Core Animation and it got about 4 fps, which was not enough :-) So I switched to openGL and could easily get 30 or 40 (though I only needed to do 12 because that was what all the sprite animations were built at), so this is the story of that transition.
The first hurdle we will cross will not require me to get into openGL just yet, but talk about how to optimize your sprite animations, even within CA, so if you only need a teensy bit of performance, you might be able to just get away with this and not actually make a full leap into openGL.
So, lets say I have a sweet animation of a dude face planting that looks like so: (Oh and BTW big mad props to the Lycette Bros makers of the aforementioned Snowdude for letting me use their raw graphics for examples)
10 frames, so 10 images.
if I want to turn this into an animation in the easiest, cocoa-tacular fashion I would do something like so:
UIImageView * spritely = [[UIImageView alloc] initWithImage:[UIImage imageNamed:@"frame01.png"]];
spritely.animationImages = [NSArray arrayWithObjects:[UIImage imageNamed:@"frame01.png"],
[UIImage imageNamed:@"frame02.png"],
[UIImage imageNamed:@"frame03.png"],
[UIImage imageNamed:@"frame04.png"],
[UIImage imageNamed:@"frame05.png"],
[UIImage imageNamed:@"frame06.png"],
[UIImage imageNamed:@"frame07.png"],
[UIImage imageNamed:@"frame08.png"],
[UIImage imageNamed:@"frame09.png"],
[UIImage imageNamed:@"frame10.png"],
nil];
spritely.animationDuration = 10.0/fps;
[myBigView addSubview:spritely];
[spritely startAnimating];
this is super duper easy. But it is slow. (well, it isnt particularly slow for a few animations, but get a couple dozen of any decent size and your performance will kinda suck)
Well, I dont know exactly what apple is doing on the backend but very likely the CA stuff is built on top of the openGL stuff. So basically what happens every time the frame needs to be redrawn is that you have to bind a new texture then render that texture out to the screen buffer. Binding textures can be slow unless they are already resident in vram, but even then if you can skip binding a texture and just do a draw call every frame then you will have significant gains) (technical note: the CA stuff is actually kinda half built on openGL and half raw core graphics (and half pixie dust) basically the way the tech dudes explained it is that CA goes out to whichever subsystem is best for the task at hand. That said; no matter what the backend is doing, the texture still needs to be bound to the screenbuffer somehow, and that takes time)
So, how do we skip that texture binding step? Well, we have to abandon the UIImage view. (well, you could subclass it and do all this in a uiimage sublclass, but for our purposes we are going to keep it nice and simple and just make a generic uiview subclass)
But first! we need to make an Atlas!
OK, what is a texture atlas? (and why do we care?)
If you want the very detailed explanation then go read the atlasing whitepaper by NVidia. (which is far more detail than I am going to go into)
A simple explanation of an atlas is just this: a big image with lots of little images inside it. Like so:
hey! that looks an awful lot like the stuff above! and it is in fact, the same thing, but now I am calling it an atlas instead of an example of 10 images. So that is it, a texture atlas is just a bunch of images in a bigger image. How does this help us? Well it means that the larger image only needs to be loaded as a texture once and then you just draw out different bits of it to the video buffer.
How do we use this new fangled atlas thinger?
First, we need to make a new UIView subclass and add a nice CALayer that we can use to store our images:
// our standard frame init method
- (id)initWithFrame:(CGRect)frame
{
if (self = [super initWithFrame:frame]) {
// our main layer, holds the main image
imageLayer = [[CALayer layer] retain];
imageLayer.frame = self.bounds;
imageLayer.opaque = YES; // this speeds stuff up if you don't need transparency
[self.layer addSublayer:imageLayer];
}
return self;
}
// set our big atlas image
-(void)setAtlasImage:(UIImage*)myAtlasImage
{
// this is basically binding the big image to our layer
imageLayer.contents = (id)myAtlasImage.CGImage;
imageLayer.masksToBounds = YES;
// this is the magic code here, this tells the layer to only display a
// subset of the larger image.
imageLayer.contentsRect = [self rectForFrameIndex:0];
}
OK, so we build a simple view with a single sublayer. When we set the atlas image for our new animated view, we set the contents to the entire image.
Now if we just did that, you would see the entire atlas (smashed into whatever frame you had defined when you alloced the view). This is not what you want, so we add the contentsRect definition, which allows us to tell the layer to just display one sub rectangle of the entire contents.
The contentsRect is the basis of the whole idea of the texture atlas. So how do we figure that out?
The contentsRect uses normalized coordinates. If you havent done much with contentsRect or the few other CA properties that inexplicable require normalized coords, then it can be kinda confusing at first, but really it is very easy (and a nice way to handle things).
So, lets have a look at our image and see how to generate some normalized rectangle coordinates for our frames:
The sample image above is 480 x 236 pixels (which is a strange size, i know, more on that later, but the upshot is that this is a much smaller version of the raw graphics (to discourage people from stealing them :-) so the example frame sizes are a bit odd… you will get over it I am sure)
A single frame is 121 x 80 (ish) pixels.
OK, quick tangent: this image is a smaller version of the images that the animator built for snow dude. Keen readers will notice some nice hot pink crop marks, those are not part of the animation, but are in there so that I can feed that image into an atlas generator program that I built and it strips out the proper frames for each animation based on the crop marks. For today, we are going to pretend those are part of the animations, and I will talk about optimizing your atlases in another post)
OK, where were we? Oh yes, contentsRect. So if frame 1 is in the upper left, then it is bounded by:
// remember that CALayers have 0,0 at the lower left
CGRect frame1 = CGRectMake( 0.0, 156.0, 121.0, 80.0);
Great! almost there, now we just need to normalize that rectangle. That means that all the values need to be between 0 and 1. so we divide all the widths by 480 (the total width of the image) and all the heights by 236 (to total height of the image). So something like this:
// remember that CALayers have 0,0 at the lower left
-(CGrect)rectForFrameIndex:(NSInteger)frameIndex
{
NSInteger column = frameIndex % 4;
NSInteger row = frameIndex / 4;
row += 1;
CGFloat x = frameIndex * column;
CGFloat y = atlasHeight - (row * frameHeight);
CGFloat normalizedX = x/atlasWidth;
CGFloat normalizedY = y/atlasHeight;
CGFloat normalizedWidth = frameWidth/atlasWidth;
CGFloat normalizedHeight = frameHeight/atlasHeight;
CGRect frameRect = CGRectMake( normalizedX, normalizedY, normalizedWidth, normalizedHeight);
}
so, our frame 1 will look something like this rect:
// remember that CALayers have 0,0 at the lower left
CGRect frame1 = CGRectMake( 0.0, 0.6610, 0.2520, 0.3389);
OK, that was pretty easy, but why do we need normalized coordinates? Well, the short answer is: you need them cuz the methods require them. The long(er) answer is that if you start to use openGL or even doing mipmapping then they become super handy. (in otherwords that normalized frame would work whether i loaded the example atlas that is 480×236 or the original atlas which is 3 times bigger, and you wouldnt have to recalculate the contentsRect. and later, when we start talking about texture binding and UV coordinate space, it will all make so much more sense)
OK! Now all we need is a timer that switches frames for us and we are all set!
Something like this will do the trick:
-(void)timerFire:(id)sender
{
// we want this to be immediate
[CATransaction begin];
[CATransaction setValue:(id)kCFBooleanTrue forKey:kCATransactionDisableActions];
imageLayer.contentsRect = thisFrame;
[CATransaction commit];
frameIndex ++;
}
What is all the CATransaction stuff? well, for better or worse, whenever you do something in core animation that CAN be animated it will be, unless you tell it not to. since we want the next frame to show up immediately, we do not want to animate the contentsRect transition (otherwise your new frame will slide in and the old one will slide out). So we use the CATransaction system to shut off the animations.
AndFinally something to call this animation timer routine:
[NSTimer scheduledTimerWithTimeInterval:1.0/fps target:self selector:@selector(timerFire) userInfo:nil repeats:YES];
And that is it. Super basic texture atlasing.
The pros of this approach: better than loading individual images into your CALayer and animating them that way (or via a UIImageView)
This also makes memory management a bit simple (since you only have to worry about a single image instead of 10)
And the best thing is, this way is halfway to openGL, which is where we want to be.
This particular implementation has lots of drawbacks, like all your frames need to be the same size, and they need to be packed into the atlas in the right way and the right order. Also we are calculating the frame rect over and over again for each frame (this is very quick, but if you have lots of sprites then it seems like a waste when you could just cache them) And because each frame has to be the same size, then we are actually wasting quite a bit of memory with all that whitespace. (turns out transparent pixels take up just as much memory as colored pixels! who knew?!?)
That is OK, we will talk about all this in the next post when i get into openGL.
Cheers!
-B
This is exactly what I was looking for! I wanted a more performant way to get animation into my application without the use of UIImageView.animationImages. However, I did not want OpenGL, as mixing UI components with OpenGL is a no no for performance.
Given that you know what the atlases dimensions are when you apply it, I suppose you could easily calculate all the frames in advance prior to it’s first run, which would save you the repeated calculations on animations that loop. Or have it calculate on the first pass, and cache the values for subsequent passes.
Anyway, thanks again!
Hey Logix,
Yes you are absolutely correct. In fact I would definitely suggest pre-calculating your atlas positions prior to your animation.
I have a little program that builds the atlases for me from individual images (if I ever have free time, i might even do a post about it).
In any case I generate a plist meta file with all the frame positions in the atlas and then load that meta data at the same time I load the image (usually sometime before the animation is required)
This is also handy because you can then be very clever about your image usage. For instance, the last four or five frames of the example animation are really only about half the height of the first few, that is a pretty big waste of space (and memory). Instead I pack them into the atlas with the minimum amount of space around them and just record a center offset so that I can (in core animation or OpenGL) move it to the right position in the animation cell.
This way you can pack your atlases much much tighter and save tons of space on the iPhone :-)
Anyway, it is early here so hopefully that makes some amount of sense.
Cheers!
-B
Ben, I would LOVE to get my hands on the atlas program of yours, I started writing my own, but hey, whatever saves time =). I also played with pixen (http://opensword.org/Pixen/) but I could not get the transparency to output for the PNG sequence. Also it would make the atlases greater than 1024, which would still mean I need to edit them for openGL usage.
Basically I vote for you doing a post on your program =)
Hi Ben;
This saved me such a lot of R&D. Before reading this post, I was actually using cached arrays of images depending on what the sprite was doing, however I was using the NSTimer to swap images much like your approach. I have gone from 60 image frames to one, and re-implemented my cached arrays of image to caches arrays of frames.
Thanks so much for the blog post….
I have one question though. In my old approach, I have two animations going, one to swap out images and the other to move the location of the spite and I am not sure I had the best practice implemented.
I would use an animation block (iOS4 style) for the center position over a duration “x” and implement the animation delegate “animation did start” where I would start a timer. The timer would call a selector to swap out the images every x/(Frame Numbers). I can’t help but feel this approach is prone to falling out of sync between the two animations. Should I be doing this all from a timer where the selector not only draws the new frame but moves it too using the transaction block?
Your insight and design pattern would be greatly appreciated.
P.S. I am in Melbourne too ;-)
Hey Nader,
For anything sprite related (even when I am not using OpenGL) I tend to implement a master clock that runs at some integer fraction of the display rate (ie nowadays I use the CADisplayLink, but used NSTimers or NSThreads before CADisplaylink was available on most devices. (I think for now it has become pretty safe to support 3.3.1 and above which mean you always have a CADisplaylink for your timing needs :-)
So with that in mind I tend to design all my animating objects with an update method built in (sort of standard game loop style) and in that update method I handle any state changes that need to occur. So if my update is running a 60 fps and my animation is running at 12 fps then every 5 frames I update the animation sprite, but every frame I might be updating the position so that the movement is nice and smooth.
Obviously this is more work than just using the core animations but if you need to sync your animations with movement, then I would say that trying to shoehorn CA into that role is probably not the best idea. (this isnt to say it cant be done, but once you get more than a few independently moving and animating sprites then it gets a bit dicey in terms of performance and keeping everything in sync)
A great deal of the really slick UIKit stuff is really good for user interfaces (where much of the animation is secondary to the actual function of the interface, so if you miss a frame here and there, or your slide out animation is a few frames too slow nobody will ever notice) Once you start doing games you generally need to have better control so it is usually worthwhile to roll your own animation system that is tailored to your specific problem set.
Cheers!
-B
Hey, if you are in melbourne you should be coming to the IGDAM meetups, http://igdamelbourne.org/ once a month, usually on a tuesday (was one just last night) come and say howdee :-)
Wow, that is a completely different approach to what I am doing today. I have never written a game before, most of my application development has been enterprise/utility based and so never had to address these sorts of issues as you pointed out.
I will need time to digest what you have written and look into it’s implementation.
I cannot thank you enough for your guidance. This site is a very valuable resource for people like me as I have out grown stackoverflow (well perhaps not from what you are showing me).
I’ll look into the IGDAM, this is all new to me :-)
This was great.
I implemented the design pattern you gave and it really has made such a difference to the animation. I have such a granular level of control and the animation is really smooth. Great tip about having odd and even frames, where by using odd and even frames for positioning and only even frames for image masking, that a fine example of knowledge vs experience.
I have a question about the clock implementation. Is it standard practice to implement a master clock that all animations are driven from or implement a clock per animation group?
If the practice is a master clock, then I was thinking of making a singleton static clock object where by I register and de-register selectors to perform in the background. But is it worth the effort, or is just having a clock per animation group adequate.
FYI, my term animation group refers to say a clock for the sprite that is allocated dynamically as the sprite does something and a clock perhaps for an animation that is constantly going in the background.
Sorry to keep firing off these questions at you but getting access to a seasoned game developer is a rarity and ever so valuable.
Hey Nader,
in relation to a master clock, I would say that in most cases this is the best way to go. If performance isn’t an issue (a rarity on the iDevices :-) then having multiple clocks is an OK solution. However, each timer you have running requires extra overhead, so if you can keep it to a single master then all the better.
the CADisplayLink is basically already a singleton master clock, however I generally make another object (sometimes a singleton) that handles all the scheduling for any objects that need it.
However, once you have a master scheduler then you can also do clever things like set up delayed scheduling (much like NSObject’s performSelector:withDelay method, but synced to your timer.) So i will set up some methods that are like: call this selector on this object after this many seconds, or call this selector on this object every x seconds etc… and let the scheduler handle the dispatch. (basically as you suggested, this is how I have objects ‘register’ for updates. but I have a few different ways so that I can have one-offs and somethings that are only updateing every 5 seconds or something as well as things that need updates every frame.)
Once you have a master scheduler, then things like sharing information between threads becomes a bit easier. You can schedule some special times to query the other threads and let them update your scene objects at times when you are not calling any other scheduled objects.
Cheers!
-B
Well I have made some really great progress but I thought I would share some interesting information I gathered along the way.
When I first implemented the CADisplayLink, I noticed that my frame rates would stall, especially under situations where there was a lot of gestures going on. I kept thinking how using NSTimer didn’t seem to suffer from this but DisplayLink does. I was beginning to question this entire approach but thought surely these guys knew what they were talking about, hence I must be doing something wrong.
I then decided to roll my own gesture recognition methods and that seem to have cleared it all up. The UIGestureRecognizer appears to play havoc with high frame rate animation. The NSTimer seems less prone to it but call still suffer dropped frames.
Now I guess I just need to do some trial and error to determine what speed gestures constitute a swipe and how much travel needs to occur. FUN…..not!
Hi Ben and Happy New Year;
I have made a lot of headway with my game, learned so much about animation, specifically, looking out for dropped frames and catching up, velocity calculations and determining key frames. However (you know this was coming), one thing still alludes me.
When I used the UIKit animation routines, the animation where faultless, especially when it came to a scaled UIView and animating position. Using the manual frame by frame animation seems to suffer from dropped frames a little when using a scale of 1, however, when changing the scale of the entire scene to, say, 0.75, I get really bad choppy movement.
I thought this was probably because I was rendering frames perhaps twice in the same position and hence giving the allusion of choppy animation and compensated by calculating the number of frames required to traverse the scaled distance based on non scaled velocity. Even though, I was rendering less frames in the scaled view and increasing my delta per frame, I was still getting really choppy animation. When I reverted back to UIKit, it was silky smooth.
I think I am missing something fundamental about animation in iOS, do you possible see what it might be?
Without looking at the code I cant say for sure why you would be getting choppy performance with the CA stuff.
I would run the performance stuff in instruments (or roll your own) and see where the bottlenecks are in the code. If it seems like you are spending too much time in the CA methods, then double check how you are loading your layer contents and make sure you arent re-loading them every frame, etc. (so load the contents once, and then change the contents rect every frame etc..)
Good luck!
-B