benbritten.com

I just put up a few new objects into the BBTouch project in the openTouch repository. Not all of it has been fully tested, but it seemed to work OK.

There is now an event controller that generates events for a delegate object. I also made a simple delegate that just resends the events as notifications. In the future there could be delegates to generate system -level events, or TUIO events on a serial port, or whatever.

There is also the stub of a configuration manager that does very little right now, but will eventually house all the configuration data and objects.

I am gong to be gone for a few days so there probably wont be any updates from my end until late next week.

It is amazing how much faster you can make something when the tight loops get called a zillion times a second.

Anyhow, the blob detector i posted yesterday was the first clean, working version. But it hadnt really been optimized (never optimize before it's time). So i went back in and looked at the tightest loop: specifically scanSrc in the detector. If you looked at the code you can see that it is written to be more programmer-friendly than fast. I tend to think that whenever you can afford to, programmer-friendliness is a good trade off for minor performance loss. However, in this case there were a few spots that needed to be changed.

The biggest (performance wise) change is the movement of the initial 'goodness' check into the loop proper. also instead of calling validLoopPixel, i just put the threshold check right there. It is generally considered bad coding to put the same code in two places. but this is a concession I am willing to make. Just taking out the call to validLoopPixel shaved off 20 or 30% of the time it was taking in that loop (let's see. 640x480 pixels times 30 frames a second = 9216000 runs through that loop per second. anything you can do to shave off even the tiniest amount of time in that tight loop will yield great performance payback)

So, basically that one change made the biggest performance change, but i changed a few other things too: I added a regionOfInterest rectangle into the mix (to cut down on the number of times through the loop). and added a class method to the image processor to make the ROI rect in the image (ie the black pixel border that keeps the blob detection algorithm from going out of the ROI)

Thats about it as far as code: here is the tar:

bbtouch2.tar

Now, this whole tar with an incrementing value in the name is great and all, but it is not a good way to release code. Luckily for me, I had a long iChat with Pawel from the NUIGroup, and he is helping me get this stuff up onto the opentouch google code repository. While I have used SVN before, I havent used the google code repositories yet. I am going to attack that problem later today. I will be putting up the same code that is in BBTouch.2 initially, but then all the updates will go to the google repository.

I am back to work on the OSX Cocoa multi touch thing after being away doing some paying work for a few weeks. I am pretty happy with the prototype, as far as most of the concepts seemed to work OK. The only thing that was a bit iffy was my recursive blob finding algorithm. It works fairly well and fairly quickly, and it was fun to figure it out, but I am sure that it is very sub-optimal. So, i went to find out how the big boys do it. I grabbed the opentouch libs (thanks Pawel for all the great work!) and read the blob detector code.
Now i am a pretty code-savvy guy, i have been coding in C-like languages for a loong time now, but these pixel-level algorithms make my head spin without some direction as to what is really going on. So i had to look further afield. Somewhere in Pawels documentation I came across a reference to this paper:

A Linear-Time Component-Labeling Algorithm Using Contour Tracing Technique

Which is basically what the opentouch algo is based on. So I started from scratch (so i would know how it all worked in the end) and rewrote the algorithm described in the paper in Cocoa. (well, with a few little C bits, but mostly Cocoa). With a nice working blob finder and a nice working sequence grabber and a stack of other design ideas that are scattered through my prototype, i started a new project form scratch to make a proper application. So far what I have is a very stripped down blob detector.

All it does so far is grab images from any firewire cameras and analyze them for blobs based on a simple threshold and background subtraction. It hasnt really been optimized for speed yet, although it is pretty quick. I think the slowest part is just setting the image into the NSImageView. Those views were never really meant to show video, so if you bump the raw camera size up to 640x480 it can get a bit slow. But I will be doing quite a bit of optimization later on. (and for actual blob detection, you really dont need to see the video, and if you did, it would make more sense to use a different view)

There is a pre-process pipeline (sorta) so if you feel the need to add any filters or anything before it gets to the blob detection portion of the show, you can do that easily. The blob detector is currently written to handle NSImages, but it would be fairly trivial to change that for CIImages or CGImages. (or all three)

In any case, here is the code for anyone who wants it:

bbtouch.tar

[UPDATE:] i managed to more than triple the performance of the blob detector last night. I will post more details and the newer code in a new blog post a bit later this morning once i have had some brekky and caffeine. :-)

Having settled on DI instead of FTIR for the time being, I went ahead and built a bigger, but still fairly small prototype table that i could use to develop the software (as opposed to a full-size one, since i dont have that much room in my office. The full size will come later when I have the software working well, and have written a few apps for it).

So, here is the second prototype (sans camera, i haven't mounted it in there yet, but it goes just in front of the mirror. I had it sitting on some books for the testing I have done up to this point, i took it out to take IR photos of single malt scotch for my previous post, and I forgot to put it back in for these, so you will have to use your imagination)

The Table is only 640mm high, and fits under my desk. I may even put wheels on it to make it easier to move out of the way when I am not using it.

The surface is just tracing paper, and the plexiglass is cut to fit one sheet exactly.

The plexi is actually two layers of 3mm (since it was easily purchased form the local hardware store). I haven't yet got around to buying a new sheet of 6mm or better. The double layer works fine for DI, i dont think it would work very well for FTIR tho. If you press hard, there is still a bit of bowing in the center, I dont really recommend the double layer, but it works if you dont have a single sheet of the right thickness.

The two IR illuminators, which are 840nm, are just in the right range for the fire-i camera. They are 12v, powered by my 5/12v experimenting supply. Eventually i will just by a 12V wall wart for them. Altho they have a pretty purple color here, (thanks to my digital camera) to the naked eye you can only see a very dim red glow when they are on (since they are so close to the visible spectrum at 840nm). I have put some of the tracing paper into the glass covers to make them more diffuse, I also taped over the photosensitive resistor so that they are always on (they were originally for nighttime illumination for CCTV cameras)

They are actually a bit close to the surface here, and they give a very spotlight-like coverage (even with the diffuser) which isnt exactly what you want. it makes the illumination very uneven, which causes the threshold filter to lose the blobs in the dark areas even though they are perfectly visible. This seems to be the big problem with the DI setup, nice even light coverage. I spent a great deal of time tweaking it before i came to the conclusion that i should be able to deal with the uneven coverage in software. Like a variable threshold.

My thought is that if i put something that has roughly the same IR reflectance and my fingers (the cardboard back of the tracing pad is pretty close) and use that to cover the whole surface, then i can take a snapshot of that and use that to analyze the coverage of the IR lights. Once i have a lighting brightness map, i can use that to scale the threshold values accordingly and hopefully be able to easily pick out the blobs i want even as they pass from light to dark. More on that later, once i have tried it out.

The projector is a ViewSonic PJ556d (which has a latency of about 1/20th of a second according to my tests, which is a bit slower than i want, but still passable). You can also see my super-duper-pro-plywood-scrap projector mount and adjustable rotation mirror. As crappy as it looks, it works fairly well, I can move the mirror back and forth to adjust the image on the surface. The surface is not 4x3 (obviously) and at the distance I have right now, my projection is just wide enough to cover the whole surface (the whole exposed surface i should say, since there is a 43mm border all around from the wood i used.) but i have spill on the top and bottom, which i use the mirror adjustment to put where i want (generally so I can see the menu bar on top). The projector has a native rez of 1024x768, so that is also nice. It can go as high as 1280 × 1024 (or maybe even higher) but that seems silly, since it is just scaling the image, and looks terrible. I am sticking to 1024x768.

As for control, I am using my powerbook g4 12" (not hooked up in the images), which also has a native rez of 1024x768, so i can easily run it in mirror mode when i am testing, or run as a second monitor so i can see all the config stuff on the powerbook and still get the 'real' image on the surface. I am actually doing all the development on a macbookpro hooked to a cinema display, which is why I am using the 12" for the control. I cant give up the big display to use the macbook for testing :-) Plus this way I also wont rely to heavily on the speed of the macbookpro. If i cant get the software to run perfectly fast on the G4 i am doing something wrong. there is no reason that simply blob detection should take very much processor power. (more on how i am blob detecting in a later post)

After much waiting (like four days, they were actually very quick, but as i have mentioned, i am very impatient) I got the fire-i camera:

There it is, with my 3-ply film-negative visible light filter professionally taped onto the 4.3/no IR lens.

Here are the obligatory IR shots:
First, here is the apple remote illuminating me making a silly face. Here is a hint tho: maybe dont use the apple remote when doing this kind of shot. iTunes went mad, and i found out if you hold down the play/pause button for too long, your computer will go to sleep.

And here is the standard: 'Hey IR makes liquids look clear! shot', usually done with a cola of some sort. Shown here to also work with single malt scotch.

Quick review: the unibrain fire-i is a pretty damn good camera. I highly recommend it for any of your fire-wire camera needs. One thing however, it is not autofocus. (which in this case, is a good thing) but you can see form the above images if you are just waving the thing around and not focusing it, then it can get a bit blurry. However, in the multitouch table it is fantastic. (and another problem i had with the iSight: autofocus. It wouldnt shut off! and since i had remove some of the optics, it could never find a good focus so it would constantly go from end to end of its focal length, which was quite annoying. You can turn that stuff off, but it is a giant pain in the ass. Just get the fire-i)