The Official BBTouch Quick Start Guide

So, you've decided you want to try this crazy multi-touch thing on your mac? Great! This here is a quick start tutorial for BBTouch. First thing is you are going to want to go and get it.

the code is here:
http://code.google.com/p/opentouch/

you will need to use SVN to download it, just follow the instructions on the google code site. Once you have the code, find the BBTouch directory, open up the BBTouch project in XCode 3.0 or better, build and run.

Or
There are binaries here:
http://code.google.com/p/opentouch/

you will need to use SVN to download it, just follow the instructions on the google code site. Once you have the code, find the BBTouch directory, open up the BBTouch project in XCode 3.0 or better, build and run.

Or
There are binaries here:
http://nuigroup.com/wiki/ and read around, then go to the forums and read around. I have documented much of my personal build process in earlier blog posts as well.

That's it!

OK BBTouch Specific Instructions:

BBTouch Interface Walkthrough

Startup BBTouch. You should get something approximating this:

Not much to see at first. Lets have a quick walk around the interface:
The big view in the upper right is the main view. Whatever view is in that position will also show it's specific settings. The three views on the left are the auxiliary views. They will stay active and live when you are using the main view, but you cannot interact with them directly. When you click on one of the auxiliary views that view will become the main view and your view settings will change to the new view's settings.

For instance, if you click on the Projection View, you will notice there are some different settings at the bottom:

Groovy!

The four main views, and their specific settings are:

• The Camera View:
  

  This shows you what your camera sees, it also shows you the camera mesh (more about meshes later).
  The Camera View options are:

  • Show Raw Video: hit this to see what the camera sees, shut it off when you tire of it
  • Freeze Frame: this will pause the video at the current frame captured. if you are detecting blobs, you will continue to detect blobs on the frozen frame
  • Video Size: this is the raw input video size. Generally you want the highest rez you can before the tracking becomes sluggish
  • Cam Settings: this opens the camera settings dialog. you can change things like white balance, exposure, gain, etc.
  • Resnap BG: this will retake a background image, use this if your lighting has changed since you first started up. also if you are getting spurious blob positives, resnapping BG often fixes it. (you might have more or less IR in your environment and not realize it.
  • Show Mesh: shows and hides the camera mesh
  • New Mesh: This will reset your meshes (both the camera and projection) to nice new square grids of the specified height and width, obliterating any mesh calibration you have done previously
  • Recalculate will force a recalculation of the distortion map, you shouldnt ever really need to do this. I should probably take that button off the interface.
  • Flip Horiz/Vertically: this will flip your camera mesh around.
• Projection View:

  

  This view is used to calibrate your projection (or whatever you are using to see your touches on. It has the same mesh options as the camera view (except you cant flip it around)

• Blob View:

  

  The Blob view allows you to see the blobs that are being tracked at any one time. It will scale itself to show as big of a blob as it can (ie if you have just one blob, it will take up most of the view, if you have all ten fingers on the surface, the individual blobs will appear smaller. Use this view to calibrate your threshold and adjust the tracking settings. (more on that later)

  
  • Show Raw Blobs: if this is not checked, you wont see anything. (you wont see anything if there are no blobs either)
  • Threshold: this is the main control for making sure that you get the blobs you want and not a big mess. This controls how bright the blobs need to be in relation to the background to be considered 'real blobs' and not noise. the specific value will change drastically based on your lighting conditions, however, if you find you are down below 5 or so, then there is something wrong. (generally speaking)
  • Blob Matching Distance: this is how far apart (in pixels) two blobs need to be to be considered separate. In other words, if you move your finger faster than this amount in a single frame, then the tracker will think you picked up one finger and put down another one
  • Blob Jitter Distance: this is how far the center of a blob needs to move to be considered moving. Anything less is considered noise
  • Find Dark Blobs: By default BBTouch is looking for light blobs on a dark background, if your setup is the opposite of this (like the MT Mini) then you will want to be looking for dark blobs
• Test View:

  

  This is just a pretty little nothing (and not even that pretty really). it allows you to check and see if the meshes are set up properly and is kinda fun to play with. It has no settings (yet)

The last few controls are always visible:

• Detect Blobs: this turns on the blob detector. If there is some problem and the blobs are taking too long to process (like the threshold is too low, or the lighting has changed and everything is showing up as a blob, then it will automatically stop blob detection. If your blobs 'stop' for some reason, check here first.
• Go Full Screen: this will take whatever view is the main view and make it fullscreen on the display selected. To get out of the full screen view, make sure the full screen window is selected (ie is the key screen) and hit ESC. this will put the view back into the main view spot.
• TUIO Host/Port: If you are using BBTouch to generate TUIO events, then put the hostname or IP number and the port of the TUIO server you are trying to connect to.
• Generate TUIO Packets: This will start generating TUIO packets, sending them to the specified host and port. If you want to change the host or port, you need to disable packets.

Calibration

Calibration is a bit of a pain, but it is unfortunately necessary. Calibration is done in a few easy steps:

Calibrate the projection. We need to know what it is you are looking at first
Calibrate the camera. We need to know how the camera image relates to the projection
Set your tracking settings. We need to make sure we are getting clean blob detection
Test it all out. Make sure that when a blob is detected and projected that it all lines up properly

Astute readers will notice a close correlation between the number of views and the number of steps involved in the calibration process.

Calibrate the projection.

Switch to the Camera view. have a quick look at your camera image, is it wildly distorted? (like the one in the examples) or are the edges more or less straight? If your image is very distorted you may want to use more mesh vertices. if it is not so distorted you can get away with less. However, more never hurts. Switch back to the projection View.

Choose a nice number of mesh vertices. I suggest 5 high and 6 wide is a good start for my surface. Hit New Mesh, and you will get a nice new mesh.

Before we go projecting this onto our surface, lets get familiar with the mesh manipulation tools.

You can move the individual vertices simply by dragging them.

If you hold down the Shift key (make sure that the view is active) then you will see all the group manipulation tools.

If you drag the handle in the center of the edge of the bounding rectangle, you can scale all the vertices in one axis at the same time. Similarly if you grab a corner, you can scale the vertices in both axes at once. if you grab anywhere in the center of the mesh (the hand cursor) then you can move all the vertices at once. And if you go a bit further out from a corner, you will get the rotation cursor and you can rotate all the vertices around the center.

OK, now we are ready to project! Choose the monitor that has your projector and hit 'go fullscreen'.

Depending on your surface, you might see something like so:

You will notice that the projection mess falls off the bottom of my surface. This is because my surface is really wide, I am probably losing nearly a third of my projection image. I will have to stretch the projection mesh to fit.

First I will scale it down:

then stretch it out to cover my working surface.

there is still edging to go around the projection surface, that is why the mesh doesnt stretch all the way to the edges. That and I have an IR shadow on one edge that you can see in the following camera views, which also limits my 'active area'. You might be using your entire projection (i should totally add a button to scale to screen automatically. hmmm.. anyway)

Great! now we move onto the camera calibration.

Calibrate the camera.

there are two ways to do this. I will go through the easiest one here, and then touch on the other (just as easy really, just a bit more work) way.

For the easiest way you will need to be able to remove the IR bandpass (or visible light) filter from your camera. so that your camera can see the projected image. After you have removed the filter, press the 'f' key if you are still in the projected view (or click on the freeze frame in the camera view settings) Hopefully you have captured a nice image of the underside of your surface. Feel free to put your filter back on now.

You might be seeing something like this:

From here you just need to match up the camera mesh with the mesh that you can see on the surface. First off, have a look at your mesh. Notice that on two of the corners there are shapes, a circle and a square. There are also a circle and a square on the projected mesh. you need to make sure that those 'keyed' vertices match up properly. In the example above, we need to flip the mesh horizontally. (use the flip horizontal button if you don't want to drag each individual vertex to the opposite side) And we also need to stretch the mesh down a bit and get it close to the projected mesh size.

Awww, thats close enough eh? Now for the grand finale: dragging each vertex to match the projected one.

Halfway done!!

Now we are all done, camera mesh is calibrated!

Tracking settings

Up until now it has been kinda boring. Now we get to start doing some interesting finger-tracking and whatnot. We are going to use another freeze frame, but this time, with a hand or some other object that you wish to track.

like so:

(side note: you can turn off the mesh at this point if it annoys you, and WTF is that black rectangle? It is defined by the mesh you just calibrated, and the rectangle is the smalles rectangle that encloses all of the mesh vertices. It is the 'Region Of Interest', or the place the detector looks for blobs. The detector draws that single pixel border around the ROI. It is kind of like a logical fence to keep the detector confined to the interior. Since it is drawn directly into the pixel buffer and BBTouch is not making any copies of that buffer (too slow) then you will see it whenever you are detecting blobs. )

OK! now we have a nice freeze frame of our hand, (or whatever) switch now to the Blob View

Wait, what the crap is that? More ROI technobabble? well, if you see a big rectangle like this one in your blob view it is a good indication that you need to resnap your BG. (this should be automatic, but sometimes these things slip past my high quality testing group. This is actually caused when you start blob detection, then change the mesh. It is kinda a bug, and I will fix it at some point :-). In the case of big blobby rectangle, go back to camera view, hit resnap bg, and then get a new freeze frame.

set your threshold to something like 75 or so. then turn on blob detection. If you dont see any blobs, start clicking the down arrow on the threshold clicker. I actually get my first blobs at 59 today.

I dont even know what that is! and in any case I am hoping for five nice blobs, not one crappy one. so keep clicking down. (note: do not hold down the clicker arrow or you will most likely time out the blob detector and wonder why nothing is changing. be patient. click click click) As your threshold gets lower, you should begin to see 'real' blobs taking shape.

once you have some nice blobs, you can keep going down until you start to see false positives (like your palm, or your arm, or your cat). I would suggest setting the threshold well above the false positive mark, somewhere in the middle of the range where you can see all your fingers.

If you want to refine the threshold to its optimal value; move your hand all around your surface freezing the image and repeating the above. Make note of the false positive threshold and the threshold where you lose one of the blobs (false negative). after you have an idea of how your whole surface behaves pick a nice threshold that works for all the areas. (like the average or something)

Test it all out

Ok, you are basically done. Now, if you want, you can load up the test view. (if your projection view is still full screen, just click over there and hit ESC) Otherwise load up the test view into the main view spot and go fullscreen on your surface. (make sure you have blob detection on)

You should be able to put fingers on the surface and have the blob targets show up. drag them around. If you are dragging and the blob number keeps changing then your threshold is probably a bit too high. If you get lots of extraneous blobs, then it is probably too low. if the blobs dont match up with your fingers then your meshes are off, and if the blobs move in a different direction than your fingers then your meshes arent keyed properly.

Hopefully however, it is all groovy!

happy mutli-touching! (ewww)