Thanks to anyone who stopped by the Unity3d booth at GDC and said howdy to Sandor and I and checked out the multi-touch table: the FishTish.

As promised, the updated uniTUIO scripts have been made available on the xtuio.com site!

They have actually been up for quite some time now, but I ahve been (as usual) either too busy or too lazy to get a post up, so here it is:

Note: they have been pretty much fully redesigned to take advantage of the ‘iPhone Input’ style of touch detection.

What does this mean? Well, the older scripts used a more centralized raycasting and distribution method. There was one main event handler that took each touch point and checked to see if it collided with any registered ‘touchable’ objects. This is a perfectly good design, and it works very well when you have lots and lots of points (in other words, you are only raycasting once per touch per frame)

However, we decided to go away from that and create a fake iPhoneInput class that you can access from the desktop version of Unity.

Why?

Well, the most compelling reason was so that we could quickly and easily port iPhone apps to the touch table. Once I had the faux iPhoneInput class that was being populated with TUIO generated touch information, then it was trivial to take a few of the iPhone apps (and most all of the Unity iPhone tutorial projects) and simply drop them in and have them ‘just work’.

In fact, funny story: On the first day of the GDC expo, I had another Unity/iPhone dev come up and he was excited about the FishTish and thought that his app: iPottery would work really well on it. I grabbed the code from him with the intention of dropping in our scripts and installing it as one of the demo apps on the FishTish. Well, that day was quite a long one and I did not have the energy to actually do anything with the script until the next morning. I spent approximates 15 minutes with a totally unfamiliar code base (iPottery scripts are all in JS and the uniTUIO stuff is all in C#, so that also speaks to how simple it is to use them :-) and I had it working no problems. I spent most of my time deleting the onGUI stuff (onGUI doesnt work with uniTUIO, soory!) and adding some guiText/guiTexture based buttons instead.

For the next two days of the expo we showed off iPottery along with all the other demo apps and it worked great!

Ok, enough of my tangent, how does this actually work?

Well, grab the sample project from the link above, there are two scenes, one called ‘basic touch’ and one called ‘buttons’. Buttons actually contains the entirety of ‘basic touch’. (it just has more buttons in it)

Effectively what the new scripts provide is am iPhoneInput class that you can use in your TUIO enabled MT apps just like you would if you were writing an app for your iPhone or iPad. I have provided a dozen or so scripts that show some examples of how to use the iPhoneInput to do things like basic drag/scale/rotate and build buttons using either GUIText/GUITextures or 3d objects.

I havent had time to put together a video yet, but the code is pretty easy. The only thing that you will need is to make sure that you have an active BBiPhoneInputManager script running somewhere in your scene.

I use the BBTouchManagerStarter script to check to see if there is already an input manager and if not then make one. This is good because if you switch scenes, the input manager does not get destroyed. In an actual deployment, you can just instantiate one in your opening scene and be done with i, but during testing when you want to be able to just look at the scene you are in, this can become cumbersome, so I use the BBTouchManagerStarter in all my scenes instead.

Cheers!
-Ben

Sorry this has taken so long. I had it sitting in my to-do pile forever. And I had figured out the problem, just not fixed it.

Thanks to Morgan at bluecrash.com for sending me an email to kick me in the ass and spend the ten minutes to fix it.

Anyhow,

If you are wondering what the problem was it was that I had forgotten to lock the CVPixelBaseAddress before trying to access the CV frame pixel data. (when grabbing frames off the QTCapture frame buffer)

On Leopard this worked fine (for some reason) but was wrong. Here is the proper way to go about it:

// Lock the base addy for the pixels
// or it will return nil
CVPixelBufferLockBaseAddress(videoFrame,0);
unsigned char * srcBytes = CVPixelBufferGetBaseAddress(videoFrame);
.
.
.
// dont forget to unlock it when you are done
CVPixelBufferUnlockBaseAddress(videoFrame,0);

And that seemed to be the problem on snow leopard. I know, lame and easy. So again my apologies for not just getting it done sooner.

Also, I do plan to release the code for xTouch, but it needs a refactor and a clean first, so that will have to wait. You can get the binary (works on leopard and snow leopard) here:

xTouch.zip

Cheers!
-B

In any case, I managed to stay up very very late a few nights ago and went through the Molecules codebase and extracted the essence into what we are calling the uniTUIO community edition.

If you have not heard of this before, the basic idea behind uniTUIO is to have a nice set of scripts to allow you to easily implement TUIO based multi touch input into any Unity3d application.

If you dont have Unity3d or dont know what it is, then you really should go to www.unity3d.com and get a copy. It is crossplatform and the basic version is free (and it really isn’t very basic, I built a couple complete games using that version (back when it was called the indie version)).

Anyhow, the home of uniTUIO is at the xTUIO.com website. Ultimately I will be publishing posts like this one up there, but for now, since it is still early days and not that many people know about xTUIO.com I will post this here and have sandor crosspost it there.

Currently the uniTUIO CE is comprised of a unity project which has a couple basic scenes and a couple dozen scripts. Also, I made a couple videos showing how to use the various scenes.

There are two basic methods to get at the TUIO event information.

The first one, which is used by the BBCrosshairController and BB3dCrosshairController scripts, is a simple polling method. These scripts have access to all of the touch information for the entire surface and can act accordingly.

These scripts call up to a singleton script: BBTouchEventManager. It is the touch event manager that actually takes the TUIO information and converts it into a BBEvent object, which basically means that it converts the information from TUIO form into a Unity friendly format.

If you are building a ‘traditional’ sort of unity game where you have a central character that you are controlling, then using this polling method makes sense. This is basically the same thing as using the Input class in unity or the iPhoneInput class in Unity iPhone.

One example of this is the particles scene which uses a script called the BB3dCrosshairController which polls for all available events and spawns a particle system under any touch events. using this method you can replicate all the fun ‘fire from the fingertips’ sort of MT demos in about 3 minutes. (or longer if you are like me and love to just tweak the particle systems for ever and ever to get them ‘just right’. Note: the particle system that I did for the examples is not ‘just right’, i just threw some values at it so that you would have an idea of how to do it yourself)

However, in many multi-touch applications you tend to have dozens and dozens of objects in the scene that all need to be aware of any touch events that are interacting with them. In this case it does not make good design sense to have each and every object grab a copy of all the events and try to figure out if any of them apply to that specific object. Instead we have a central distribution point that checks the incoming events against the scene full of objects and sends out event messages as appropriate.

This is where the BBTouchable scripts come in. The BBTouchEventManager takes the raw TUIO events and raycasts through the scene looking for any objects that are both in a special ‘touchableObjects’ layer and have a BBTouchable script attached to them.

The BBTouchable script provides some overrideable abstract touch input handling methods. For the most part, when doing MT apps that deal with lots and lots of different objects that need touch inputs, you will be subclassing BBTouchable to get your custom functionality.

In the uniTUIO scripts I have made a few example subclasses of BBTouchable. one is called BBBasicTouchManipulation, and it provides any object with the basic single touch to drag/double touch to scale and rotate gestures that are very common in MT apps.

The second one, BBTouchableButton shows how you can make a button that lives in 3d space and reacts to touch events.

You should have everything you need to get started with Unity and TUIO now. If you have questions, dont hesitate to comment here or on the xtuio site, or email me directly, or pm me at the unity forums, or whatever.

Cheers!
-b

Since we announced uniTUIO a few months ago, and entered a uniTUIO-enabled app into the Unite09 contest, there has been a flood of emails coming in asking me if we are releasing the uniTUIO source.

So! Here is the answer:

Yes!

But I have to find a few free hours to rub together to get the code into a useable and clean form. We set out with the goal of having a nice framework of scripts to make using TUIO inputs in UNity3d trivially simple. I think we succeeded.

uniTUIO was built during (and for) the Molecules project (which was really a half-dozen different media presentation applications). As the project deadline began to loom, making the Molecules apps became a bigger priority than preserving the purity and re-usability of the uniTUIO framework. As a result, I had to cut many corners in order to meet the deadline (which pretty much always happens) so the uniTUIO stuff is riddled with Molecules specific code which is neither generically useful, or particularly re-usable in any easy way.

So, I need to go back in and refactor the uniTUIO stuff to make it not suck basically.

However, I will explain to you what the uniTUIO stuff includes, so that you can at least not get your hopes up too much :-)

First off, at the low level, it starts with the c# TUIO/OSC implementation by Martin Kaltenbrunner, which everyone can get ahold of here: http://reactivision.sourceforge.net/.

On top of that is pretty much where uniTUIO starts. It is a collection of about a dozen scripts. There are a few static singleton ‘manager’ style objects that start up the TUIO scripts and listen to the events as they come in. These convert the TUIO data stream into some portable ‘event’ objects that include the local Unity environment information. (in other words they convert from TUIO screen-space to Unity viewport space).

Also there is an event manager that does raycasting into the unity scene and picks out any objects that are in the right layer (a ‘touchable’ layer) and passes the events onto those objects.

There are a few more scripts that are meant to be attached to touchable objects.
One is a high level touch event handler that accepts touch events from the event manager and applies them to it’s gameObject in a generic fashion.
Inherited from that object is a basic touch manipulation script that provides simple gesture support to handle the basics like dragging, rotating and scaling a gameObject.

That is what uniTUIO includes. It is meant to be a simple starting place for you to be able to build on for your own MT projects in unity.

So anyway, we will announce it when we release it (hopefully sometime this month) so keep an eye on the RSS feeds. Cheers!
-B

ps: for those of you who are very impatient, have a look at this thread on the unity forums. User Jorgen posted a simple unity TUIO implementation project, which is a good place to get started.

So, it has been a very long time since the last big update to BBTouch. But thanks to Sandor yet again for finding some budget to do some cool projects and including me in those plans. The past year or so I have been immersed in multi-touch apps on a small scale (ie the iPhone). This has kept me quite busy so I am excited to be able to do some bug MT stuff again.

Anyway, enough boring crap about me, let’s talk about BBTouch. BBTouch is great, and it was excellent fun to write and design and work out all the issues. It is still quite a good little tracking app, however there are some known issues. The big one is a crashing bug that crops up only after you have been running the tracker for a long time, (over 20 hours). This has been an issue for awhile and I have been meaning to fix it.
Some other things:

• Multi-Cam support. BBTouch was designed with a single camera in mind, and that architecture is fairly deep. It would not be impossible to make BBTouch a multi-cam system, but it would be quicker to build another tracker.
• Image Filtering: In the early days, before I realized anyone would actually use BBTouch for more than a lark, I wanted to build it so that it would be lightning fast and not require any complicated filtering. However, once people actually started using it for things like professional installs and conferences, it became apparent that I needed to step up a bit. I added openCV filters in the last big update, but they are very very kludgy.
• Fiducials: similarly to multi-cam, fiducials would be a bit of an architectural modification. and now that I have this big hacky patch that is the filtering system, adding fiducials would mean even more work.

So! What is the solution? A new tracker. I have been slowly working on a bunch of changes to BBTouch over the past few months in my limited free time.

The changes started to add up and Sandor and I had been thinking about renaming BBTouch to xTouch for awhile. The name ‘BBTouch’ was meant to be a temporary moniker at the outset (I name all my prototype stuff BBSomething, this encourages me to think of better names later). But instead of just a rename, I decided to make xTouch a new tracker.

What were my new design criteria?

• First and foremost: reuse as much of the BBTouch code as posible. I am a lazy lazy programmer, and I dont like reinventing the wheel again.
• GUI: cut it down. The original BBTouch is way too flashy. At the time, I built BBTouch I wanted to add stuff that made it all cool looking so when people asked me what the hell the giant odd table in my living room was I could have something to show them. Now, however, I have lots of TUIO enabled apps that are much much cooler, so I can just use those like everyone else. So I sat down and figured out the absolute minimum amount of functionality necessary for a tracker configuration and came up with this:
  
  Thats it.
• simplicity: Along with the GUI trim I also systematically went through all the code and refactored and removed all the extra crap. xTouch currently has about half as many lines of code as BBTouch. (presumably once I add in multi-cam and fiducial support, it will be more on par with the current BBTouch)
• modularity: The BBTouch architecture was grown from scratch basically. I didnt really know what I was doing (and still dont!) But I have a much better idea now what needs to go where and how to push all that data around a bit more efficiently. All the new code is much better OO. The various components are much more loosely copupled than they were in BBTouch. This should aid the process when I can get around to adding fiducial support and multi cam stuff.
• multithread support: At the start of BBTouch I was developing mostly on my G4 powerbook. Somewhere along the line I upgraded to the MacBook Pro which is a dual core, but much of the single-core mentality is still in the original designs of BBTouch. BBTouch is very linear in the way it processes each frame. xTouch has a separate thread for each camera and yet another thread for each blob detector. This makes it much speedier on muti-core machines.
• non-obsolete camera support: thie basically meant abandoning the old Sequence Grabber code which has been depricated for awhile now, and going with the newer QTCaptureKit stuff. This makes the frame grabbing code about 500 lines shorter, but the downside is that you do lose the ability to futz with the various old camera settings like exposure and focus.

So! lets have a look shall we?

Main Interface

We already saw the minimal main interface, here it is at load. If you are familiar with BBTouch, then this is a stark contrast. Not that BBTouch was ugly per se, but there was much more going on. Also with BBTouch in order to make the tracker get going, you had to load it up, then turn on blob detection. xTouch is a bit more clever, and it loads up, grabs a few background shots, and starts detecting and sending TUIO commands right away. This means that you can open it remotely and restart it remotely if you need to.

When you first setup your surface with xTouch, you just click the buttons from left to right. Simple.

Camera Selection

Hitting the ‘Cam’ button brings up a little viewer window that allows you to select which camera you are configuring. For now, this is your only camera. Later this window will allow you to pick multiple cameras and configure them separately.

Filter Config

Step 2 in the process is filter configuration. In the early beta version this is tailored to DI setups since that is what I use and what Sandor uses, but it is very easy to add new filter modules which are mostly just swapping in a few OpenCV filters here and there.

Here you have the choice to see the filtered image at various stages of the filter process and tweak the various filter parameters for your particular setup. The way that this all works in the code is much improved over the old way which was really a hack. Now the filter chain is a module that you can easily switch out, so making an FTIR specific filter or whatever will be much easier than previously.

Some quick stats from the beta version:

On my macbook pro (2.33 core duo) the filters run at about 120 fps and the blob detection (with ten blobs) runs at about 150 fps on average. So, all up from frame capture to TUIO distribution: about 66 FPS. (and of course, the camera only runs at 30 fps, so there is lots of time to use your cores for 3d imaging, or whatever it is that you are doing with your touches.

On my mac pro, the filters run slightly faster at 160 fps and the blob detection runs at a staggering 280 fps on average. This adds up to a bit over 100 FPS. So, much better than BBTouch :-)

(note the above metrics were with 640×480 input images. at 320×240, holy shit is it fast, from cap to TUIO: 250 FPS

Mesh Config

Here is where it gets more interesting. Those of you familiar with the way BBTouch worked will either love this or hate it. For BBTouch I had sort of settled on a hybrid configuration style that was most closely patterned from the Reactable folks. It worked very well, and was quite accurate, but the downside was that it was a colossal pain in the ass. lots of dragging of vertexes around the screen, having to take a screenshot with the IR filter off the camera.. ugh. So I finally took the hint and moved towards a more Touche/tBeta/touchlib approach where you just touch a bunch of points on the screen.

Here is the process: First you define the projection bounds by using the keyboard to shift a big box around the screen.

Next you decide how many points you want, the more the merrier (actually, the wider the angle of your camera lens, the more vertexes you will want. I have a very wide angle lens on my unibrain fire-i, and while I was building the config wizard, i was only using 6 vertexes (because, really, when you have to go through the config wizard a zillion times during testing, let me tell you you dont want to have to hit 40 points each time) and was really surprised how far off the points were in the center of the mesh boxes (where they will be the most interpolation going on). So: moral of this story: when you are configging for real, just add a few more vertexes and take the extra 30 seconds, it makes it all so much nicer in the end)

After that you go into the touch-each-point mode where you put your finger on the point till it turns green, then move on. Easy-peasy.

Next! Testing! Everyone’s favorite bit. Just hit the ‘t’ key once all the points are green and you can test out your new mesh. Exciting! (blue circles == touch points)

TUIO Config

Last but not least, TUIO. xTouch is designed for TUIO out of the gates. In BBTouch, to be quite honest, when I started I hadnt really thought that far ahead. It wasnt until later when it sort of ‘became’ something that I finally added TUIO support.

Still only at TUIO 1.0 support (because it is mostly just copied over from the BBTouch code), but I plan to add TUIO 2.0 stuff at some point. Anyhow, unlike BBTouch, xTouch kicks off the TUIO right at startup. Again, no going into the app and turnign everythign on at first.

So! anyway, just a teaser for xTouch really.. no code yet, Sandor and I need to do some more testing before I unleash it on the world.

Cheers!
-B

I must apologize as I have been out of the loop on NUI group happenings for a few months due to high levels of busyness. Luckily, Sandor has pointed me at some TUIO formatting discussions and I am going to try and keep BBTouch up to date with those sorts of things (i fully support expanding the TUIO protocol in a nice standard way, It sounds like tBeta is leading that charge and good on ‘em.)

Anyway, from what i can gather in my 5 minutes of research (reading some NUI forum posts) is that currently some of the Quartz composer and Max plugins that NUI group helps maintain are basically advancing in TUIO protocol technology and BBTouch and some of the other ‘old’ trackers are not working well with them.

This is due to the fact that tBeta is using a slightly newer form of TUIO which includes the height and width of the cursor object in the data stream (ie the blob bounding box) I think this is a great addition and could use it in my own TUIO client apps straight away.

Anyhow, I am going to try and add the height and width params to the BBTouch TUIO stream (in some configurable way so that you can choose to not use them if you are using an older client) hopefully by the end of this week, busyness permitting.

That is all for now!
Cheers!
-b

update your stuff to r158. (there is a new binary on the opentouch code site)

NOTE: Don’t forget that BBTouch requires openCV.framework now. so be sure to grab that as well and stick it into your ~/Library/Frameworks folder. (you may have to make this folder).. It is actually a private framework, and I have been playing around with trying to get it to be part of the BBTouch codebase, but googlecode is not playing nice with it (it claims to be locked, which is possible since I got it from someone else who had compiled it. So anyway, i haven’t had the time to go in and see why SVN thinks it is locked and so it is still not technically part of BBTouch, so you have to install it manually, my apologies)

But! if you do not install it, then BBTouch will brash on load. so if you re having crash on load issues, start with openCV. (BTW I uploaded the version that I am testing against to the opentouch downloads page, so you can get it there. (the link to the right that says “Fairly recent binaries”).

Anyhow, I am in the process of trying to find the time to get around to thinking about updating the documentation to include the above warning about openCV as well as all the new filtery stuff. But I havent had the chance, so tell your friends.

Anyhow this is all running on my prototype table which is a diffused illumination setup, the tracker is BBTouch and it is all using TUIO to communicate. The tech is all core animation using CALayers. there is lots of mipmapping going on to keep the performance up even when there are tons of images, and it works quite nicely even on my macbook pro. BBtouch never tops 50% processor and xPrexxo is similar.

I just commited a few more files, bringing BBTouch up to r157. I had been running 152 all week with no trouble, but Sandor managed to find a handful of bugs (of course, right when he is trying to use it at a show :-) so I made some changes and hopefully it is a bit better.

The biggest changes between 152 and 157 are that I fixed some very minor UI issues where the TUIO settings fields were not being properly disabled so you could change them out from under the OSC objects (causing some odd behavior). But the big one was that in some circumstances BBTouch would boot up and the filter settings (and the dark blobs pref) would be basically disconnected from the detector objects. no matter how much you changed them they didnt really affect anything. In any case, with 157 that is all fixed.

That is all for now :-)

I should be posting the newest BBTUIOTest.app soon, once I get it all back together.

cheers!
-b

Anyhow, r152 is up now with working TUIO again. (like, 3 lines had to change, i suck!)

here is the new binary:
bbtouchr152app