Time of Flight Cameras - How Project Natal Works

I have to admit: ever since I first saw Microsoft's Project Natal, I was wondering what they were using "under the hood".

It's no secret there are a lot of "pseudo-3D" technologies out there, but for MS to be announcing it as a truly revolutionary experience, they had to be using something... amazing.

Well, at last - and actually, by chance - I found what they're using: it's a technology called "Time of Flight", and it allows the creation of a real 3D camera!

Forget everything you knew about 3D technologies. This is no "tricked" 3D image analysis or stereoscopic wizardry. I'm talking about real 3D information from a camera, where each pixel represents a well know 3D point in space.

It uses the same principle of SONAR (with sound) and LIDAR (light), emitting a modulated light pulse and waiting for the time it takes for the reflected signal to get back. Unlike those, however, which require mechanical "swiping" methods to create an "image", time of flight cameras use thousands of receptors - each pixel acts as a tiny range finder - allowing to capture a full 3D image in real time speeds.

And, as each pixel is tuned to look only for a specific modulated light spectrum, it doesn't require - nor is affected - by other light sources.
In the end you get 3 images: one where each pixel intensity represent the distance to that point, one showing a "grayscale" image of the scene, and one other showing a "distance accuracy probability" that tells you how "certain" the camera is of each pixel measurement.

Câmara ToF da MESA Imaging

This kind of technology has therefore nothing to do with "simple" image processing done via regular webcams like the EyeToy. Stuff that is very hard to do with a regular camera (like recognizing overlapping moving objects) suddenly becomes possible, in a much easier way.



The only "issue" is that these cameras currently go for well over $5000 USD (though it's quite feasible to make it a lot more affordable once you built it in mass scale as MS intends to do with its Project Natal.)

If you think all the projects the Wiimote enabled enthusiasts to do... Imagine what an affordable sub $100 ToF camera would allow us to do!


Sean G. x Project Natal from Matthew Pantoja on Vimeo.