The Case for Kinect

As we move closer to Kinect's November release date, games developers are talking in more detail - on and off the record - about the new motion control system: what it can do, what it can't do, and what we should expect from the system going forward.

Microsoft itself is stepping up its marketing efforts. This week, two articles (one from T3 and another more impressive piece from Gizmodo) arrived, giving us our first peek inside the Kinect camera and giving us enough technical info to banish the somewhat unkind "EyeToy HD" talk that has dogged the internet since E3: Kinect is a state-of-the-art consumer-level piece of motion capture equipment with voice recognition and biometric ID capabilities and Microsoft wants you to know that.

In the meantime, since E3, Microsoft has shown the system working on TV shows, and rolled out playable demos in Macys stores across the USA, leading to a range of Kinect "fail" videos appearing online, along with attempts at other events to deliberately trip up the sensor with the use of baggy black, reflective clothing amongst other things.

Mileage may well vary, but we've been hands-on with Microsoft's demos and aside from an unfortunate experience for Eurogamer TV's Johnny Minkley with racing title Joy Ride, the system has performed pretty much flawlessly. Within the scope of the "experiences" the game-makers have sketched out, Kinect works, and the best games are going to prove compelling to a cross-section of the gaming audience that probably aren't so keen on the likes of Halo, Alan Wake and Crackdown 2, but more in tune with titles like Wii Sports and Wii Fit.

While core gamers fret about whether you can play Kinect sitting down, game developers are busy making titles that are, for the most part, physically impossible to play from a seated position any way: even the aforementioned Joy Ride has you swinging your ass about in order to drift the cars, and gameplay is for the most part structured into "bite-size" components so you don't get too tired, or spend too long on your feet. It's all about energetic gaming in a social environment - elements of the formula that made Wii such a success.

That said, an investigation into the perceived limitations of Kinect reveals an intriguing truth: the device itself is merely a vehicle for supplying data to the console. What defines most of its capabilities is the software, and that is in a constant state of evolution. Forgetting this is a big mistake in evaluating the potential for Kinect as a gaming platform.

"Every day the technology and software improves, so we're constantly tuning it," bona fide games legend Tetsuya Mizuguchi told CVG, discussing development of new title Child of Eden. "Nothing is optimal at the beginning of a new technology, but it's been getting better and better the more we work with it."

At its most basic level, Kinect - as a device - offers up nothing more than raw data to the Xbox 360: a traditional webcam-style RGB image, a depth map from the 3D sensor and audio from the multi-array mic. The discussion about what the system can or cannot do is mostly about the interpretation of that data: many of the system limitations are actually software limitations, and software can be improved. Microsoft audits feedback from the developers of Kinect titles and improves this interpretation layer in regular SDK updates, so similar to the console development tools themselves, these will gradually become more powerful as the Kinect-specific elements of the SDK are worked upon.

Other developers are steering clear of the Microsoft tools and handling the raw data directly, as Blitz Games CTO Andrew Oliver - currently working on a Biggest Loser fitness game - revealed to Gamasutra this week.

"There are various technologies involved. Some people are using a skeletal system, and it takes a little bit of time to calculate. It's only a split second. We're actually using a different masking system, which can tighten things up," Oliver says.

"But this is all software-based, so where some people might see some little cracks, they're easily fixable by software. That is, the camera fundamentally works and gives you the input; game designers are running forward in a completely new area and learning this stuff. It's like any console. The first few games will look like nothing compared to second and third generation."

It's interesting to point out that Oliver is dismissive of the inherent latency built into the system.

"It depends on what technology you're using. I have seen a few games with a bit of lag, but that is the software choice of the creators; they've programmed it a certain way, and they'll come up with new techniques," he says.

"We will tighten and tighten it. There doesn't need to be a lag. We can get it down to maybe two frames behind, which is pretty insignificant; you won't notice. We're just learning new tricks. Ours is pretty tight."

It'll be intriguing to measure latency in the final version of Oliver's title because as regular Digital Foundry readers will know, a 30FPS game using a conventional controller typically has 100ms/six frame latency at the minimum, and that's without any Kinect processing.

An additional two frame lag is still within the ballpark latency for many pad driven games, but as discussed in our E3 hands-on, the sensation of latency with Kinect is increased by several factors: firstly, the human body as a controller is nowhere near as "quick to respond" as your fingers: the human body itself is "laggy". Secondly, gesture recognition incurs some pretty severe latency, most noticeable in jumping - and there's a lot of that in Kinect Adventures and Kinect Sports, to name just two titles. An on-screen gesture-driven animation can only kick in once the software has figured out what you're doing: it's not 1:1.

In truth, the two-frame claim also seems at odds with the realities of the Kinect set-up - processing the optical feed, producing the depth map and transporting it over USB 2.0 incurs a base-line latency even before the game itself gets its hands on the data. And of course we need to remember that Kinect itself only scans at 30Hz - in pad-driven games, latency leaps from a minimum of 66ms to 100ms when frame-rate drops down to 30FPS from 60FPS. A lower scan rate absolutely translates into higher latency.