What Went Wrong with Stereo 3D?
Why mainstream 3D is losing traction and how John Carmack could change everything.
It was supposed to be The Next Big Thing - a perfect storm of movies, TV and games. The future of display technology was stereoscopic, bringing a new level of immersion to both passive and interactive entertainment. James Cameron's Avatar set the stage for a mainstream takeover, sporting events looked phenomenal in three dimensions, while 3D gaming got the best possible backing via exceptionally strong support from Sony. Nintendo even launched its new handheld off the back of its glasses-free auto-stereoscopic display.
30 months on from Sony's initial reveal, the picture looks a lot different: 3D movies - often simply converted from 2D source material - are losing traction, TV content remains thin on the ground and stereoscopic gaming seems relegated to "value added extra" status. What went wrong? Was it only ever a gimmick? Is there any future for 3D gaming?
There is perhaps a sense that 3D arrived too early, that current-gen tech couldn't represent the concept to the best of its potential. Regular Digital Foundry readers know all about the technical challenges of rendering in 3D: for true stereoscopy, discrete images need to be created for each eye. On a pure brute force level, this means twice the amount of pixels to generate, and 2x the geometry in order to accommodate the difference in perspective required for each view. This represents an enormous challenge to six or seven-year-old console hardware, resulting in compromise. Impacts to frame-rate and image quality are inevitable in all but the most exceptional of cases.
Even John Carmack, one of the greatest software engineers in the business, seems a touch ambivalent about it all.
"We had to figure out something to differentiate ourselves [for Doom 3 BFG Edition] and I said, 'Well I've been doing all this stereoscopy stuff and VR work.' We can at least do 3DTV support for consoles on there, and both Sony and Microsoft are pushing that," he told Eurogamer at E3.
"I'm not the world's biggest 3DTV booster. It's a little bit of a dubious feature on there, especially when you have to trade on frame-rate for the 3D effect - but it turned out a little neater than I expected."
Which says it all really - 3D is a "neat" bonus feature, a point of differentiation, not a game-changer. The challenges facing stereo gaming are more profound than a lack of graphics horsepower though - the fact is that the vast majority of the audience for any given console game will be playing in 2D. Developers can't be expected to build an experience from the ground up with stereoscopy in mind when only a minority will be able to play it.
Super Stardust HD and Uncharted 3: Console 3D At Its Best
Essentially what we are left with is a game that plays exactly as it would normally, just with a sense of added depth. The impact can be impressive, it can increase levels of immersion to a certain extent (and the effect is seen at its best in NVIDIA 3D Vision where PC owners can power past the technical limitations) but fundamentally, the core game is almost exactly the same.
"I'm not the world's biggest 3DTV booster. It's a little bit of a dubious feature on there, especially when you have to trade on frame-rate for the 3D effect."
John Carmack, id software
Proponents of 3D will point to certain strengths - such as increased depth perception for racing games, but it rarely translates into any tangible in-game advantage: the fact is that both players and game-makers have adjusted to the challenges of representing and judging 3D depth in a 2D plane. In a majority of cases, 3D gameplay simply looks cool, often striking, but rarely with any tangible impact to how we approach and play the game.
There are exceptions to the rule of course. Housemarque's Super Stardust HD (seen here captured in stereo 3D) packs the screen with hazards, to the point where the player can be overwhelmed by the process of depth-sorting the hundreds of 3D objects in view. In this case, 3D serves a fundamental purpose - new asteroids propelled into the gameplay arena literally stand out from the plane the player's ship inhabits. It makes navigation easier, the game feels fairer - and thanks to the developer's substantial engine rewrite, resolution and frame-rate are identical to the 2D game. For this reason, Super Stardust HD remains our favourite 3D title on PlayStation 3. It looks spectacular and improves the gameplay: win, win.
In other cases, the technical accomplishment and the raw spectacle perhaps impress more than the overall gain to gameplay. There's a resolution and frame-rate hit to Naughty Dog's Uncharted 3, but when playing the game in 3D, the sense of wonder and incredulity the core game engenders is heightened still further. The third chapter in Drake's story defies technological limits in 2D - when you see those stunning environments and brilliant set-pieces operating in stereo, the feeling that you're pushing your PS3 hardware to breaking point is almost as satisfying as the quality of the 3D presentation itself.
Above you'll find a stereoscopic capture of the stunning Rough Seas level from Uncharted 3. If you've got some anaglyph paper glasses, YouTube's 3D player does a great job of recreating the effect and the video should also work on some Smart 3DTVs. The "Drake on Horseback" level also looks pretty damn special. In all cases, it's worth selecting the 720p or 1080p options on YouTube's 3D player - bandwidth doesn't really cut it on the lower-quality levels.
With Nintendo 3DS, you would hope that we would be seeing more games uniquely tailored to the strengths that 3D offers - after all, every unit has the required 3D hardware. However, the vast majority of the titles we have played once again follow the same pattern we have seen on existing games: a window of depth is placed into the game world, which can be visually appealing but has very little practical appliance in the interface between player and software. Looking at this stereoscopic 3D capture of Kid Icarus, we once again see an emphasis on spectacle over any actual game-changing application of the technology.
The Miyamoto Approach
That said, Super Mario 3D Land is an interesting example of a more successful stereo implementation - as you might expect when you have one of the most creatively talented R&D teams in the world working directly with 3D technology. It ticks boxes from a functionality perspective - mostly in that platform games are suited to the tech, because ambiguity in the positioning of objects in a 3D world is completely removed, making navigation easier.
But perhaps of more interest is how Nintendo created its worlds and positioned its cameras in order to create a more visually exciting presentation: plunging drops, askew angles, simple gameplay mechanics that play with depth (including delightful Escher-style perspective puzzles) - what we're seeing here is a light, artistic take on stereoscopy that won't break the game when viewed in traditional 2D, but feels interesting and fresh when viewed on the 3D display. It's an approach that is almost entirely at odds with Sony's efforts at a more cinematic, spectacular use for the technology in epics like Uncharted 3 and MotorStorm Apocalypse. If anything, Nintendo's take feels oddly reminiscent of the more subtle approach to 3D we've seen in Pixar's movies.
However, 3DS also demonstrates quite spectacularly that the full promise of 3D gameplay isn't being met. While it provides depth, negative parallax - the process of lifting objects out of the screen and "into your face" - doesn't work quite so well. The illusion is framed by the edges of the screen, and on the existing handheld that screen is very small. There is the argument put forward by some developers that negative parallax is not a good thing, that it brings on 3D fatigue and is little more than a party trick - a quick look at Super Stardust demonstrably disproves this, but the fact remains that using 3D for a more immersive experience is always going to be more limited owing to the field of view imposed by the edges of the screen.
"Super Mario 3D Land uses stereo rendering for a more exciting presentation with plunging drops, askew angles, and simple gameplay mechanics that play with depth. Subtle, but pleasingly accomplished."
We've previously tried to mitigate this by checking out bigger and more feature-rich displays, culminating in our hands-on test of the extremely impressive Sony HMZ-T1 personal 3D viewer, but even the 45-degree field of view offered by the head-mounted display still doesn't deliver on the immersion we really want. There's also the strange disconnect between a 3D image on what the brain knows is a 2D plane - proper depth doesn't feel properly resolved somehow, and often there is the feeling that we are looking at a pop-up book or diorama with various levels of depth rather than a cohesive stereo image.
John Carmack's radical solution in the form of his own custom head-mounted display proved to be one of the most exciting gaming innovations seen at E3. Generally dissatisfied with what 3D had to offer, id software's tech mastermind produced his own custom set-up combining motion sensors with a Palmertech Oculus Rift set-up, offering the first truly immersive 3D experience. There's a 90-degree field of view to the x-axis and a 110-degree aspect to the vertical. You are completely absorbed into the game world. 3D suddenly starts to live up to its potential. Curved lenses appear to resolve most of the "diorama effect" by wrapping the image around your field of view, while custom pixel shaders in-game are mapped to the shape of the lenses to kill any fish-eye perspective issues. The impact is sensational. It's not just the 3D effect you've been waiting for, but the closest thing yet to a fully viable VR experience.
Immersion: As Much About Control As The Display?
"No matter what you do on a screen, you can't have that same level of impact. You can be rendering movies in real-time on there but if you're just watching the screen it's not the same thing as being in there," Carmack says.
"All that we've been doing in first-person shooters, since I started, is try to make virtual reality. Really, that's what we're doing with the tools that we've got available. The whole difference between a game where you're directing people around and an FPS is, we're projecting you into the world to make that intensity and that sense of being there and having the world around you... and there's only so much you can do on the screen you are looking at."
As Oli Welsh reported, Carmack's prototype is astonishing stuff. It works not just because of the screen but because it is attached to low-latency motion sensors that take this concept beyond a mere display technology - immersion is directly linked to player input. Carmack reckons that it would be relatively simple to map the player's position in addition to where he is looking. Peeking around corners, instinctively dodging - movements like this could be replicated in-game. The proof-of-concept demo in Doom 3 BFG Edition is perhaps the genuine game-changer we've been looking for, but it's interesting to note that stereo 3D is only one element of the overall package.
"John Carmack's immersive VR demo solves a lot of 3D's fundamental issues through heightened immersion. It's the first truly game-changing example of stereo tech."
Regardless, reports of the death of stereo 3D - even in its current form - are perhaps a touch exaggerated. The lack of immediate mainstream take-up in 3DTVs will have disappointed Sony in particular, but the future of display technology includes higher refresh rates as standard - refresh your display in the next few years and the chances are you'll be getting a 3D upgrade by default. In time, the glasses issues will be resolved too.
Games based on the "value added" principle of 3D support will continue to be made, and with next-gen processing power, many of the technological issues will be overcome. But 30 months on from our first promising look at 3DTV tech, it's difficult to envisage a scenario where stereoscopic software in this format will genuinely offer ground-breaking experiences. Carmack's concept has changed everything - and it's perhaps not surprising to learn that he is already talking to Sony about VR-related matters.
Regardless of where VR eventually leads us, stereo rendering almost certainly has a fundamental part to play in the future of gaming. Valve's wearable computing concept - based on augmented reality glasses - will surely rely upon 3D. Flat 2D imagery sitting on top of the real world just won't cut it for anything other than a basic Terminator-style HUD. As usual, John Carmack is several steps ahead of us.
"I'm going up to meet up with Valve next month. I actually think that augmented reality has more commercial potential. It's the type of thing where you can imagine everyone with a smartphone five years from now having an AR goggle on there," he says.
"It's something that can really touch a lot more people's lives but my heart's more in immersive VR, creating a virtual world and putting people in there. If anything, this is a little closer to reality than the AR goggles. The Google glass vision video - that's not reality yet. It's where reality may get."