Inside Digital Foundry: What Grand Theft Auto 4 did for us
How the Rockstar classic shaped the way that we look at games.
We're just three days away from the release of Grand Theft Auto 5 - closer to two if you're planning to attend a midnight launch for the year's most eagerly awaited game. At Digital Foundry, we're aiming to bring you our findings on both the PlayStation 3 and Xbox 360 versions of the game as soon as we can, but in the meantime we thought we'd take the time to pay tribute to its predecessor, not least because so much of what we do and the way we approach games today hails from the seismic release of GTA4 five years ago, and that Face-Off we produced at the time.
Way back in 2008, it was early days for Digital Foundry on Eurogamer. A couple of years earlier, at the dawn of the HD console era, we developed our own direct-to-disk capture solution, giving us complete access to lossless digital feeds from the HD consoles that formed the basis of our comparisons - but it was the arrival of Grand Theft Auto 4 which made us wonder what more could we do with this material aside from the standard screenshot galleries.
Access to these assets opened the door to techniques like pixel-counting - a form of analysis first discussed on the Beyond 3D forum. Here, long horizontal and vertical edges are isolated and analysed, with the ratio of rendered pixels compared to actual screen pixels, giving us the dimensions of the final framebuffer before it is scaled to 720p and dispatched via HDMI to the display. We could tell from screenshots that the Xbox 360 and PlayStation 3 versions of GTA4 ran at different resolutions - native 720p (with 2x MSAA) on the Microsoft console and an upscaled 1152x640 on PS3, with a blur filter in effect.
Pixel counting is an example of how a proven methodology could provide actual metrics to better inform our articles. However, beyond that, our understanding of how games worked was still very much in its infancy and we wanted to do better: GTA4 was the release that made us fundamentally reappraise the way we looked at games, and how we would present them on the page - and it was almost certainly the first console game that underwent the now standard performance analysis.
Back then, there were no frame-rate analysis tools and certainly no means by which to plot performance over game video in the way we do now - the fact that various outlets were confidently saying that both versions had the performance advantage didn't help, so we decided to do something about it. The general theory was simple enough: given a pure digital output, we would assume that dropped frames would be mathematically identical to the last unique frame to emerge from the HDMI port of the source console: count the unique frames, count the duplicates, and you're on your way.
For GTA4 analysis back in the day, our tool was primitive, taking the form of a tweaked version of Loren Merritt's DeDup filter for the AviSynth video scripting language - originally designed to remove identical frames from any given video stream. Instead of actually removing them, the revised version we worked with simply logged the percentage difference from one frame to the next in a text file, with a zero per cent reading signifying a dropped frame. Internal running totals were kept to work out the frame-rate average which would pop up at the end.
"Grand Theft Auto 4 was the game that inspired us to develop performance analysis for console games, way back in 2008. The initial tools were hopelessly basic but still managed to provide the raw data we needed."
Two problems immediately became obvious - Xbox 360 frame-rates seemed a little higher than they should have been, while PS3 results weighed in at a locked 60 frames per second! What we learned here is that some games on the Sony console (and all of them, if you own a Slim or Super Slim PS3) add a small dither to the HDMI output, meaning that the duplicate frames aren't 100 per cent identical to the last unique. To solve the PS3 issue, we added a threshold function which added tolerance for what constituted a duplicate frame - 0.5 per cent did the trick here.
On Xbox 360, the slightly skewed results came down to two factors - a completely unlocked frame-rate and the inclusion of screen-tear on a small minority of frames. In the case of GTA4, these frames should be interpreted as duplicates, but as they were different to their predecessors, they were processed as uniques. The fix here was effective enough: as the tearing was confined to very specific areas, we'd simply crop those areas out of the video data being analysed - something that is easily done in the AviSynth scripting language. It was a primitive fix and wouldn't work for most games but it did the job here. With those issues resolved, we went to press on GTA4 with the very first performance analysis on a console game - the idea being to resolve the frame-rate furore once and for all with actual hard data.
It would be a long time before we could consistently include performance analysis in all of our articles - principally because screen-tear was so hard to track with our tweaked "DeDup" tool. But more importantly, we simply weren't happy with the metrics that were produced. Most console games cap at 30fps, dipping beneath when the engine is under stress. So the fact of the matter is that the longer your sample clip, the more likely it is to hit the 30fps cap, and the closer the averages between two Xbox 360 and PS3 versions will be. At the extreme, one version could lock up for several seconds, but that event wouldn't be evident from the final average at all. Indeed, if your sample clip is long enough, it could be all but wiped out in the averaging process.
What we learned from this is that while metrics may have value, context is everything. Average frame-rate metrics may work on PC - where benchmarking is carried out with no frame-rate cap - but ideally, you really need to see how the game performs in motion and where the problems manifest. The next logical step was to move away from context-insensitive numbers, and simply show how the games worked in motion by overlaying the current performance over the actual footage being analysed. Rudimentary DOS command line tools evolved into the standalone processing tool we have today.
"Head-to-head video is taken for granted today, but it was the arrival of GTA4 that set the template for our approach to comparison assets."
What hasn't changed quite so much from the GTA4 days is the way that we present head-to-head comparison videos. At the time, we were keen to show you GTA4 in its best possible light and to present the comparisons in an easy to digest manner. To make the most of our pristine captures, we cropped them down and edited them so that one pixel on the webpage corresponded with one in-game pixel - all the better for representing in-game artwork as you'd see it on screen, not to mention demonstrating the effects of various anti-aliasing techniques.
We then slowed down the video playback so that the meagre bandwidth available to streaming video could produce high quality results. To ensure that quality held up onto the final page, we would encode the final files you see embedded in the video player, using processor-intensive h.264 compression techniques the backend encoder simply would not have time for. Encoding the video on our own PCs meant that the backend encoder in the video content management system wouldn't re-encode the files again and hurt the quality.
In time we moved on to full 720p HD versions of the comparison videos (still encoded "by hand") when bandwidth became cheaper - and these days, YouTube has overcome the vast majority of its encoding issues, making it viable for our work. Suffice to say that the key idea in backing up our findings with the best quality video we could muster all harks back to that Grand Theft Auto 4 Face-Off, which set the template going forward, while the behind-the-scenes tech is still used today, albeit sparingly, for our 60fps video posts.
Even after release, GTA4 continued to influence us - and others, introducing us to further techniques in game performance measurement. For example, as good as the game is, it's fair to say that controller response - especially in the PS3 version - is somewhat sub-par, leading Neversoft co-founder Mick West to choose the game specifically for his pioneering Gamasutra piece on measuring input lag in video games. West figured out that by using a high-speed camera to film the screen and the controller in the same shot, end-to-end latency can be calculated by counting the amount of frames between the moment the button is pressed and the resultant action on-screen.
"We enjoy producing game-based time-lapse videos and the results of our efforts with GTA4 told a great technical story about the sheer sophistication of the stunning game engine."
Even with the lag of his display factored out, West calculated that GTA4 operates with an end-to-end latency of 166ms - ten frames - which would rise to 200ms (a fifth of a second - Kinect territory!) once flatscreen latency was included. You'll note that in West's testing, he's using an internal location - out on the streets, GTA4's performance varies dramatically, resulting in a much more wayward response level from the controls. Looking back at the original Face-Off, the feeling is that while the performance analysis we did laid to rest the controversy at the time about the slower update on the PS3 version, what we didn't explore was the impact on gameplay. Playing the game now, control is clearly significantly crisper on Xbox 360, while the physics (particularly in the car handling) give a lot less feedback on PS3. Returning to the game for this feature, this stood out just as much as the differences in the visuals.
Regardless, later innovations in the area of lantecy analysis - such as Benjamin Heckendorn's latency controller monitor board - allowed for the introduction of much more dynamic testing across a range of scenarios, but the methodology West established here allowed for us to measure gaming lag not just on console games, but on PC, handheld and cloud-based systems too - and you can be sure that PS4 Remote Play latency will be tested in exactly the same way.
Many months after its release, GTA4 continued to surprise and amaze us. After some initial experimentation with Crackdown and a couple of other games, we wondered just how good Liberty City would look if we presented the GTA4 cityscape using time-lapse video. We captured a multitude of first-person perspective views using the cellphone camera mode, recording them at 1fps and then speeding them up during editor, queuing them up with the game's theme tune. The result was phenomenal - an incredible showcase for GTA4's real-time light and shadow systems, the varied NPC AI, the atmospheric effects, weather rendering, not to mention the remarkable simulation of road, sea and air traffic.
Subsequent time-lapses we produced for other games simply didn't have that level of sophistication across all of their component systems. They just didn't need to - the game environments would never be viewed in that way and during standard gameplay, cheaper solutions could be found to produce approximate effects. The fact that GTA4 holds up so well is testament to the attention to detail the Rockstar North team put into its approach to world-building. Indeed, by our reckoning the only other game that got close was GTA4's spiritual successor - Red Dead Redemption - which runs on an adapted version of the same RAGE engine, and clearly features many of the same systems adapted for a very different world.
While we've been waiting five years for the full-blooded sequel to GTA4 to appear, Rockstar itself expanded the GTA4 universe with the release of two DLC packages - The Lost and the Damned and The Ballad of Gay Tony. These were timed Xbox 360 exclusives, but eventually turned up on the PlayStation Network and were also released in Blu-ray format for PS3, bundled with the original game in the form of Grand Theft Auto 4 and Episodes from Liberty City: The Complete Edition - which leads us to some unfinished business we'd like to address before we finally leave the game behind for good, and move on to its successor.
"Did the GTA4 Episodes DLC feature an improved PS3 engine? And were those enhancements rolled back into the original game for the Complete Edition? We decided to find out."
When we first looked at The Lost and the Damned and The Ballad of Gay Tony, it certainly looked as though performance was closer. With the release of the Complete Edition, rumours began to circulate that engine enhancements from the Episodes had been rolled back into the original game, making it the definitive edition for PlayStation 3 owners.
Do these claims hold water? Thanks in no small parts to the methodologies and tools we started to develop in the wake of the original game's release, we can now put the matter to bed.
Beginning with the idea that the Episodes saw a tangible performance boost over the original PS3 game, the answer there is a resounding negative. We planned an identical route through the city in the initial release and in The Ballad of Gay Tony, capturing it at the same time of day - and the result is virtually identical performance, well within the margin of error. On this page you'll see like-for-like video from the game's initial driving section, captured from vintage GTA4 and then revisited in The Ballad of Gay Tony.
However, there is some merit in the claims that image quality has been cleaned up. GTA4 and its episodes spawn you at a hospital location at either 7am or 5pm, allowing us to produce like-for-like comparisons across all three releases. The target resolution of 1152x640 remains in effect across each title, but you'll note that the blur filter in the original game is removed completely in The Ballad of Gay Tony, giving a brighter, crisper presentation (the blur is clearly still present in The Lost and the Damned). Overall then, it's clear that there never was any kind of revelatory increase in PS3 engine performance in the move from GTA4 to the DLC, just some minor tweaking to help improve the overall look of the game in its final DLC outing. Therefore, the Complete Edition wins out as the version to buy simply because all of the content is contained in one package.
All of which brings us to the arrival of the new GTA5. We can't talk about the new game right now, and to avoid any hints inadvertently creeping into this article, we've written it before we've even gone hands-on with either version of the code. For its part, Rockstar itself believes that there's little cause for concern.
"This is probably the first game we've done where both platforms have been running in parallel," Rockstar North president Leslie Benzies told Computer and Video Games, all but confirming our long-standing theory that GTA4 was developed on Xbox 360 first, primarily because dev kits were available that much earlier than they were for PS3. "There's very, very little difference in the development state of them both. They're both very similar. It would be hard to tell them apart."
Backing up Benzies' viewpoint, the world of multi-format development has moved on massively since GTA4's debut in 2008. Where the Xbox 360 once dominated the vast majority of cross-platform titles with tangible image and performance-based advantages, game-making techniques have evolved considerably over the last five years, with developers embracing a multi-core approach that favours both Xbox 360's six CPU hardware threads and the PS3's six available SPU satellite processors - not to mention the six available x86 CPUs available in next-gen consoles. That said, Rockstar has apparently used the PS3 build for all its presentations and demos we've heard about thus far and everything we've seen suggests that the sub-HD resolution of GTA4 no longer applies - this game is running at native 720p. Whether that means it's the version to go for remains to be seen - and it's something we'll report back on as soon as can, using the evolved tools and methodology that its predecessor inspired us to develop so many years ago.