Xbox One backwards compatibility: how does it actually work?
Digital Foundry talks with Microsoft about Xbox 360 support, including the 4K-enhanced Xbox One X titles.
One of the most memorable of E3 reveals also happens to be one of the greatest technological achievements of the current-gen era. Two-and-a-half years ago, Phil Spencer took to the stage at Los Angeles' Galen Centre to showcase the original Xbox 360 version of Mass Effect running on Xbox One hardware. Following an impressive early access period, the backwards compatibility program rolled out in full force, and hundreds of Xbox 360 titles are now available to play on Xbox One, with a clutch of enhanced 4K games now available for Xbox One X. So how was this all achieved? How does back-compat actually work?
Across the years, we've only picked up a few titbits on the process - essentially that the original Xbox 360 PowerPC executables are reverse-engineered into an intermediate, then recompiled into x86. There's also been talk of some level of hardware compatibility integrated into the Xbox One processor to make the job easier, but beyond that, technical details on the process are thin on the ground - until now.
"Basically, we have a VGPU - or an Xbox 360 GPU that we've recompiled into x86 - and we run the entire 360 OS stack," explains Bill Stillwell, Xbox Platform Lead. "We take each game, we recompile it so that it runs, but basically we're running it still in a 360, and the team goes through the game with multiple passes."
In the first pass, the team collects GPU shaders, and what Stillwell describes as 'Enlightenments', a term we hadn't come across before. "Enlightenments are our starting point for making game transcompilation better," he says. "This is more a CPU requirement than a GPU one. Enlightenments tell us what instructions wrote to hardware, function entry points, and so on. Basically, the set of information we need to know upfront when we first build."
Helping the task immensely is the fact that certain aspects of the Xbox 360 hardware design are indeed built into the Xbox One processor - specifically, support for texture formats and audio. "It's what makes this sort of possible for us, because then we can take all of those shaders that we collect and we can package them and all the Enlightenments, and then we just go through and we do actual performance playthroughs to determine that the emulator is executing everything right."
It's not an easy task because fundamentally, the Xbox 360's PowerPC processor is worlds apart from Xbox One's x86 foundation. Floating point calculations need to be adapted from 40-bit to 32-bit, with potential implications for aspects like collision detection, but Microsoft's aim here is clear - to be able to host game code on their virtualised Xbox 360 and for it to run as close as possible to original hardware.
"The way we talk about it offhand is that effectively we try to make sure that the game never knows that it's not running on an actual 360 console, and so whatever value the game presents, we return the value that it expects to get from the 360 hardware and we just do all that in software," Stillwell continues. "So, there's a multiple playthrough session where we have an army of over 100 testers who go through, collect thousands of hours of video data and then from there we'll get back data that lets us tune, and so the emulator gets a little bit more robust every cycle that we go through with game releases."
The ambition in adapting code designed for a tri-core PowerPC CPU running at 3.2GHz on low-power, low-frequency x86 cores is remarkable. In theory, the Xbox 360's six hardware threads could be mapped to specific cores in Xbox One, but issues cropped up as the team took on more challenging games.
"What we found was it's not just the threads, the different cores. In fact, the Xbox One has a thread scheduler - it's similar to Windows in that it apportions based on need whereas the 360 was locked, it was very consistent," Stillwell explains. "And what we found happening was that when we used the Xbox One thread scheduler, we had a whole bunch of issues occur, so we actually rewrote our own thread scheduler... Whenever you boot up that emulator layer so that that logo pops up for the 360, we're taking over thread scheduling from the Xbox One and locking it down to the 360 timing, so that we don't have to deal with the translation across the two or worry about thread management in the Xbox One directly."
But fundamentally, Xbox 360 back-compat works on the principle of an emulation layer. There is some hardware assistance and - yes - some 'secret sauce' (Microsoft didn't want to be drawn on how the emulator supports Xbox 360's VMX128 vector units, for example), but the team is now at the point now where everything an Xbox 360 can do, its emulator can mirror in software. And that's impressive - by the end of the 360 era, developers had leeway to bypass the DirectX API and address the GPU more directly, potentially making transcompilation of code more challenging. Stillwell confirms that this isn't a problem for the emulation layer and in fact, the only thing that isn't supported is XNA.
"I mean, I'll never say never because no matter what we think is the way people will write games, someone will write games in a way that no-one ever thought of doing before," he adds. "We've run into a few of those but in general we had some games early on that we thought would never work that we now know work."
And then there are the games that previously worked, but didn't work very well. Our early tests of Xbox 360 titles running on Xbox One threw up a host of issues - specifically, areas where the emulator's enforced v-sync could see games that ran at 30fps drop down to 20fps. And then there was Halo Reach, a flagship title of crucial importance to the backwards compatibility programme that just didn't come up to the required standard. It proved to be the catalyst that saw Microsoft revisit its testing procedures, leading to existing back-compat titles being revamped with improved performance.
"Yeah, we absolutely did [that] and take some credit for it, actually. It was the review of Reach that sort of put us on the spot. Basically, what happened when we first started doing these games, we got things to run and because we were able to emulate the 360 on the One, we were quite pleased with ourselves and so we started pushing out games and a lot of our testing was subjective," Stillwell explains.
"I mentioned we had an army of testers, but we were pretty raw in terms of how we evaluated the games' capabilities. In a lot of cases, it was just side-by-side playthroughs. And Reach was sort of a key moment for us because we thought the game looked great. When we first started doing Reach, it was a mess. It was one of the most complex games we had done. We've got some early videos where nothing renders right, so when we got it playing and when we started playing it ourselves, it was a milestone achievement for us from the standpoint of the emulator rendering the game properly. Then we saw the analysis of the game and it was pretty clear we had missed."
However, the more titles the backwards compatibility team worked on, the more accurate its emulation layer became. We know that titles like Halo Reach and Gears of War 3 now run much more smoothly than they did previously, but in actual fact, all prior back-compat releases have been revamped not once, but twice.
"Now, some games already run at max capabilities so you don't see it changing them, but other games see improvements in frame-pacing or improvements in input lag that maybe casual players don't notice. And we've done that twice - but predicating all of that work was after Reach was shipped and we realised that we hadn't met our performance benchmarks. We invested a fair amount of money in the team to go build tooling that allowed us to actually do objective measurements of the performance of the game."
And how Microsoft shaped its tooling has similarities with how we work here at Digital Foundry. In essence, being able to match performance data in context with what's actually happening in-game can prove illuminating.
"I forget how many years of video we now have, where we can pull - just like you guys do, right? - we can pull gameplay video, we can see frame-pacing, we can see frame-rate empirically and we can graph it over time, whereas before we were looking at fps counters, but you weren't necessarily able to extract out how much that was just the game and how much of it was our emulation of the game and the differences between the two platforms," Stillwell says.
But this kind of methodology goes beyond improving Xbox 360 emulator performance. The same tooling also proves highly useful in testing Xbox One titles running on the new Xbox One X hardware. Bill Stillwell talks about ERA titles here - Exclusive Rights Applications software, a category that all games sit in (there are also SRAs - Shared Rights apps like the Store, Spotify etc):
"We've spent easily a year just investing in tooling to the point that when we started doing the Scorpio compat - and I'll give kudos to the team a little bit here - when we started looking at Scorpio and ERA games running on the new hardware, we reapplied our same tool set to testing all those games, so that we knew that we could make the claim around compatibility and didn't have any issues with the ERA. The team tested all the pre-shipped Xbox One games on early Scorpio hardware to ensure that we can make a compatibility claim there, and because we've gotten so robust at measurements, we are now able to get very precise in how we tune and what the emulation layer needs to be able to address."
And we're at the point now where Xbox One routinely outperforms the same games running on original hardware, something that was a design goal for the team since the project's inception: "Parity wasn't good enough. We actually wanted these 360 games to run better on Xbox One hardware because we knew we had headroom and we wanted to take advantage of it," points out Xbox Program Manager Kevin Gammill.
All of which leads us onto Xbox One X, which has a range of back-compat enhancements for both Xbox One titles and Xbox 360 games too, which we've discussed in-depth in the past, the highlights being improved performance, faster loading times, 16x anisotropic filtering and even FreeSync support. In addition to this, a range of seven Xbox 360 games have received specific Xbox One X enhancements. The results are extraordinary, utilising what's called the Heutchy Method, a technology designed by Microsoft Software Development Engineer Eric Heutchy.
"When the game renders to its original render target, our virtual 360 GPU recognises the dimensions and scales up the render target, so then we have effectively a high resolution copy of the asset stashed in Xbox One memory," explains Bill Stillwell. "The game doesn't know about that but because we are the emulator, when that surface is used as a texture on the screen we can swap it out with the high resolution version effectively. And these are all in-game assets, but because we're using that RAM cache, that allows us to get assets out faster. This is a case where we're more craftsmen than a factory. We have to go through each game and measure how it changes, especially as we're in a situation where because we're using higher render assets, and because we're doing that 9x resolution increase (3x on each axis), we have to be able to manipulate what assets are shown at what geometry, so that we get the same or better performance with the higher resolution textures."
There are certain principles that remain in place with all back-compat titles. So for example, Microsoft never actually changes a single line of game code, even in scenarios that suggest the opposite. For example, Halo 3's frame-pacing issues on Xbox 360 are fixed on Xbox One - but that's all down to additional GPU power, where the original code was starting to hit limits on 360 hardware. The extra overhead offered by Xbox One simply fixes that issue. Any and all tweaks and improvements are carried out at the emulator layer, though with the X-enhanced titles, game-specific changes are implemented. The results can be extraordinary. Fallout 3 and Halo 3 deliver a huge increase in detail, not just in terms of pixel-count but also from what look like better assets, or even improved LODs. The comparison shots here speak for themselves.
"It's all to do with the way texture filtering works, called mip-mapping. Textures are pre-filtered to create increasingly smaller versions of the same image, known as a mip chain. This is done to prevent shimmering and aliasing on textures and is actually one of the very earliest forms of anti-aliasing in games and is now almost taken for granted," explains Stillwell. "As pixel count decreases, eventually you get to a point where a pixel is either 100 per cent opaque or 100 per cent transparent. Higher levels of the mip chain have more opacity, and we are approximately 1.5 levels higher, so what was previous completely transparent is now partially opaque."
But there is more to it than that. One of the most obvious visual drawbacks in the early Asassin's Creed titles was the low resolution shadow cascade. Remarkably, this is significantly improved in the Xbox One X enhanced version. Similarly, while Mirror's Edges pre-computed 'baked' shadows remain unchanged, dynamic shadows also possess a higher resolution unseen in the original game.
"Specifically, for Assassin's Creed, we have the ability to increase the rendering resolution of any surface used during rendering. In cases where dynamic shadows have more detail, we are rendering the shadow buffer at the same 9x pixel count as the final render target," reveals Stillwell. "The increased resolution means there is significantly more detail in the shadows themselves. We don't do this for every game's shadows because we do not want to impact performance but we were able to do this in Assassins Creed without impacting performance. This same technique applies to Mirror's Edge as well."
Halo 3 and Mirror's Edge also support HDR too, and this is a collaboration between the backwards compatibility team and the Advanced Technology Group (ATG). High dynamic range rendering has been with us across a couple of console generations now, but in the Xbox 360 era, the final result would always be tone-mapped down, typically from 10-bit to 8-bit. ATG's Martin Fuller explains:
"We can take the SDR signal, if you've got the 10-bit in there, and we can look for data that would have been clipped and say, 'Okay, we're going to map a brighter-than-white signal and send that out to the television'. It's pretty powerful, it's pretty effective. Reverse tone-mapping essentially unpicks part of the tone mapping curve applied by the title, meaning that instead of smoothly transitioning to paper-white at the top end, we straighten out that signal top end, and then apply a new curve that takes that top end instead into the HDR range."
In theory, all 360 titles with a 10-bit pipeline could be mapped to HDR, but in practise, it's not so simple. Not all games are like Halo 3 or Mirror's Edge - 10-bit processing may not persist through the entire rendering pipeline, for example. And as you may note from the comparisons elsewhere on this page, sometimes the look of the game changes significantly via this process. While Halo 3 remains true to its original presentation, Mirror's Edge changes quite markedly when played in SDR, as Bill Stillwell explains:
"In the case of Halo, we went back to 343 and brought them builds of each one and said, 'Well, what do you think?' and they said, 'Yeah, that's what we wanted it to be, we just couldn't.' But we've had other developers that have looked at the enhanced titles and have said, 'No, that's not really what we wanted it to look like. Don't use that technique.' The TV will still do some stuff but they generally want the same spectrum that they had set-up before. [Mirror's Edge] was an interesting one too because it's so vibrant and bright. That was one where we weren't sure where the developer would want to go."
A couple of weeks back, we set ourselves the task of playing all seven Xbox One X enhanced titles on the new console, the standard Xbox One and the original Xbox 360. It's been a fascinating exercise - enhanced performance now seems to be baked in, as we discovered with dramatic improvements over prior testing we carried out on Gears of War 3. Despite the 9x resolution increase on X, 360 titles still perform just as well - if not better (in the case of Oblivion) on the new console. Further X-enhanced titles are to come, along with another welcome feature, as Bill Stillwell explains:
"Because this list is curated and we go through and test every game, the other thing that we have found is that some games will get a visual bump that keeps them at parity with how they play on the [Xbox One] S. Other games, if you didn't do the enhancement, would get a performance bump that would exceed what they've done on the S, so we are going to build into the next round - and we'll retrofit it to the old ones - the ability to turn the visual enhancements off and rely on only the hardware improvements and then allow performance to be maximised at what the game's original resolution was."
In a world where Xbox and PlayStation consoles are essentially built with the same technological building blocks, with a software library mostly populated by the same titles, Microsoft's focus on bringing its back catalogue across to modern hardware is a clear point of differentiation with the competition. Even OG Xbox titles are now arriving on Xbox One, again with Heutchy Method enhancements in place - and in that case, both the base Xbox One benefits from a 4x resolution boost on all titles, increasing to 16x on Xbox One X. Meanwhile, even the games of today are now being architected in such a way that could see further improvements on future consoles - something we'll be looking at in more depth soon. But the bottom line is that the ability for your games to live in across generations is clearly important for the Xbox team.
"I know I sound like a PR marketing person, but the nice thing about it that is from the top, from Phil [Spencer] down, the organisation definitely believes that games are just like any other form of entertainment or art. Just because they're older doesn't mean that they're no longer any good, right? The original Star Wars versus the new Star Wars, they're all good movies," says Stillwell.
"It's been a long slog from putting capabilities from the 360 into the Xbox One and then upgrading the capabilities for the X. It's taken years to get to this point. I played Fusion Frenzy with my son the other day and he never played it before. He's 10-years-old, the game's older than he is. This was a game that came out at the launch of our very first Xbox, so I take a lot of pride in the fact that we are part of an organisation that believes it's not the way it used to be and that these are legacies we need to maintain."