Tech Interview: Titanfall on Xbox 360

Very scary! This was the hardest project Bluepoint's ever done, and super high profile too. We really didn't want to screw this one up! In the end though, it basically comes down to getting your head down, avoid the shiny distractions and produce the best job you possibly can.

We had a mysterious meeting scheduled with EAP at GDC March 2012. Wow, that was a long time ago! They wouldn't say what the project was before the meeting, but that it was a big deal.

"This was the hardest project Bluepoint's ever done, and super high profile too. We really didn't want to screw this one up!"

The meeting was in a suite at the fancy Four Seasons hotel in San Francisco, and I remember being pretty nervous and talking too much, though that might have been too much coffee! It must have gone OK though, as a few weeks later we were off to talk to the Respawn engineers in LA about the project.

Cue another nervous meeting (where I definitely drank too much coffee!) where we discussed how BP was going to fit 5GB of assets into 500MB on the X360. I think the general response from Earl and Richard was along the lines of a positive, 'Sounds like you guys know what you're talking about' and, 'Well, we don't think it's really doable on 360, but yeah, good luck with that.' Got the gig though!

No.

Well, now I think about it... actually, it's still no. [Laughs]

I'm selling the Source Engine a bit short here, it's got a lot of cross-platform support, and we definitely didn't start from scratch on the 360 port.

The modified version of Source that Respawn are using did run at one point on the Xbox 360, but the DX9 code path was pretty knackered by the time we got to it. All the 360 rendering code was still there, disabled, along with the fundamental stuff you need to get something running cross-platform. It was basically a heavily modified version of the Portal 2 codebase that just ran on high-spec PCs.

Things started getting interesting when we finally got the thing to compile and boot on Xbox 360 with the realisation that half of our memory was gone without loading any models, textures or maps!

So, the Source Engine is around 10-12 years old or something like that, and has had a lot of changes made to it over the years, so you can't just throw it all out and start again from scratch with the expectation that you'll rewrite the entire thing along with perfect copy of game in 14 months!

We started to figure out how hard this was going to be the first time we got the game running on the 360; with a single player, our average frame-rate in Fracture was around 5fps, and that's without any textures. That's an, 'Oh bugger, what have we got ourselves into?' situation that you're not going to fix by giving the engine a bit of a tickle.

Long story short, we've replaced the world renderer, collision system, visibility system, animation system, asset system, asset pipeline, audio system, stuffed in a streaming system and compressed the crap out of the assets to make it fit on a DVD. Yeah, we're not running the vanilla Source renderer [laughs].

We basically ended up with a crazy Frankenengine by the time we shipped.

Actually, it's kinda the other way around. You're basically integrating your tech into someone else's game.

You need to keep the gameplay of the original intact, so you make the minimum number of surgical changes to the existing code to make it work well on another platform.

In this case though, I'd say the changes were less surgical and a bit more like wholesale amputation [laughs].

Source is such a big codebase, and a large amount of the code is dedicated to game logic and entity systems, so the bulk of the compiled code would still be considered part of the Source Engine.

That's kinda important because these systems include all the fun bits from Respawn that everyone plays!

I'd say that fitting everything in memory was the hardest problem we faced getting Titanfall running on X360. We started off by putting Source on a diet, stripping things out like the local server, moving away from the DLL system so dead code stripping would work, etc. That brought down the fixed overhead and allowed us to get the game running. But as soon as we started loading assets we were still struggling to get it in memory on a 1GB devkit with minimal textures and no sound.

We knew that if we started changing the core assets too much, we'd be in real trouble towards the end of development. Generally, code and asset changes tend to accelerate towards ship, so we'd have been stuck with trying to redo work at the last minute. We also didn't want to compromise on quality, so it seemed like a no-brainer to focus on making the existing assets fit somehow instead of chopping them to bits.

So normally, a really large amount of the memory budget on a game is taken up by textures, which it was, but we also found that animation and other systems data were surprisingly large. I could bang on about engineers battling decade-old animation systems, tech art trying to figure out if this 4MB animation really was used anywhere the game, and many, many spreadsheets from our awesome QA and production team, but suffice to say, a lot of really talented people worked really hard to make this fit.

Top and bottom of it, Respawn were pushing the Source Engine to its limits, and we just had to make it fit. It doesn't matter how pretty the game looks on a devkit or all the clever algos you came up with to make it run fast, you've got a hard limit on memory and you can't ship it if it doesn't boot on a retail console.

Good question! The Source Engine was written back in the days before dual-core CPUs were mainstream and was refitted later by Valve to support multi-threading. Respawn made a lot more additional changes to take advantage the extra CPUs in the Xbox One, but our three PowerPC cores aren't exactly a 1:1 match for the CPUs in the Xbox One.

The other problem is that threading is inherently difficult, and even if you get some massive wins, you're still going to run into Amdahl's Law. This is a fancy way of saying that you're only as fast as the slowest task in the system, or putting it another way, that bloody single-threaded raycast across the map is going to cause you to drop to 20fps no matter how much you thread the engine.

We ended up with a dual approach for optimisation: remove fixed CPU overhead to boost the overall performance, and find stuff that causes CPU spikes to raise the lower end.

Surprisingly, while we did get some good benefit from the threading in the Xbox One version, most of our gains came from reducing cache misses, a pretty typical weak spot on the Xbox 360 and PS3.

It's the same. Well, actually, we did cheat a bit. When you play Last Titan Standing, which is crazy stress on the CPU, we dial down some of the grunt spawn in rates to reduce network packet processing and entity simulation load on the client. We only do this for LTS, because when you've got 12 giant robots bashing the crap out of each other, you tend not to notice the absence of things to squish underfoot.

All the other changes are cosmetic or optimisation-related, such as reducing the number of client-side flyers in Boneyard and Airbase, and switching out the, 'Can I see that guy on the other side of the map so I can draw his name?' deferred raycasts with GPU visibility queries.

Yeah, that an interesting one, though I wouldn't call it extreme, because that's exactly what we would do if we had gobs of memory!

Anyway, you can't really compress textures much beyond DXT1 in memory and still have them immediately usable by the GPU. There are a few tricks you can do with assets that are really memory-hungry - we used the old vertex colour trick to reduce the memory footprint of the skyboxes - but aside from a few tricks, you can't dial texture quality up and down like a JPEG email attachment.

So, there something like 600 materials and 2000-odd textures in the Fracture map alone. That's a lot of material variation, and one of things that I think sets Titanfall apart from other games. It's also a real nightmare on Xbox 360 due to GPU memory alignment restrictions etc.

We've got enough memory, including fixed overhead for UI and dynamic model textures, for 64x64 resolution DXT1 on the Xbox 360 with all those materials. That's N64 quality, and that's really going to be problem. So, to keep the texture quality up, and to squeeze something like 1GB of DXT1/5 textures into an 80MB RAM cache, we stream the textures in dynamically on top of super low res version.

So, in a single-player game, say like Metroid Prime, you enter a new area or elevator or something, and you quietly stream the new textures in, and 'bing', the door opens when they've finished loading.

Getting this working for Titanfall multiplayer was a bit trickier; the maps are massive, there's loads of interior detail and tunnels, and the player can double jump and run like crazy. Not to mention those giant robots again, and, 'Eject! EJECT!' etc... You get the idea.

So we have to be able to load these high-res textures into a 'tiny' cache, really fast. Basically, we're always streaming like crazy, and to get the best streaming performance we stream from the HDD and DVD at the same time. We also stream UI textures and audio, and keeping everything prioritised correctly while still avoiding the overhead of seeks is pretty tricky.

For the technically minded, we stream medium-resolution textures from the HDD, which has the best seek time and can load textures quickly. The top texture mips are streamed from the DVD with some additional compression for throughput. Also, having the top mips on the DVD avoids us having to copy them to the cache partition, which is good because we're pretty tight on space there!

This is why there's no Games on Demand download available as we didn't want fob people off with an inferior experience. Similar deal for 4GB Arcade kits and USB sticks, flash devices have much better seek times than a DVD, but the throughput is lower. For lot of games, the seek time dominates loading, but on Titanfall 360, we worked really hard to optimise loading for DVD/HDD, and running off flash just didn't give the same performance.

So, we copy the texture-streaming data to the Xbox 360 cache partition, which is located on the fastest, outside part of the drive. I'm pretty sure this is what Halo is doing when it's showing the pretty particle effects before you start a new level. Also, using the cache partition guarantees that the installed files aren't fragmented, giving really consistent streaming performance. I tested it at home on an original 20GB HDD that I've had since 2005 and never re-formatted, it works great!

I think people are used to installing games because generally they load faster from HDD, but in our case it just increases load times and degrades streaming performance.

It's a bit of a no-brainer really, you've got a limited amount of eDRAM, 10MB, and the Xbox 360 does MSAA for free if you can fit your render targets within that 10MB limit. The maximum resolution render target for 2xMSAA is around 1040x1200 which, with the tiling restrictions, uses about 95 percent of the 360 eDRAM. This gets resolved to the 1040x600 that you see upscaled to 720p on your TV.

Actually, no we didn't. We did make some performance tweaks to the particle code to optimise for Xbox 360 write-combined caches and tweaked the threading a bit, but the core code is pretty much the same as Respawn. We also tuned some of the effects for performance; Respawn really helped us out here by reworking a lot of the particle assets to perform better on the 360. Thanks Robot!

Actually, that's another advantage of the X360 eDRAM and the lower resolution buffer. We did tweak the particle shaders slightly for performance, but the X360 has some pretty tasty fill at something like 4Gpix/s. Gives us a decent amount of screen fills to handle effects overdraw at our 'more than 540p' resolution [laughs].

Well, we couldn't afford the same dynamic shadows as the Xbox One version, as we'd have to render the scene again from the viewpoint of the shadowcaster; there's a lot of geometry in those maps! Fracture has something like 22 million triangles for static level geometry. We had the VRAD-baked lighting for static geometry that was used for objects in the distance on the Xbox One, but the baked occlusion was pretty low-res and we needed more fine detail otherwise the shadows would have been really blurry.

I don't know if I've mentioned this already, but we didn't have any memory [laughs], so we couldn't just increase the resolution of the occlusion textures. Instead, we used an offline renderer, Mental Ray, to create super high-res shadow maps from the map data, something like 16,000 pixels width and height. We converted these high super-high shadow maps to distance fields, and voila, shadows that still look sharp at a much lower memory cost.

We'd set a 30fps cap for large scale test, and it had 'stuck' due to concerns about tearing with a variable frame-rate. While we were working on the day-one patch, we snuck in a 60Hz on/off option and didn't tell anyone about it. It didn't take Respawn very long to find it, and they much preferred the feel of the unlocked frame-rate like we did. EA found out about it a bit later [laughs].

There's also part of this that's kind of a statement of intent, because if you lock the frame-rate at 30 for too long during development for demos or stability etc, 30fps becomes the expectation and you'll never ever hit 60.

Later, we did a poll across Bluepoint, Respawn, and EA-QA and small but significant group preferred the frame-rate locked at 30fps, so we left the option in but changed it to 30Hz, defaulted to off.

Absolutely! We'd much rather be running at a frame-locked 60, but we're high enough above 30fps that the improvement in the feel of the controls when the frame-rate is unlocked outweighs the downside of seeing ugly videos on YouTube. Two words, Nyquist frequency!

No, not really at all, though we did manage to get the option backwards at one point I think! The server is authoritative, and the client really only performs prediction/interpolation, so you're unlikely to get the kind of problems you might see if all the logic was running locally with inconsistent time steps, such as your physics solver having a meltdown.

A lot of the big wins for the GPU came from looking at performance from a very high level. This sounds obvious, but the fastest pixels are the ones you never draw, so we found ways to not draw stuff if you couldn't see it. The Source Engine already has a PVS system, but it's BSP-based and didn't work very well with the kind of maps used for Titanfall.

Respawn wrote a new PVS system that worked dynamically, likely because that's much better for iteration but it need additional CPU performance we haven't got on Xbox 360. We took a different approach and pre-calculated static visibility using a big farm of Xbox 360s; these ran multi-hour visibility calculations which reduced the amount of stuff the 360 had to draw without requiring runtime computation and allowed us to use mostly the same geometry as the Xbox One version.

Same deal for shadows, pre-calculate. Also creating occlusion geometry to reduce pixels shaded, tech art to the rescue there getting those built!

For the CPU, it's a bit trickier as there's lot of code. We got some good CPU gains by using the GPU for culling and LOD calculations instead of using the CPU. Then we moved onto more traditional optimisations, such as finding code that is obviously slow and speeding it up, eg rewrite the animation system in VMX.

Once you've optimised things that are obviously slow, it gets a bit trickier, as most of the time nowadays, poor CPU performance is death by a thousand papercuts.

The real story is that it runs well because of loads of hard work by the team. Basically a talented group of guys and girls working really hard.

Yep, exact same as Xbox One/PC, aside from some minor back-end changes for encryption, and those grunts we spared from an unpleasant death in Last Titan Standing mode.

I'm not a network programmer so take all of this with a pinch of salt.

So, it does help quite a bit, though we're mostly CPU-limited and even the overhead of packet processing and simulation is a hit on Xbox 360.

I think the big deal with having a dedicated server is that you wouldn't even try to make this game using a standard peer-to-peer model, at least on console.

The server performance requirements for Titanfall are pretty high, so you can't run peer-to-peer. This also means you can't have a many virtual machines per physical server, meaning if you've got a popular game, then you're going to need a pretty big investment in server hardware.

So, this is kinda hard to explain in a 'the power of the cloud' soundbite, but using Thunderhead actually makes a lot of sense as it allows pooled resources to spin up and down for a given title rather than having to figure out some way to get some ungodly amount of servers for a day one launch player spike. I think you'll see more interesting multiplayer games because of this but it'll take a while.

It's difficult, like trying to change the transmission on car while it's still moving. We share code via Perforce, but we work in a different code branch. The hard part is to add features for Xbox 360 without breaking the PC or Xbox One version and we do this by a process called integration, where we work on a known stable code for 2-4 weeks then bring new changes over from the main Respawn branch. We had a good bit of practice working like this with Superbot on PlayStation All-Stars Battle Royale for PS Vita.

Respawn basically trusted us to get on with it, and I think it worked out well for both teams. Luckily for us we weren't doing cross-platform multiplayer on Titanfall; that would have been really difficult!

The other part is the general production kinda stuff you get on large projects, scheduling, conference calls, bug databases, lots of email! And general production stuff is hard! Generally doing production is a pretty thankless task as the best producers make problems invisible to the rest of the dev team. A lot of the time devs only hear about what's busted, wonder why we're getting dinner from that restaurant again, and don't get hear about all the problems headed off at the pass. Production on both ends did an awesome job.

I like to point out that our production team was better though because they also bought everyone ice-cream on Saturdays [laughs].

So, the number of code and content changes always accelerates towards the end of the project, and we're downstream from Respawn and obviously need to bring those bugfixes and tweaks over to the Xbox 360 version. We did some frontloading of work knowing this was going to happen, but it's still rough.

So yeah, everyone at Bluepoint put in some serious work to get the game shipped and spent far too much time at the office working overtime.

It's an online game, so unfortunately it's not like the old days where you stick a disk in the box and it's done. Daryl, our producer, and Peter, our tech director, are up in Seattle along with EA Partners and Respawn making sure everything goes well as we launch in different regions.

We're all a bit knackered after shipping Titanfall so lots of people are rotating out and taking comp time before we bring over the DLC.

After that, we've got something tasty lined up, but if I told you, I'd have to kill you [laughs].

The reaction to Xbox 360 version has been amazing. It's the best we ever could've hoped for. It's always scary putting something out and you think it's good but you don't really know for sure. For all the tech stuff we do, putting something out that's fun is the real job. People seem really happy with our version of the game; they're having fun, and after all is said and done, making something that people appreciate and enjoy is the real reward.

Though I must admit, I'm really looking forward to a vacation and some beers!