Performance Analysis: Titanfall revisited
The latest patch promises "delicious frame-rate optimisations". Has Respawn delivered?
Two days ago, Respawn furnished Titanfall with its fifth title update, addressing a whole bunch of issues but most tantalising of all, promising significant performance improvements. The developer says that the patch includes "lots of yummy bandwidth optimisations as well as various delicious frame-rate optimisations... lovingly hand-crafted and squeezed into the game".
It's music to our ears. Titanfall remains one of the most intensely satisfying multiplayer shooters on the market regardless of the platform it's running on, even handing in an excellent experience on Xbox 360. But it's safe to say that from a technical standpoint there are obvious issues to address and Xbox One performance stands first and foremost amongst them.
The timing of the new update is also fortuitous in that it comes weeks after Microsoft released its June XDK - a milestone in development as it allows developers to 'turn off' the Kinect reservation on the GPU, and dedicate those resources to gaming. While the timing of the update is right, there's no indication from Respawn as to whether or not the new XDK is utilised. In a previous Digital Foundry tech interview, lead engineer Richard Baker discussed the issue with us:
"They were talking about having it available for launch and I think there were some issues for how it was going to work," he said. "It's not available for launch but we're definitely going to take advantage of that if they give that as an option. And the plan that is they will make that an option, so when it's visible we'll enable it for our game and we should be able to crank up resolution proportionally."
Well, it's visible now, but what's clear from an initial playthrough is that resolution remains at the 1408x792 pixel count of the launch code with the same 2x MSAA (multi-sampling anti-aliasing). Running key clips through our performance analysis tools, our take on the Update Five is that there is a clear improvement in frame-rates, but we do wonder if the average gamer is likely to notice - baseline performance is still mostly in the 40-60fps range, and there are still dips into the 30s when the game is really struggling under the load.
Indeed, there is no game-changing improvement here to the point where some may wonder if these optimisations exist at all. It's a difficult one to test, owing to Titanfall's inherently dynamic gameplay. Thanks to the multiplayer nature of the game, no two sessions are the same, therefore like-for-like testing is simply not possible. Only in the intro sequences to each stage do you get the chance to test performance in this way, but it is not representative of gameplay and rarely is the engine under load in a way in a similar manner.
The exception is one truly brutal sequence - the penultimate level, the Battle of Demeter, played from the perspective of the Militia faction. Kicking off with the troop carrier crash-landing, we have an interactive scripted sequence that tests performance like nothing else. Players and grunts alike storm the city in the company of a Titan that bursts through the gates before exploding. It's a repeatable sequence that can be used to measure performance - which led us to the remarkable screenshot below, showing the Xbox 360 version outperforming Xbox One in the same scenario.
The campaign level select has been removed from Update Five, so capturing this sequence again proved... challenging. However, the results are intriguing. The improvement ranges from nothing at all to a boost of around 10 per cent. So an Update Four 46fps becomes an Update Five 51fps. However, in amongst all of that tearing, it's difficult to tell the difference.
The sequence also demonstrates the challenges of optimisation. It's sort of automatically presumed that more GPU power (via the return of the Kinect reservation) means an automatic boost to performance. But what if it's actually the CPU holding you back? Look at the screenshot below and check out the frame-time graph on the left. This visualises the smoothness and fluidity of the action, measuring the amount of time in milliseconds that each frame is on-screen. In the Demeter sequence we are seeing many big drops in performance. As the game is running with v-sync off and the demands on the GPU are not changing a great deal from one frame to the next, the obvious conclusion is that in this section at least, the CPU may well be the problem.
Indeed, Respawn's Richard Baker confirmed as much in a launch interview with German publication, PC Hardware.
"We are trying to better distribute the particle rendering on multiple processor cores. There is potential power, just like physics calculation," Baker said. "Therefore, the frame-rate drops when tons of simulated physics objects are flying through the area and countless particles are shown. Therefore, we are also working to improve these two systems."
We suspect that some optimisation along these lines has made it into Update Five, but the issue is that once you have addressed one bottleneck, the game may simply find itself hitting another. When certain elements in the game are viewed in isolation we do see some genuine, impactful improvements to performance. For example, check out this shot from the Militia intro to the Here Be Dragons campaign stage. It's nowhere near as testing on the Xbox One hardware as the Demeter level, but there was one element in the sequence that could see performance stutter for a short while.
You'll note that with the arrival of Update Five, this single pronounced blip in performance has been totally addressed, with a clean 60fps throughout the sequence. In this case, a bottleneck has been addressed and there are no further issues in simulating the scene CPU-side and rendering it on GPU that prevent the game from hitting its target frame-rate.
Overall, Update Five is an interesting example of the challenges facing developers when it comes to optimisation. Streamlining one particular system may result in substantial improvements in one scenario, but elsewhere the same work may not deliver that much of a boost when there's another bottleneck just around the corner. Our initial impression of the patch when we first played it (24 hours after capturing a bunch of Update Four footage) was that performance was improved, but the overall presentation of the game remained essentially unchanged. Analysis suggests that this is indeed the case - we suspect that if we could measure like-for-like gameplay across an extended series of clips, we'd see an overall rise in the average frame-rate, but the prolific tearing remains an issue and gives the impression of an unchanged experience.
Whether the June XDK has been utilised in delivering this boost remains unclear. But if it has, the limited improvements are an interesting contrast to the claims that Bungie's Destiny has moved up from a 900p rendering resolution to full 1080p. For the record, our take on the XDK is pretty straightforward - returning mostly dormant rendering resources to game-makers is obviously a really good thing, but it is not a magical cure-all that will change everything. We expect that Destiny's resolution boost is more about good old-fashioned optimisation, and a 10 per cent GPU boost a useful additional tool. RedLynx's Sebastian Aaltonen explains the situation nicely in our Trials Fusion tech interview, revealing that they started at a low resolution then scaled up as development progressed, when they were confident of hitting their performance targets.
"Launch games never show the true long-term potential of the consoles. Locked 60fps is a very hard goal for any launch title. Developers needed to start programming their next-gen engines before they have access to the final hardware. Lots of educated guesses must be made, and hitting them all right isn't easy," he said.
"In our case we started at 720p on both next-gen consoles, because we wanted to ensure that our gameplay programmers could fine-tune the game mechanics and physics using a build that was running smoothly. Hitting our target frame rate (60fps) was more important for us than hitting a certain resolution at the beginning of the project."
Attaining a consistent, sustained 60fps is a real challenge - Update Five shows that Respawn has made improvements, but there's still a long way to go yet.