Tech Interview: Gears of War 4
"Not only the best Gears game for fans, but also a defining visual showcase on Xbox One."
The Coalition has been hard at work on Gears of War 4 for years now, but 2016 is the year when fans will finally get their hands on the game. Microsoft showed off an impressive demo of Gears 4 at its E3 media briefing this year and we were impressed to learn that it was shown running on Xbox One when a PC version is also in development. The Gears franchise has always pushed the boundaries of console hardware, but the transition to a new hardware platform using a new engine with a new team is no trivial matter.
As striking as the E3 stage demo was, it's the 25-minute demonstration shown off later that impressed us the most. The game's co-op mode was demonstrated using both the Xbox One and PC versions of the game thanks to the Xbox Play Anywhere initiative. The PC was connected to a massive 98-inch 4K LCD display with the game operating at this extremely high resolution and it looked great. Next to this, however, was an Xbox One S connected to a 65" HDR-capable LG OLED display. Despite the higher resolution on PC, the strength of HDR combined with the remarkable performance of the OLED display resulted in a breathtaking presentation. We had our doubts about HDR displays but, after the demo, we immediately started price-checking HDR-capable OLED TVs. It was that impressive.
Beyond that, the game was shown running at a consistent 30 frames per second on the Xbox One S. The quality of the animation, post-processing, materials, and effects really pushes Gears 4 to the forefront of real-time rendering on the platform and we simply had to know more. The result is this interview.
What is it like working with Unreal Engine 4 on Xbox One? What impact does HDR have on workflow? What features are going to be included in the PC version? Mike Rayner, Technical Director for The Coalition, answers all of these questions and many more.
Digital Foundry: Many developers have struggled with Unreal Engine 4 on consoles yet it's clear that Gears of War 4 is shaping up to be one of the most impressive-looking titles on Xbox One. Could you elaborate on your experience with the engine and how things have evolved since development first started?
Mike Rayner: All past Gears titles used Unreal Engine 3. Gears of War 4 will be the first Gears game on Unreal Engine 4 and is the technical foundation for a new generation of Gears games. UE4 is optimised for modern triple-A content workflows, with a completely new rendering engine, and major improvements in almost every part of the engine.
The Coalition was one of the first studios outside of Epic Games to get early access to Unreal Engine 4 in early 2012. We have a great working relationship with the Unreal Engine team. As a member of the Unreal Engine Technical Advisory Board we have continuously encouraged improvements to the engine for triple-A development. Microsoft has contributed to UE4 platform features such as DirectX 12 as well as numerous optimisations and bug fixes. Epic Games constantly improves and evolves the engine and we have kept pace by upgrading to the latest versions throughout development. Gears of War 4 will ship on UE 4.11 along with many of our own custom extensions and improvements.
Through our team's experience with Unreal Engine Development and Microsoft's deep platform experience, we have been able to push the engine to new heights and deliver what we hope is not only the best Gears game for fans, but also a defining visual showcase on Xbox One.
Digital Foundry: With Gears of War: Ultimate Edition, there was a decision to stick with a forward rendering path. Has that changed with the move to Unreal Engine 4 while developing Gears 4?
Mike Rayner: Gears of War: Ultimate Edition was running on a heavily upgraded Unreal 3 engine and was still using a forward rendered path. With Gears 4 we have entirely switched over to Unreal Engine 4's fully deferred rendering engine. Deferred rendering allows for a richer visual quality feature set and more advanced post-processing with attractive performance characteristics enabling increased lighting complexity through decoupling scene geometry from lighting.
Digital Foundry: Speaking of deferred rendering, any chance we could get a breakdown of the G-buffer?
Mike Rayner: Here's a detailed breakdown of our G-buffers:
- SceneColor: GI/Emissive : R11G11B10
- GBufferA: Normals : A2B10G10R10
- GBufferB: Metallicness(7 bits)/CastScreenSpaceShadows(1 bit), Specular, Roughness, ShadingModel(4 bits) and Output mask(4 bits):B8G8R8A8
- GBufferC: Diffuse color(24 bits), IndirectIrradiance(8 bits):B8G8R8A8
The following buffers are selectively written to, depending on shading model etc:
- GBufferD: Custom Data for extended shading models (subsurface, hair, clearcoat, etc):B8G8R8A8
- GBufferE: Shadow channels for deferred lights:B8G8R8A8
- GBufferF: Velocities:G16R16
We have made some changes and additions to the layout to accommodate new shading models.
We extended the G-buffers usage for our hair shading model. This model is based on Disney's 'An Artist Friendly Hair Shading System'. This breaks the hair shading model down into three discreet lobes fully controllable by the artist. Since in this case we don't need the material parameters we use for the regular BRDF, such as Metallicness and Roughness as well as static shadows, we could re-use some of those channels to communicate the many parameters of the Disney model:
- GBufferB: R: Primary Lobe, G: Secondary Lobe, B: AO, A: Tertiary Lobe
- GBufferD: RG: Two channel tangent, B: Shift Map(7bits)/Tangent Sign(1 bit)
- GBufferE: R: Primary Width, G: Primary Offset, B Secondary Width, A: Secondary Offset
Digital Foundry: How has the rendering system changed since Gears Ultimate?
Mike Rayner: We started with a clean slate on Gears 4. There were a few key optimisation tricks, using the hardware optimally, that we picked up during the development of Gears of War: Ultimate Edition that we have moved over. Then we've scrutinised, analysed and optimised every part of the Unreal 4 rendering pipeline. Epic Games has some of the brightest graphics engineers on the planet and they've really outdone themselves this time with Unreal 4 and provided a great starting point for developers.
We have worked closely with Epic Games and exchanged ideas for improvements and optimisations along the way. Parallel rendering was a new feature that was developed by Epic Games and was made available to us during development of Gears 4. This really transformed the rendering pipeline and was a massive power boost in terms of throughput. The Coalition then, working closely with Microsoft's internal D3D12 dev team, transitioned the engine fully to D3D12 on Xbox One - and this again boosted performance greatly for us, and there is still potential to push the engine even further under D3D12.
We have added a number custom rendering extensions to the engine, some examples are volumetric light scattering, enabling physically accurate atmospherics, real-time global illumination bounce lighting, enabling flashlights to bounce light off of surfaces to indirectly light the surrounding environment, and geometry caching, enabling advanced film VFX rigid body simulations and destruction on the GPU in real-time.
We created a custom layered material system that sits on top of the existing UE4 material system. This allows the artists full control of layers and blending of their materials, while also maintaining a tight performance profile as all the information is baked down to the simplest possible form. Since everything was baked down, this allows us to leverage automated texture generation processes to enforce consistency across thousands of assets and save on content creation time. We auto-generate a roughness, height, AO, cavity, composite, and curvature map for each asset as an artist baseline to build upon.
Digital Foundry: The specular on materials looked quite free of aliasing during the E3 demo, could you share some information of how this was achieved?
Mike Rayner: We use what UE4 refers to as 'composite' texture maps on all of our roughness textures to prevent any sort of specular aliasing as you go down the mip-chain in the distance. It essentially modifies the roughness per-mip as it relates to normal map detail. We also employ Epic Games' latest temporal AA implementation with a nice sharpening filter to ensure all our materials look clear and alias-free.
Digital Foundry: Rod had mentioned during E3 that Gears 4 used a dynamic resolution for Xbox One - what was your approach to that (anisotropic scaling?). Is it the same as in Halo 5? Does it operate in both story and MP?
Mike Rayner: Dynamic resolution used in games like Halo 5: Guardians, Doom, and Call of Duty: Black Ops 3 allows games to lock at their target frame-rate and more easily absorb GPU spikes and heavy fire fights, all while running with a higher baseline GPU budget for visual features and increased base-image quality.
For Gears of War 4, our target for Campaign is a solid 1080p at 30fps on Xbox One. Dynamic scaling is used to cover the five to ten per cent of scenarios where we are GPU-bound. The levels we showed at E3 represent our target performance where we were at 1080p 94 per cent of the time. We dynamically scale the 3D scene in the horizontal direction only. UI elements are always rendered at 1080p.
For multiplayer, we lock at 60fps on Xbox One for the low latency twitch responsiveness required for player versus player competitive gaming. Since the Gears of War 4 open beta, we have continued to increase overall visual fidelity of our multiplayer through enabling more visual features, improved lighting and content polish. With dynamic scaling MP versus will be locked at 60fps with higher quality visuals at 1080p the majority of the time.
Digital Foundry: Are you able to clarify what Rod meant in regards to Xbox One S having a slight performance advantage?
Mike Rayner: We render our game at 1080p with some extra computation to support HDR and hardware scaling to 4K resolution if enabled. The Xbox One S allows us to add support for HDR and hardware scaled 4K output without additional demand on our game or compromising performance in any way.
Digital Foundry: With the game returning to its roots in Gears 1, will we see complex shadowing in certain segments? What kind of shadow systems are employed?
Mike Rayner: Yes, we are leveraging the extensive shadowing feature set of Unreal Engine 4 in Gears of War 4. We heavily employ the use of UE4's dynamic inset shadow system - mixing both static and dynamic shadow systems seamlessly which allows for the highest quality lighting and the most optimal performance profile. Every asset using dynamic inset shadows receives a crisp 1k shadow map.
We complement this with screen space shadow technology that Microsoft created to get high frequency animated shadows on small assets in the scene. For indirect shadows we use authored material AO combined with SSAO and capsule shadows to really tie the scene together. The generation and application of the AO [ambient occlusion] textures and composite maps is fully automated in our custom material pipeline.
Digital Foundry: Given the various render targets involved, does ESRAM management become complicated with a dynamic scaling?
Mike Rayner: ESRAM management when done right is not super complicated, but is certainly key to achieving optimal bandwidth performance on Xbox One. The simplest methods are the best as it often turns out. We have reached a point where ESRAM allocations and memory aliasing is performing well for us. We're using dynamic scaling modestly, taking the edge off some of the spikes during heavy action, rather than lowering the density of some of the effects. The ESRAM allocation system we have in place is able to deal with the changes to layout well because the changes are controlled and kept minimal.
Digital Foundry: How did the idea for the weather systems come about? The rendering time budget must be fairly significant - is there some amortisation with async compute? Do the animation systems handle characters?
Mike Rayner: Gears of War 4 is set 25 years after the end of the Locust War. Sera is healing now that Imulsion is gone, through this healing process planetary changes have resulted in extreme weather and wind flares.
To enable wind flares we developed a global procedural wind system for the game. This wind system can vary in intensity from a slight breeze all the way up to hurricane-force winds. The wind system is dynamically scriptable and controllable by our designers, allowing changes in direction and intensity at as the gameplay evolves.
Wind affects everything in the game: players and enemies react to the wind affecting their motion, animation and player state. Objects are affected by the force of the wind: cover can be blown away, and enemies can be sucked up by wind forces. The environment provides protection from wind as you would expect with our wind occlusion system, and safety while in cover. Objects, characters, weapon projectiles, trees, grass, foliage, rain, particles and more can all be influenced by the wind.
During windflares, we have a special GPU budget we enforce with content creators to ensure we can absorb the cost of the weather system and all the associated visual effects which is usually around five milliseconds. There is a fixed CPU overhead cost of the wind system during regular play which allows us to have a wide variety of wind scenarios throughout the game.
Digital Foundry: How does the support for HDR output support affect the pipeline? Is the tone-mapping stage skipped?
Mike Rayner: The majority of modern triple-A game engines have full HDR rendering pipelines and Gears of War 4 is no exception. For the 3D scene, we ultimately tone-map down to a different curve than for LDR and this gets sent to the monitor.
One complication is some of the colour grading is done in LDR sRGB space so we end up doing a traditional LDR tone-map, followed by grading and then conversion to the ST.2084 color space. For UI and HUD, we only want to limit its rendering to a smaller subset of the full range to stop it being too bright. This required conversion prior to compositing and some tuning to get the right look. We did tune some content to make fuller use of HDR space such as some skyboxes and VFX textures but overall >99.9 per cent of our content just worked in HDR.
Digital Foundry: Presumably, you're still making use of Lightmass for global illumination - how is that budgeted in terms of memory consumption? Does that not also present issues with dynamic weather?
Mike Rayner: Yes, we use Lightmass. We budget roughly 250MB of shadow maps and lightmaps. UE4 is fantastic at blending real-time lights and shadows with cooked lighting so we really lean on that for our dynamic weather moments. There is always a back and forth with lighting and tech art as we try and figure out the ideal balance of visuals vs. performance, especially during a windflare.
Digital Foundry: Followers of the UE4 blog might note certain additions specific to Xbox over the past year - might we be able to expect some of those for Gears 4 (or even added because of Gears development)? For example,. async compute for reflection environment and SSAO passes, distance field shadowing/AO etc...
Mike Rayner: Yes, we are leveraging async compute heavily with Gears 4 - for SSAO and for reflection environment apply passes to confirm a few. We adopted async compute early on and learned how to extract significant gains from it. It's designed to allow for squeezing the last bits of performance out of the GPU and that it does well, when applied right.
Digital Foundry: What sort of differences are there when going between the SP and MP renderers?
Mike Rayner: Multiplayer responsiveness at a smooth 60fps is a key focus for which we needed to take a different approach than the 30fps in campaign. For SP we have everything on that UE4 offers and more - allowing the campaign to deliver the highest visual showcase possible. For MP, we disable some secondary systems such as refraction, lens flares, screen space reflections (for some levels), and motion blur. We also did some heavy modifications to UE4's bloom, tone mapper, and decal system to make them as optimal as possible. We wrote our own flavor of the 'significance' system that Paragon uses, enabling player characters to have the highest rendering quality with intelligent dynamic level of detail management for non-player characters.
Digital Foundry: Is the game built as a DX12-only title (with certain low-level Xbox additions console-side)?
Mike Rayner: Gears of War 4 is a DirectX 12 title on both Xbox One and Windows 10. Working with Microsoft's Silicon, Graphics and Media team and Epic, we have transitioned Unreal Engine fully to DX12. DirectX 12 has allowed us to increase performance by giving us more direct control over the hardware, simplifying the driver layer, and allowing us to make fully informed and optimal decisions on how to manage graphics resources.
Digital Foundry: What enhancements might we expect for PC gamers?
Mike Rayner: Our focus on PC is to deliver a highly optimised, customisable experience first and foremost. Support for v-sync tearing has recently come to UWP and we will be able to offer proper unlocked frame-rate support that gamers expect on day one. With Unreal Engine 4 and our own custom modifications, we can take much better advantage of multiple CPU cores, alleviating the game from being CPU-bound and allowing more room for the GPU to shine with enhanced visual quality or higher framerates. Single player will not be locked to 30fps on PC.
A few items of note that we are adding to the PC version:
- Dynamic resolution support (any resolution will be supported with a sensible dynamic field of view - including ultra-wide).
- Increased texture resolution (up to 4K resolution for select texture groups).
- A benchmark mode.
- Full keyboard/mouse input remapping (including cover/run and roll split).
- Far more comprehensive video settings, today we sit at 28 different settings and we're thinking of ways to add more for people to really have full control. Along with more settings, we're putting extra time into making it clear what impact a setting will have on GPU/CPU/VRAM so there is a clear correlation between a setting change and its performance impact.
- We will support super-sampling and have come up with a neat system using the dynamic scaling created for Xbox. You can super-sample, let's say up to 4K, and then enable dynamic scaling and set the maximum amount of scaling you want to allow so you can maintain a very crisp image with scaling kicking in where needed and only to the amount you consider acceptable.
Digital Foundry: At the Gears UE launch on PC, there were notable complaints surrounding features and performance - will there be enough time to ensure a smoother launch for Windows 10?
Mike Rayner: Gears of War: Ultimate Edition was a great learning experience for us. We were the first major DX12 UWP title to market and while we did have issues to work through at launch, we managed to get through them quickly. Speaking directly to our community was a big deal for us post launch and they were fantastic at providing feedback, which we are adapting, where appropriate, into Gears 4. Our relationship with AMD and Nvidia is better than ever as we move towards Gears 4 launch on Windows 10. They are looking at builds of the game today and are providing great feedback to get the best performance out of their respective hardware. We are confident people will have a great launch experience playing the game.
Digital Foundry: Does the game make use of pre-recorded cut-scenes or can we expect real-time sequences?
Mike Rayner: We have both real-time and in-engine pre-recorded cut-scenes. We emphasise real-time cut-scenes and smaller real-time vignettes for in-level story beats and transitions. In-engine pre-recorded cut-scenes are used for a few of the larger cinematics as we transition between Acts or between very different environment sets that require background loading to ensure we have a seamless experience throughout the campaign without any load screens.
Digital Foundry: What are your thoughts on the changing console landscape we're seeing with Project Scorpio and the PlayStation Neo? How much of an impact do you feel this might have on development?
Mike Rayner: For Gears development, we are in a really good position to leverage Project Scorpio to its fullest while continuing a no-compromise approach for all platforms we currently ship the game on. We have invested in scalable features in our engine to support a broad range of GPUs and performance profiles for PC and Console. We have authored our content at 4K, which allows us to show higher resolutions with increased source content detail on the highest end GPUs.
As a developer I find it very attractive having a universal platform to target and optimise our game across multiple Windows Devices: Xbox One, Xbox One S, Project Scorpio, and Windows 10. With the universal platform, we can do this in a manner that allows everyone to play together over Xbox Live without compromising our ability to push each device to its full potential while leaving none of our fans behind, no matter what device they play on.