Zombie Army 4: the making of an 'impossible' Switch conversion
Rebellion North on bringing PS4's 1080p30 experience to Nintendo's console hybrid.
Switch's 'impossible ports' are the new 'arcade perfect' - games devised for much more powerful hardware somehow, miraculously, receiving remarkably impressive conversions onto relatively meagre hardware. However, variable performance and low resolution are also hallmarks of these otherwise astonishing technical achievements and it's here where we need to highlight the work of Rebellion North in delivering some exceptional conversions. Zombie Army 4 is its latest Switch conversion, running at the same 30fps as the PS4 game at a target 1080p resolution and retaining the title's signature 80-100 strong horde of on-screen zombies. Below, you'll see our analysis work on this exceptional port, but the bulk of this piece isn't about the 'what', it's about the 'how', with the developer itself sharing its methodology and insights into the conversion process.
At first glance, Zombie Army 4 on Switch is a ringer for the PS4 version and seemingly matching the resolution and frame-rate goes a long way - just as it did for the firm's excellent Sniper Elite 4 conversion. As you'll read in the interview, development targets native resolution and 30fps in Switch's mobile mode, before scaling up to the docked experience. Dynamic resolution is used, with a 918p to 1080p range, dropping to 684p to 720p handheld. Beyond that, the nips and tucks are many and varied, but crucially, not especially noticeable. There are some exceptions though: cutbacks to dynamic shadows and a lack of screen-space ambient occlusion.
The changes from here onwards are more subtle. Geometry quality is dropped to the far distance - with a more aggressive LOD change-over that doesn't notice in regular play. Effects such as particles and transparencies are also dialled back in quality. Likewise for the texture situation. With only 3.5GB of usable RAM to work with on Switch, memory was a huge constraint for Rebellion's port of Zombie Army 4. Still, the team found smart solutions. Above all, they were determined to avoid a blanket dropping in res to every texture across the game. Reflections are also adjusted, yet screen-space reflections are maintained on water bodies, backed up by an enhancement to Rebellion North's existing cube-map tech for Switch.
The upshot is that the studio's proven track record in delivering excellent Switch conversions is only embellished with this latest release. Now, let's find out more about how this was delivered in this interview with Rebellion North's studio head, Arden Aspinall, senior programmer Alex Houghton and environmental artist, Reece Parrinder.
Digital Foundry: What was the biggest challenge in bringing a game like Zombie Army 4 to switch?
Alex Houghton: To be honest, every limitation of Switch's hardware was a challenge with this one. Previously, CPU and GPU performance was a big issue, but unique to ZA4 was the memory pressure this time. Because it's such a big game, and there's so much going on, we actually had to spend quite a lot of time tackling memory. Now that's not a thing where you can have DRS [dynamic resolution scaling] to scale back on. If you run out of memory, that's the end of it. It crashes. Still, the performance was the biggest issue as you'd expect, really, especially with that many zombies on screen. I mean, I think in some sieges, you can get up to like 80 to 100 zombies at any one time, which is kind of insane really, knowing how much is going on - updating all their logic and rendering them to all the various different passes. That's quite a high level.
Arden Aspinall: It took us quite a long time to just dial it back to the memory. The way we've always approached the Switch ports is, we want to keep try and keep the quality of what we're doing in terms of the resolution. And then once we've got that parity [with other consoles], where it's running exactly the same as it does - irrespective of what the frame rate is, even if it's one frame a second - we get it running exactly like it does on the PlayStation. Then, we can start twisting the dials and pressing the buttons to try and find out what's the best compromise we feel is acceptable to make. But to get to that point with Zombie Army 4 we had to cram the whole thing in memory. And even using the Nintendo dev kits, even that was a stretch. Just getting it actually running on the Switch dev kits with the extended memory, usually that gives you a leg up. So that kind of an interesting challenge.
But it meant that we couldn't see the full [picture]. It's like you climb to the top of a mountain, you see a whole range of mountains, you think, "oh God, we've got to climb all these mountains as well." It took quite a while to get to that point. And I think the team has shown lots of patience, because we were really keen to start putting in all the tools that we already had in the box for Sniper Elite 4, to kind of say "ok, let's see." And we were kind of saying to everyone, "look just hold off. Let's just get the thing working first." You know, we put everything we'd done in Sniper 4 [into the game]. And we'd obviously done Zombie Army trilogy as well. So there was some nice little optimisations that we could do there. Things like shadows when they're off in the distance, rendering them at less quality and things that you wouldn't notice unless you were comparing A and B from different platforms. So we could put some of that in as well [for Zombie Army 4], which we didn't need to bring across for Sniper 4 - because there just weren't that many enemies on screen at once.
Reece Parrinder: One of the main differences I saw from the art team's side of things was that with the older games, like Zombie Army Trilogy and Sniper Elite 4 and Strange Brigade, we were kind of a GPU-focused team. We could just focus on the GPU stats that were coming in, where these really smart guys focused on sorting the CPU side out. So we get them eventually [to help on CPU]. And it was the same with the memory side of things. The biggest change with Zombie Army 4 was that we were then having to come up with new tasks that we could help out on from the outside to try and get extra things back from the CPU and on the memory side of things.
Digital Foundry: When you first started Zombie Army 4, what was the state of it on Switch versus where you ended up? What was it looking like, right at the start when you just said "let's let's just see how it runs?"
Alex Houghton:From the very start you begin, obviously, just getting the code compiling, but then it's mainly about plugging in the NVN renderer. So the first time it runs, and we haven't put in the renderer yet, you're just sat there on a black screen, and you fix the first error with the renderer. And then you just start working through the problems, implementing more bits of the renderer. So you'll start off with a black screen, which is actually a massive milestone, just getting to a black screen.
Arden Aspinall: A black screen with a game running behind... if you're really lucky, you've got the audio so you can hear the music in the background!
Alex Houghton: There's quite a leap from that black screen to just rendering even the loading screen, which is just a static image. You must have so much working: shaders building and executing properly just to render that. So that's another massive milestone in where we begin, and then eventually getting into the 3D. I mean, I do remember seeing that zombie rendering on the front screen for the first time. That was immensely satisfying.
Reece Parrinder: I don't know if any of you guys remember specific stats, but I went back and dug out some of the earliest smoke tests that we had. One of the levels on that for example - I'm not sure when this lines up, or if it's a certain amount of time before we got final month's build working. But for one of the levels, that was running at 116ms on the GPU [around 8.5fps, the target being 33ms/30fps].
Arden Aspinall: I should mention the other thing that we got working out the gates, we really levelled up on Sniper Elite 4 when it came to doing our smoke tests. The artists would find all the real hot spots around the levels and put performance points for the coders to hook in loads of performance stats, where we could break down where all the cycles were going both on CPU and GPU, and also where the memory was going - so that was really good. Right from early on in the project, we could literally just graph our progress. We had like a burn-down over the task to completion, like our own fall-graph to getting to our magic 30 frames a second. So that was kind of fun. But it's also kind of fun to look back at now.
Digital Foundry: Is there a stress test area, where if you can nail performance, the rest of the game should follow? Or was it mostly uniform, just testing everything?
Reece Parrinder: Early on, when I got into the project I was thrown onto the worst performing level, that was the 160ms one that we were looking at. And a lot of the stuff I was doing was clarifying, okay, if we do everything that we need to do - big bulk tests like dropping LODs down and stuff like that. If we can get this level to a reasonable point, all the other levels in theory should come down a lot easier.
Alex Houghton: I mean some scenes you'll be in a forest and foliage rendering is inherently slow, just because there's so much overdraw in it. And then other places, when you can see really, really far into the distance, you end up having to render so much. So, you know there are certain sections in the game where there's loads of foliage, and it was really far into the distance, and then you had 80 zombies rendering at the same time. Those are the kinds of real hotspots in our profiling tests that we knew we had to get under control.
Reece Parrinder: I think as well as that, you'd come across a level that didn't have that sort of stuff. So if it wasn't in a forest, it was in like an underground bunker or factory. And you'd be like, "okay, great. This will run really well!" But then there's like four times the amount of lights because they've had to light that whole underground bunker, and then you're dealing with the lighting cost instead.
Alex Houghton: Yeah. I think it shows that every level is quite unique in the problems it had. So there definitely wasn't just one [solution] that fixed everything.
Digital Foundry: I did notice there's some finessing of some areas, like applying screen-space reflections to certain points in the game. Were many specific tweaks where you decided, "okay, we've got some GPU cycles we can use in this area, can we just throw on more effects and add more to this bit"? Was there a lot of fine tuning between areas?
Alex Houghton: In terms of the screen-space reflections, in the Asura engine there are actually two forms of SSR. You've got the SSR which is applied to everything - so any shiny surface will have SSR applied to it. And then you've also got SSR specifically for materials, which are water. So [on Switch] we kept the SSR on the water. But then there were things like decors, like puddles, which weren't classed as water - so they wouldn't show SSR. In those cases, I don't think it was a conscious decision. It was just that we wanted to keep it on the water for the levels that did have water because it makes such a big difference, like when you have a big ocean. But there are certain bits where the original artist will have authored something as water, so that would have still got reflections on Switch. But unfortunately, it was too expensive on Switch to have full SSR on everything.
Arden Aspinall: This is where we come to the dials and buttons to press. Our goal has always been with all our Switch ports to try not to heavily rely on DRS and try and keep the resolution as high as possible. And you know, from both the art team and the code team, we've tried to kind of reinforce that - try and find ways to keep that [resolution] up, because for us, it makes the experience a lot better. And we find we enjoy games more when it's not too blurry and drops too low in DRS.
Alex Houghton: Yeah, it's probably worth mentioning, regarding the cube maps, that we did actually make some improvements on that compared to Sniper Elite 4. So in Sniper Elite 4 since you're usually just in the big outdoor area, we had one cube map for the entire level. Whereas in Zombie Army 4 we wrote a new system to basically allow you to have local area cube maps - since you know in ZA4, it's very different outside to the indoor areas, which especially can be quite dark. So we did have artists do a pass on various indoor sections to generate and add cube maps for those indoor areas - just to get the lighting looking right. But then you also get more accurate reflections in the puddles which weren't classed as water.
Reece Parrinder: It can kind of make up for the for the SSR that we've lost in some places.
Digital Foundry: All of this again comes back to the idea that 30fps is a huge priority for the team at Rebellion in its Switch ports, and Zombie Army 4 really seems like a success in this regard. What ultimately proved the biggest bottleneck in getting there in ZA4 specifically? Was it the high number of enemy AI?
Arden Aspinall: Yeah, we always say it felt like a death by a thousand cuts when we first started working, but I guess it all depends on what part of the development cycle you're at. I think at the end, it was definitely the zombies. That's where we got all our real serious edge cases where we're thinking, "well, wow, where are we going to get more cycles now?"
Alex Houghton: I think ultimately, it was the number of zombies, to be honest. As I say, when you've got 100 of them, there's so much processing to do and so much to render, and especially with the number of the number of variations in the zombies as well. We obviously didn't want to cut down on anything like that. There are so many different types of zombies to be rendering, which, you know, adds complexity to it all. From the CPU optimisation side, that's where we did focus a lot, trying to multithread more of the update code for all the zombies in their AI and processing to allow us to have so many zombies on screen.
Reece Parrinder: I think from the GPU point of view, the two things that generally stuck out from the beginning were just geometry, whether that's just sheer amount of triangles, and then we had all the transparency stuff thrown in on top of that - and the lighting, really. It was a weird one, because there'd be some areas of your level that had like, no cost for lighting whatsoever. But then you'd go into another area, and you'd have like, 20+ milliseconds, just on the lights. They were the main things that we dealt with. And again, like you mentioned, we had the variations in the characters. With it being the newest project that we've worked on, they'd pushed the amount of unique assets even higher. I did a comparison where, generally in the Strange Brigade levels, they tended to range between between 400-600 unique assets per level. And then in comparison with the ZA4 ones that I tested - four or five levels - you're pushing 1500-2000 unique assets. You're talking about over double, if not triple the amount of unique assets in the level, which was, again, a particular challenge for us to go through and optimise.
Digital Foundry: To achieve a consistent 30fps, did you require a certain amount of overhead? How fast would this game run on Switch without a 30fps cap?
Alex Houghton: We wrote a new system for our QA so that when they run through the game, it collects detailed stats about their playthrough and it breaks it down into sections on how well the CPU and GPU performed. And honestly some levels literally will not drop below 30fps. I mean, I don't think if we'd be quite hitting 60fps honestly, just probably from the GPU point of view. If you lowered the DRS [range], maybe you'd hit close to 60 in certain sections looking at the floor. But some levels do run - even with zombies on screen - at about 20 milliseconds on the CPU. So there is definitely headroom in certain areas and levels to just accommodate for those really bad sections when you're absolutely swarmed,
Reece Parrinder: The reason why we probably wouldn't hit the 60s - like, we probably could have done via the processes that we were doing. But the mindset, especially from the art team, was really to try not to strip out more than you need to. So, there are a lot of cases where instead of just stripping out massive amounts of assets, we really didn't strip out assets that often. I was more like "we're leaving it till the end, if we really need to". Maybe pulling light points or max distances back, so that a few assets in the distance don't render close to you - stuff like that. But it was more just iterating bit by bit to try and push these areas over time to where they needed to be. Just so we didn't over-optimise essentially, because we really wanted to keep that graphical fidelity, that you could lose pretty quickly.
Digital Foundry: You've mentioned already that DRS is a big factor, although you don't want to over-rely on the technology there. And for the most part it seems you are at 1080p docked and 720p portable?
Alex Houghton: I think you'll find it is rendering native 1080p a lot of the time. I can't remember the exact percentage, but some levels when docked are rendering a full 1080p over 90 percent of the time. In terms of anti-aliasing, we didn't go down the route of TAA or anything like that. We actually do just rely on old school FXAA, just for the balance between performance and visuals - and we felt it did actually look pretty good, especially with us targeting a higher resolution, we could get away with the performance of FXAA.
Arden Aspinall: In terms of the DRS, when we're developing we have DRS turned off as well. We never assume DRS is on. It's only right at the very end of the development cycle that we enable it. We just want the engineers and the artists to experience what it will be at with 1080p all the time.
Digital Foundry: I had a go on docked and portable modes on Switch. I was wondering from your perspective, what is easier to optimise for? Do you start with just portable and then expand out to docked? Or do you work from docked and then trim down to the portable side? Which way round do you configure it?
Arden Aspinall: I think with the exception of Rogue Trooper Redux we always go from handheld to docked. We made that mistake with Rogue Trooper Redux, which was obviously our first Switch title. We had fun and games trying to get it working at a reasonable performance but yeah, I think right from day one [on Zombie Army 4] we were just doing the handheld mode.
Alex Houghton: We target that because that's the least powerful mode. We know if we can get it running in handheld, we know it's going to run even better in docked and that we can then boost resolution or some other graphics options in docked. Handheld is definitely where the focus is initially.
Digital Foundry: Something inherent in the way Sniper Elite plays - and Zombie Army - is you've got to zoom in very quickly across a large terrain, while also avoiding obvious pop-in. Are there any workarounds for Sniper Elite 4 or Zombie Army 4 in terms of optimising terrain on Switch - the meshes and textures - to make sure everything fits into memory from the outset, without having anything glaring pop in?
Alex Houghton: In terms of what's streamed in the Asura engine, we only stream in textures and I think certain audio tracks. You'll never get a situation where a model is going to pop in because all the models required are already loaded in memory, so the only thing you could potentially see popping in is in terms of textures. We changed the compression system from .zlib to .zstd, which is actually faster at decompressing textures. That actually helped quite a lot with the texture streaming to allow stuff to load in faster - as well as the changes to the texture size - because obviously, it takes a long time to load in a 1K texture for someone's eye pupils, when you'd [rather] it to be loading in the big rock in front of you or something like that as the priority.
Digital Foundry: Something I was very impressed about - something you achieved with Sniper Elite 4, too - is the game size. The overall install of the Switch package is 6.4GB - a big drop from the 21GB install on the PS4 version. How did you go about that?
Arden Aspinall: I think there are multiple levels to that, really. We are really proud of our ability to compress down these games, where we always look for ways to find duplicate [assets]. That's always the first kind of pass: find all the duplicates - get rid of all that - find all that orphan data left by the wayside during development. That's always an easy target. And then you go by via custom development, where we have tools on top of the Asura engine. Because we know how the data is structured, we can find patterns that a generic compiler can't find. So, we can have reference points back to duplicates of this particular type of data packet, which an off-the-shelf competitor wouldn't really pick up on. But then as Alex mentioned, we switched over to .zstd, from .zlib, which got us extra compression. We [also] changed the texture formats from DXT to ASTC.
Alex Houghton: Obviously, with texture streaming it loads in a low resolution version first and we reduced that slightly on Switch - not that you'd ever notice, to be honest. Other than that, we optimised some of the light maps in the levels. This is another custom tool we wrote for the artists to visually colour the resolution of light maps attached to certain assets. And we were finding things, like where a small rock at the start of certain levels have a 256 by 256 light-map attached to it. We were just really thorough with every level and went through it with a fine-tooth comb to find everywhere we could scrape back memory.
Arden Aspinall: Compressing the vertex buffers...
Alex Houghton: Yeah, yeah. All the model geometry we've optimised further on Switch so it is better compressed and smaller, basically. Yeah, I'm still quite impressed myself that it's such a small package size.
Arden Aspinall: I remember we were at 9GB. And we were thinking we need to get below that 7.5GB limit. So yeah... 6.4GB, we kind of overshot it in the end didn't we?
Digital Foundry: What is special about the 6-7GB range?
Arden Aspinall: Just for Rebellion's benefit, we wanted get it onto an 8GB cart. Obviously, the cost of manufacturing is a lot cheaper, so there's a lot to be said for trying to do that. But there's a benefit, obviously, to the consumer as well, because there's less memory on the Switch as well - so less of a footprint means that they benefit as well.
Digital Foundry: Are we now approaching a point of diminishing returns on Switch in terms of what we can get back? Or is there still so much more you could do in your next project.
Arden Aspinall: What I always say to the team is there are always ways to optimise further. Whenever people say "that's it, we're done," there's always a step further you can take it. It's just time and ideas. One of the things that I really value about the team here at Rebellion North is that we all get together and brainstorm ideas. I look back from our first game on Switch - Rogue Trooper Redux. I'd love to go back and apply everything we've learned, to see what it would look like now if we did a new version of it. And I'll also say, literally, we have to tell the team that this is good, what we've got. We've got to submit now. We were literally days before submission, [and] we're going "oh we've got a new idea for optimisation. Can we get away with this one final optimisation?" And we had to say no, that's it. We could have carried on, but there's a point where we've got a duty to the group to get that game shipped. And I'm really happy with what we got out with Zombie Army 4, I'm really proud of the team. They did an amazing job.
Digital Foundry: A very quick wrap up question would be: what is the part of Zombie Army 4 that you're most proud of? Maybe something we've not talked on or highlighted or we haven't discussed yet?
Alex Houghton: For me personally, I'd probably say it was the addition of the area cube maps because we were quite late in development at that stage, and it was deemed a little bit risky, and it was like rewriting a brand new system for the Switch. I'm quite proud that we managed to get that in, and the visual difference it makes to the indoor and outdoor areas when the cube map is accurate to what's around you. It's also a brand new feature to our Switch ports that wasn't ever present before. It's one of those kind of subtle things where you don't notice it, unless it's actually wrong - but I'm quite proud that we managed to get that working.
Reece Parrinder: One thing I'm always really happy about with on these [Switch] projects is how little of the time through development we actually pressed the delete button. As far as quantity of assets, we do retain quite a lot of them, especially in the areas close to where the gameplay is happening, where the player is. And when you're looking at these levels, initially, you're kind of looking around and you're thinking, "oh my god, there's a lot of grass everywhere, we could just delete all this grass, and it would help a lot." But it turns out, a few months down the line, in the end, we've been able to retain pretty much everything, and just strip back bits a bit more in the distance that really, you're only going to notice five percent of the time.
Even when it comes to the lighting side of things as well, there are a lot of little tricks that we've been able to do, where the fall-off on the edges of the lights are a little bit harsher [on Switch]. But it meant that because the fall-offs are less, we can also pull the radius of the lights in more. And then the overall cost of the light comes down a lot, whereas the visuals don't change a hell of a lot. So, it's little things like that. It's us being able to almost build up this optimisation library - a bible of little neat tricks that we can always apply to projects going forward.
Alex Houghton: Something else I'd like to mention is that it would have been so easy on day one, when the performance was bad, to go in and just cut out a quarter of the zombies. But we didn't do that. You get the same gameplay experience that you do on all the other platforms, on Switch. I'm quite proud that we didn't compromise anything in that respect from the gameplay side.
Arden Aspinall: There were a lot of times when we have to push back quite hard as engineers, to say that we can get this [working]. They were saying "you're running out of time," and we were like going, "Yes, we can. We'll go away, we'll come back, we'll find a way to optimise it. We can do this." It was great that Chris and Jason gave us that opportunity to do something that was considered to be impossible - to be brutally honest - by the rest of the group. Getting Zombie Army 4 to work on Switch was considered impossible. And we did it.