Making Call of Duty: Ghosts
Infinity Ward on next-gen, mega textures, dedicated servers and the future.
Making Call of Duty is hard; hard for both Infinity Ward and Treyarch, the two studios Activision charges with sharing the responsibility of producing a new game in the gargantuan first-person shooter series each year.
But this time, for Ghosts, making Call of Duty was harder. Developer Infinity Ward created five versions of the game: PC, PlayStation 3, Xbox 360 - as normal - but also PlayStation 4 and Xbox One versions in time for the consoles' launch. Five versions, two of which intended for release on for what was for much of development theoretical hardware.
Here, in a sweeping, candid interview with Eurogamer, Infinity Ward executive producer Mark Rubin, who has worked at the studio since 2005's Call of Duty 2, reveals the enormous challenge of creating Call of Duty: Ghosts in time for Christmas, the "engineering nightmare" of developing launch titles for PS4 and Xbox One, and discusses the tough decisions that affected the current-gen versions as a result.
Unlike previous versions, Ghosts was a five version project.
Mark Rubin: It's six. Two current-gens, two next-gens, and PC, plus the Wii U, which we didn't do.
Who created that?
Mark Rubin: Treyarch. Treyarch has a Wii U team specifically for just doing Wii U. That's all they do. They only do the Wii U games for every title. It was a Wii team that turned into a Wii U team.
But still, yeah, five platforms.
That's a lot.
Mark Rubin: Yeah. This was the hardest game I've ever made, by far. It's interesting. When we first started no-one expected us to do both generations. They thought, do you guys want to do next-gen and someone else will do current-gen?
Who is they?
Mark Rubin: Activision. Activision came to us and were like, okay, next-gen is launching within your time frame. Do you guys want to work on next-gen and have someone else do current-gen - a port? Do you want to work on current-gen and have someone port the opposite way? It didn't take us more than three seconds to decide we absolutely wanted to do both.
We want the next-gen title to be really strong and good and stand on its own, and we wanted current-gen to be better than any current-gen game we've done in the past. So we instantly said, we want both. We want to do both. We got this. And then we looked at the work and were like, oh god.
It was almost a perfect storm. We are doing current-gen and next-gen, plus PC. We have a new story, a new world, new characters. That's a whole lot of new thinking you have to do, and new designing and a lot of new work. Then on top of that we have a new engine. Then on top of that we have a whole new mode, Extinction.
"It was a massive undertaking. Looking back at it, wow, that was the hardest thing I've ever done. And then if somebody asked, would you do it again in the same way? I'd probably say yeah."
So the amount of work and effort that has gone into this game is monumental. And so as a team we ended up growing significantly. There's probably about 120-125 people that are IW people at the office, and another 20 people who aren't IW but are at the office working. And then Neversoft and Raven helping out as well. That was the only way we could have done all of this work.
It was a massive undertaking. Looking back at it, wow, that was the hardest thing I've ever done. And then if somebody asked, would you do it again in the same way? I'd probably say yeah.
Yeah?
Mark Rubin: Yeah. We really wanted both platforms to be great. Here's a sports analogy: good players want the ball. That's the same thing for us. We wanted that work.
Looking back, do you feel you bit off more than you could chew? Did Infinity Ward spread itself too thin by taking on so many different versions?
Mark Rubin: We did a really good job of managing that actually, under a really difficult situation. It would have been easy to do exactly what you said - to be spread thin - but we have such a strong group of people who have a strong drive over the game and opinion on the game that no-one quietly faded away and just let whatever happened happen.
It was always, this isn't good enough. We don't have a lot of time left. It doesn't matter. Not good enough. Cutting it. Doing it again. Doing it better. That happened a lot throughout the entire project in a time when there's already more work than we think we can do - people still kept their quality bar high.
I can't even begin to guess how many multiplayer levels got cut. There's probably a good 8-10 on the cutting room floor of levels that didn't make it. They looked really good and could have, in some people's minds, shipped, but they just weren't quite there. We are very particular about the quality that goes in. If it's not good enough for us then we don't put it in. And not many companies can do that.
Especially given the situation of how much work there was to do next-gen. Art assets probably took two to three times as long to do than normal. Environment art took longer. Everything took longer.
Is that just because greater detail is required on next-gen?
Mark Rubin: Yeah. We also did something different this game. On the previous games, for current-gen, it was 360, PS3 and PC, and the art asset was for the most part shared across those platforms in a lowest common denominator sense. Well that's not the case this time.
What we did was we made an art asset that looks phenomenal on PC. Then we made another art asset that looks great on next-gen. And then another art asset that looks great on current-gen. We didn't do the one art asset. We did three art assets for every art asset, which is why it's two to three times as long to make the art assets.
I think that pays off. Our guys are proud of the work. That's why I say, if asked, would you do it again? It's, yes. Although the road to this place has been very difficult, we feel we've been very successful at what we've done and we're very proud of the work that is going out.
It's been an enormous challenge. I know people will want a feedback of how it went and what you could do from it. I think most people are going to be so exhausted they just want to put this behind them and move on to the next one.
"It's been an enormous challenge. I know people will want a feedback of how it went and what you could do from it. I think most people are going to be so exhausted they just want to put this behind them and move on to the next one."
In creating the next-gen versions, you were working to hardware that was ever-changing.
Mark Rubin: Theoretical, basically.
How do you deal with that? It sounds like a nightmare for a developer.
Mark Rubin: It is, for our engineers especially.
Fortunately, both Xbox One and PS4 are very much like PC, more so than the last generation. That helped enormously. If the systems had their very peculiar architecture, like they did in current-gen, this would have been a different conversation. But because of that the development this time around was significantly easier.
I have experienced the current-gen launch. I was at Infinity Ward for COD 2. When we launched it was just PC and Xbox 360, but that was our first console, period. There was a lot to learn leading up, but that was just one console, when all it was was PC and that new console. And that was an interesting challenge.
So to do PC, current-gen two SKUs and next-gen two SKUs, was a massive challenge. Working with the theoretical hardware would have been a disaster if... really it was helpful it was very PC-based architecture.
Honestly, the hardest thing to deal with is not the architecture. It's the OS (operating system) of the systems. That's the thing that comes on the latest. The Xbox One's OS on their box versus the Sony OS, becomes the hardest. All the SDKs and stuff you have to work with - that's the stuff that changes, not the hardware itself.
What is it about the operating systems, exactly, is the problem?
Mark Rubin: There's stuff in the console's OS that interacts with the game. So, for instance, voice chat is often supported by the hardware manufacturer rather than the software, and you're just using their channel. When that stuff is changing - because they're developing it on their side - and the resources they're using are changing - your, from a game design standpoint, challenge is with trying to make enough room for those resources to be used but at the same time use as much resources as possible.
One of the greatest challenges the engineers have to deal with is memory management, or thread management. There are X number of threads in your CPUs. Where in those threads is the stuff that's Microsoft or Sony? Where does it fall? How does it work? We don't have the SDKs for those features yet, and then they come in and you go, okay, well it needs 3MB of RAM - oh, crap, we only allocated two! You can't just take a MB from anywhere. It's not like there's tonnes of it just laying there. You have to pull it from something else. And now you have to balance that somewhere.
It becomes a massive change internally for our entire engine, if they add a few MB to the amount of resources they need, or if they require all their processes to be on one thread. If it's not multi-threaded then we have to put it on one thread. Now we have to find space on one thread, where that can live, that it's not creating a traffic jam on that thread. Sometimes we have to be like, okay, we have to move all this stuff over to a different thread and then put that in to that thread, just to manage traffic.
That's what engineers are often doing: managing the traffic of CPU threads and memory and where that's going. What kind of memory is it? Is it dynamic? Sometimes what has to happen is we have to allocate the 3MB straight off the bat and just say, these 3MB, specifically, these actual memory addresses, have to be used for this. They can't be used anywhere else. Whereas dynamic, it's like, okay, I need 3MB but it doesn't matter where those 3MB come from.
So all that stuff can change on the fly. And you're trying to develop your system to match with that. And it's two systems, now, not just one: Sony and Xbox. That creates a massive engineering nightmare.
Wow. It sounds harder than I realised.
"To do PC, current-gen two SKUs and next-gen two SKUs, was a massive challenge. Working with the theoretical hardware would have been a disaster if... really it was helpful it was very PC-based architecture."
Is all that you've just described the reason the Xbox One version is native 720p and the PS4 version is native 1080p?
Mark Rubin: In a way. I don't know if I can point to one particular cause. Early on, we didn't know where exactly the resolution of anything would fall because we didn't have hardware or the software to support it. We tried to focus in on 1080p, and if we felt like we were on borderline of performance somewhere... We tried to make the best decision for each platform that gives you the best-looking game we could get and maintains that 60 frames a second.
There's no specific, oh, well, the VO chat on Xbox took up so much resources that we couldn't do 1080p native. There's no definitive one to one per se cause and effect. It's just an overall thing. We took each system individually and said, 'okay, let's make the best game we can for each system.'
I think both look great. Some people might notice if they had them right next to each other. Some people might not. The Xbox One is 1080p output, it's just upscaled hardware wise.
It was a late decision, too. That call wasn't made until a month ago.
Put me in your shoes when you were told this was going to be the case. I assume your engineering team explains to you this is the way it has to be. How did you guys react internally? Could Microsoft engineers not have helped?
Mark Rubin: It's not a thing, like pointing to the day he came and said it's 720. It wasn't like that. It's a long process. And we're always working with both platforms. There were Microsoft engineers there throughout development. They were always there. There wasn't an event, per se. There wasn't a meeting. It was just something that developed over time. Everybody was involved.
Obviously the PR guys, when they found out, when they were told, that was more of an event than the devs sitting at work working on it. So you'd have to ask them.
"If people are fearful one system is more powerful than the other or vice versa, it's a long game."
What everyone will ask is whether this is the result of the Xbox One simply not being as powerful as the PS4, and you're doing your best with the hardware you have, or whether for future versions you may be able to get the Xbox One version running natively at 1080p?
Mark Rubin: It's very possible we can get it to native 1080p. I mean I've seen it working at 1080p native. It's just we couldn't get the frame-rate in the neighbourhood we wanted it to be.
And it wasn't a lack of effort. It wasn't that it was like last minute. We had the theoretical hardware for a long time. That's the thing you get pretty quickly and that doesn't change dramatically. It was more about resource allocation. The resource allocation is different on the consoles. That huge web of tangled resources, whether it's threads-based or if it's GPU threads or if it's memory - whatever it is - optimisation is something that could go theoretically on forever.
I definitely see slash hope both platforms will look way better the next time we get a chance at it. As an obvious analogy - and if people are not sure about this it's pretty simple - look at Call of Duty 2 versus COD 4. It was a massive leap forward in graphics, and that's just because it takes time to get through this.
First launch, first time at bat at a new console is a challenging one. That's just the way it is. If people are fearful one system is more powerful than the other or vice versa, it's a long game.
The next-gen version is huge. I've seen it's a 49GB install on PS4.
Mark Rubin: Yeah it's massive. It's all textures. Textures just take up massive amounts of space. When you increase a texture from say a 1k to a 2k texture, it's quadrupling the size. We did that. Actually, there are some textures that are 4k textures.
On console?
Mark Rubin: Yeah.
"I would be very upset if we limited ourself on the visuals because we were worried about hard-drive space. It's not our responsibility."
Why did you do that?
Mark Rubin: Because we're crazy. Just to clarify, there's a difference between a 4k texture on a model and the resolution of 4K. Totally different conversations.
LG gave us an 84 inch 4K TV. One of their early ones, before it had even come out. We hooked a PC up to it - a beast PC - and it was gorgeous. It looked amazing.
A glimpse at the future?
Mark Rubin: I hope so.
What does this mean for gamers though? The hard-drives the consoles come with are not infinite. If you install Call of Duty: Ghosts at 50GB, that's a tenth of your hard-drive with just one game. And who knows how much your operating system will reserve. What does this mean for the future of console gaming, where these games are so massive and hard-drives are only so big?
Mark Rubin: We have to try to make the best game we can, which means we're going to continue to put lots and lots of textures and more variety.
I would be very upset if we limited ourself on the visuals because we were worried about hard-drive space. It's not our responsibility. It happened on every console that's been out: they come out with versions that have bigger hard-drives, but they don't need to right away. But they will eventually when games start to get bigger.
People might look back on 50 and go, wow, remember the days when they were only 50GBs?
Switching to the current-gen versions, with Ghosts there's a maximum of 12 players online, whereas with previous versions there was a maximum of 18. Why?
Mark Rubin: Honestly, we played with it for a while, and just felt, performance wise, it wasn't a good experience on current-gen. We had done enough new stuff to the engine, added a lot of new features, that the current-gen systems became bogged down under 18 players.
Yeah, it sucks. I know a lot of people play Ground War. But it's funny how many who play Ground War always complain Ground War has issues. I'm like, well, it's because you're 18 players on a tiny little map. Of course you're going to have spawning issues.
That being said, people still enjoy it. And we totally wanted to do Ground War on current-gen. It wasn't an easy decision to cut it. But we wanted to make the game good. Character customisation, for instance, takes up a massive amount of resources. So having 18 people on the map with unique characters is a challenge compared to having 12 people on the map with unique characters.
It really was holding to our pillars of performance and fun and making sure the game feels smooth and feels good.
"People still talk about that level with such high esteem. And by high esteem I mean it was such an impactful moment for gaming. I think parliament here was debating about it."
In Europe Ghosts' age rating is lower than previous Call of Duty games. Did you make a conscious effort to tone it down?
Mark Rubin: It just happened. We have no idea what the guys would rate the game. We've always been, I think, particularly at IW, pretty understated with the violence. There's not like decapitations and body parts flying everywhere. Previous games have had some things in them that caused the rating to be higher, because it fit with those stories.
No Russian is often brought up.
Mark Rubin: Yeah, No Russian being the obvious thing to point to. It's really interesting. People still talk about that level with such high esteem. And by high esteem I mean it was such an impactful moment for gaming. I think parliament here was debating about it.
So, we have no idea what the game is going to rated every time we try to make the game. And it wasn't any conscious effort to reduce the age. It's still mature in the US. UK dropped but US stayed the same.
All versions of Ghosts has dedicated servers. We know you're using Xbox Cloud for the Xbox One version, but what are you guys doing for the other versions and does this apply to the Wii U?
Mark Rubin: I can't speak to Wii U. I don't feel comfortable enough to talk to what the Wii U's doing.
But on all systems - I'm going to use generic terms instead of... - but, on those other systems, we are paying for massive amounts of servers, an enormous number, because of how big the game is. If we were a smaller game, dedicated servers would be easier, actually, but because we're so huge it's an enormous machine that's moved forward to create a big giant backend of dedicated servers for all of our games.
It doesn't sound cheap.
Mark Rubin: It's not cheap. Yeah. Not even remotely.
But you get it for free on Xbox One, from Microsoft, right?
Mark Rubin: Free-ish. It's a complicated situation that is less easy to get into. We're probably going to do a bit of mixing for a while, until we figure out what the best serious solution is. Part of our initiative here is to figure out how to make the backend that much better, and it's a continuous process. We're not like, okay, there you go, hope it works!
When you say mixing, what do you mean exactly?
Mark Rubin: From a player's perspective we're trying to figure out what the best situation is for you. One thing I talked about was this hybrid system, which is a mix of listen servers and dedicated servers. Let's say I am in this point here, and there's a dedicated server not far, because we've seeded them throughout the world, basically buying them and putting them in, and then Microsoft has its cloud, which is also all around the world. So we put these servers all around the world so people who are near can be near. But if you're not, or if you're in a situation where you live a little farther, sometimes there's a listen server right near you that's actually going to be a better experience than going to the dedicated. So, if that's the case, you'll hook into the listen server.
It's impossible to cover the whole entire world with servers. We would be talking billions of dollars a year. It's funny, we looked at dedicated servers back on COD 4. And at the time we could only run four instances on one computer. Do the math of how many users we have, it would have been billions of dollars of computers, and we'd have to spread them out across the world. It was just financially not possible.
For non-Xbox One platforms, who are you buying the servers from?
Mark Rubin: I don't know if I can say.
I know Microsoft's Azure platform is available to anyone. Can you make use of that for other platforms at the same time as what Microsoft reserves for Xbox One?
Mark Rubin: The Thunderhead stuff? We're still figuring that out. I think Microsoft's still figuring that out too. We're working together to help them perfect the cloud as well. It's not done and ready. It's like us, where even when we launch we're still going to be tweaking and working and making sure we can keep improving. Xbox is doing the same thing there, taking their cloud and improving. A lot of what we learn from launch is going to help them improve that system further. So anything I say today could change tomorrow.
But the cool thing is, whatever we need to do we're going to do it to make sure it's the best possible experience. We'll keep going at it and keep trying to make the whole set-up better for every platform.
"It's not about, does my weapon shoot bullets or laser beams? It's about, are we telling compelling stories and are we making fun gameplay?"
What happens next for Infinity Ward and Call of Duty? Do you think it's now time Call of Duty left the modern warfare setting behind, or do you feel there's still a lot more you guys can do in that space?
Mark Rubin: There's tonnes of time. I don't think the modern era holds us back at all. Ghosts is such a departure from the Modern Warfare universe. It's got its own new universe. There's tonnes of stuff we can explore.
It's not about, does my weapon shoot bullets or laser beams? It's about, are we telling compelling stories and are we making fun gameplay? Honestly, the brainstorming sessions I've talked about in the past are starting right now at the studio with small groups, particularly on the single-player. They're trying to think about what they want to do next.
If you haven't played the campaign yet... One of the things I'm really happy about with the way the campaign came out is how rich that world we've created is and how much cool stuff there is there to do. Let's say we do a couple of games on the Ghosts style, and then we could go and do another subtitle game.
So Ghosts isn't designed as a one-off? You want to establish it as a sub-brand like Modern Warfare?
Mark Rubin: Yeah. It doesn't have to, but it could. We didn't say at the beginning, okay, this is going to be three games, or whatever. We just started out to make a great game. Fortunately I feel the world has a lot to offer and it's a really interesting world I'd love to do more with.
But, we could figure out something new. Maybe we'll do Jupiter missions or something! Move into all space or something. Who knows? We put all the ideas on the board and everything is viable.