A closer look at PS5 ray tracing in Marvel's Spider-Man Remastered
How a key Sony first-party developer is using the bleeding edge tech.
We're finally getting to see more of the launch wave of PlayStation 5 titles, with Insomniac revealing a first look at hardware accelerated ray tracing effects in Marvel's Spider-Man Remastered and Marvel's Spider-Man: Miles Morales. A huge improvement is clearly evident in reflections from the tallest skyscraper to the smallest puddle, but bafflingly, the reveal was met with some vitriol on social media. Artefacts and compromises were highlighted, with some even suggesting that we weren't seeing ray tracing at all. The reality is clear though: hardware RT is an emerging feature on brand new consoles developers are just getting to know, it's computationally expensive so compromises are inevitable - and what we're seeing here is very impressive stacked up against similar hybrid RT implementations seen in the PC space.
In terms of Marvel's Spider-Man on PS5, there are two rendering modes - a 4K quality mode pegged at 30fps, alongside a 60fps performance mode. The former shows obvious RT reflections but ray traced ambient occlusion is also mooted for inclusion. This is harder to identify in screenshots or in videos, but in general, ambient occlusion adds shadows into scenes to simulate the darker effect in nooks and crannies where light bounce is more muted. From what we can tell from the released footage, the 60fps performance mode sticks with the screen-space reflections seen in Spider-Man on PS4 - the key limitation being that only detail in the camera view can be reflected, leading to many visual discontinuities. Skyscraper reflections drop back to much rougher approximations based on captured cube maps that change according to height.
RT is a revelatory improvement here, but some people don't seem to be impressed. It's true that the resolution of the RT reflections is lower - perhaps quarter resolution compared to the main image. Others weren't impressed that certain objects seen on-screen aren't represented in the reflections, leading to suggestions that what we're seeing can't be ray tracing. Meanwhile, the leaves in trees appear much reduced, while reflections also miss some shadows. As I see it, all of these characteristics make a lot of sense and are unsurprising. What we're seeing in Spider-Man is the simple result of the fact that RT has multiple performance sensitive steps and developers have a limited rendering budget for ray tracing. I've tried to rationalise this into four distinct steps.
The first step acknowledges that RT reflections are based on light that can be coming from any direction - from areas not rendered on-screen and can even be very far away. Already the performance implications are building up: the game needs to generate an easily readable version of the game scene to test ray tracing against. The larger and more detailed it is, the more GPU time is required to generate it.
The second step is the actual ray tracing, where tays are shot into the structure defined in the first step. To accurately map collisions against geometry, all of the objects in the game scene have multiple invisible boxes of increasing size around them. The rays shot into the scene move through these boxes until they hit one that contains geometry. The more rays, the more collisions, the more detailed the geometry, the longer this process takes. It's this step that is accelerated via hardware.
The next step is about shading pixels where a hit was registered. Simply put this means shading the hit or choosing the colour for a pixel - like painting the glass side of a skyscraper accurately, for example. The more hits there are to shade or the more more complex that shading is, the longer it takes the GPU to complete. Because ray count is limited, the end result looks very noisy and clean-up is required. This leads us to the final step, where the reflection is denoised - essentially, smart algorithms are used to fill in the blanks, giving a more convincing reflection.
Even with hardware acceleration on the ray tracing step itself, this is still hugely stressful on the GPU and compromises are required. So that's why Spider-Man's reflections are of a lower resolution - it cuts down the ray count dramatically, giving a huge speed-up to the whole process. In the video embedded on this page, I use the excellent RT implementation in Ghostrunner operating on Unreal Engine 4 to give some idea of why these compromises are in place. The bottom line is that even in a simple scene, dropping RT reflections to quarter resolution increases performance by 28 per cent on an RTX 2060 Super, rising to 58 per cent in a more reflection heavy scene. This explains why Spider-Man's reflections aren't running at the same resolution.
As for why certain characters and objects aren't seen in the reflections, this too is a performance optimisation. Going back to step one, where the structure of the scene is built, it stands to reason that the simpler the scene is and the fewer dynamic objects are in that scene, the more performant it becomes. However, simplifying the scene can be counter-productive. Pop-in on elements like skyscrapers can kill the illusion of the reflection - so while nips and tucks can be made, the performance win may be offset to a certain extent by having to increase detail in other areas. This likely explains why leaves on the trees in reflections are pared back in Spider-Man: transparencies (leaves count here) or particles are very heavy to render in RT, so it makes sense to reduce them. You'll also note that elements like fog or steam in Spider-Man aren't reflected either - it's for the same reason.
We've also seen some commentary suggesting that this isn't 'proper' RT, citing Microsoft's Minecraft DXR demo on Xbox Series X as RT 'done right'. The truth is that these are two entirely different approaches to ray tracing, and Minecraft DXR's path tracing majesty is only really possible owing to the fairly basic nature of Minecraft itself (and let's not forget, it's rendering at native 1080p). Marvel's Spider-Man is using a combination of rasterisation and strategic, cost-effective use of hardware accelerated ray tracing to embellish the existing game, and we're seeing a similar strategy in play in other titles too: there's Insomniac's own Ratchet and Clank: A Rift Apart, just for starters, then there's Capcom's Special Edition for Devil May Cry 5, which does the same thing (and offers quality and performance modes too). Gran Turismo 7 is fascinating in that to hit a 60fps target, Polyphony Digital is making its own compromises - with RT reflections presenting with a lower resolution and a checkerboard style effect.
All of which is to say that it's very much early days for ray tracing and to begin with at least, we're looking at 'bolt on' applications of the technology. Regardless, I'm fairly confident that once the head-to-head comparisons for RT reflections in Marvel's Spider-Man appear, it'll be self-evident how much better they look compared to the approximations seen in the PS4 version - and, I suspect, the PS5 game's performance mode. On a more general level, I'm just excited to see hardware-accelerated ray tracing in a console at all, something that still seemed in the realm of fantasy even when Nvidia's Turing GPUs rolled out in late 2018. And on top of that, the fact that we're seeing so much support in launch titles is also remarkable. There's also further reason for optimism: what we're seeing here in Marvel's Spider-Man is just the beginning, it's a first-gen iteration. We've already seen a huge improvement in RT features and performance in the PC space, and with low level access available to the console developers, I can't wait to see how this technology evolves across the coming generation.