Digital Foundry vs. Vita Remote Play
Can the PS4 hook-up boost the handheld's fortunes?
PlayStation 4's Remote Play connection with PS Vita is exciting stuff - a value-added extra with the potential to revitalise the fortunes of Sony's under-performing handheld. Nintendo's Wii U GamePad established the notion of decoupling from your HDTV and playing remotely with excellent image quality and no noticeable lag. Since then, Nvidia's Shield has come along - not quite as impressive right now, but great stuff nonetheless, with a new update coming up soon that promises significant improvements. Call it wishful thinking on our part, but we couldn't help but hope for similarly impressive results from PlayStation 4.
All of these streaming gameplay set-ups are based on very similar principles: the current frame is grabbed from memory by a hardware encoder, compressed into h.264 video, beamed over WiFi to the handheld, then decoded and displayed on-screen while control inputs are transmitted back to the host. Wii U accomplishes all of this extremely quickly - we ran the GamePad alongside a PlayStation 3D screen with 33ms display lag, and via high-speed photography we noted absolute parity in the refresh. Nvidia's Shield system offered no direct connection between handheld and PC, but even via a router we noted remarkably swift response - despite a downgraded 30Hz display refresh. Nintendo also took the honours on image quality, with macroblocking artefacting much more noticeable on Shield, mitigated only by the small display.
The stakes are high for Sony then. There's a really high expectation level with Vita Remote Play because the core technology has been proven to work on not one, but two different systems. Indeed, Nvidia manages to hand in a highly playable experience despite the lack of a direct hook-up between the PC and the Shield handheld - traffic needs to travel via a router, presenting its own latency implications. That expectation level is raised still further by the upcoming arrival of Vita TV - a screen-less version of the handheld that, in time, will also support PS4 Remote Play (we've got a Japanese unit in-house now and rather annoyingly it's not enabled in the current firmware). Sony's new venture doesn't just need to work on the smaller screen, it needs to scale up for HDTV usage too.
So just how good is the new streaming gameplay set-up? Before we kick off with the inevitable PS4 vs. Vita image comparison, we should explain how Remote Play works. As we understand it, the image is downscaled from 1080p to 720p and downscaled again to 960x540 when it's displayed on the Vita screen (leaving a tiny four-pixel black border on the bottom of the screen). Sony's Shuhei Yoshida confirmed the encoding resolution for Vita TV, and we suspect it's the same for the standard handheld too. Bottom line: the no-show of Remote Play on Vita TV means we had to grab the downscaled 960x544 image from the handheld, rather than the preferable 720p feed from its screen-less sibling.
"There's a really high expectation level with Vita Remote Play because the core technology has been proven to work on not one, but two different systems - the excellent Wii U GamePad and Nvidia's Shield handheld."
Image quality generally looks pretty decent, but despite the resolution advantage, it's far from the high quality presentation we get from the Wii U - closer to the picture you tend to get from Nvidia Shield, but still not quite up to scratch. Vita Remote Play appears to run on a constricted level of bandwidth, and coupled with the limitations of the hardware h.264 encoder, high frequency detail is often smudged away in fast motion. Generally speaking, the faster and more detailed your game, the more likely the dreaded macroblocking is going to appear. In the range of games we tested, Resogun definitely had the most issues in retaining image integrity, but Killzone Shadow Fall also took a hit to its beautiful, intricate detail.
Also curious is that the macroblocking has soft edges, suggesting that we are indeed looking at a downscaled 720p stream. The fact that we see them at all on a downscaled video isn't good news - assuming the same stream is used on Vita TV, we strongly suspect that the image quality just won't pass muster on a living room HDTV. For the handheld Vita, it's a different story - the downscale, combined with the small, high pixel density display does wonders in hiding most of the issues you see here. The image sometimes looks blurry, but passable - even on Resogun at its most intense.
We can also confirm that Vita Remote Play does indeed run at 30fps - a commonality it shares with the current version of the Nvidia Shield software (though this is set to be addressed in next month's update, which should offer 720p60 - and even 1080p60 if you tether via USB), but a world away from the beautiful, full 60Hz refresh of the Nintendo GamePad. Similar to Shield, we couldn't help but think that this wasn't a smooth 30fps cadence, definitely detecting some judder as we played, so we decided to run our captures through performance analysis, comparing them to the original PS4 output at the same time.
Resogun was smoothest of all in our testing, but on all other games we noted occasional frame-rate drops that introduce a small element of stutter to the otherwise smooth flow of gameplay. Also interesting is the amount of correction we had to do to the analysis - Remote Play has an interesting issue where small segments of any given image are displayed one frame ahead of the rest of the image. This confused our analysis tools, which recognised these frames as unique images when really they're not. The phenomenon is really difficult to spot by eye, but it adds a touch more to the slight judder issue.
So thus far, Vita Remote Play is looking solid but a touch sub-par. Image quality works on the handheld display, but it's far from best-in-class, while refresh has been compromised - not an issue whatsoever on 30fps games like Need for Speed: Rivals and Assassin's Creed 4, but the experience of Battlefield 4, Call of Duty: Ghosts and Resogun is clearly diminished. In truth, while image quality and frame-rate matter, the more readily apparent issue is the size of Vita's screen - lining up shots in Killzone and Battlefield 4 proved extremely difficult because the targets were so small. It's surprising just how much of a difference that extra inch on the GamePad's display makes.
But what's truly going to make or break Vita Remote Play is the issue of latency: the added time required to encode, transmit and decode the image. We've done plenty of this kind of testing in the past on cloud services like OnLive and Gaikai, but like Nvidia Shield, Vita Remote Play is different in that we can see more of the pipeline - the time taken between pressing the button on the Vita, seeing the reaction on the HDTV, and then the same frame appearing on the handheld.
However, judging the exact frame when the button is actually pressed isn't easy and can be prone to error. Again, if Vita TV had its Remote Play functionality working, we could have used the PS3 version of our Ben Heck controller monitor board, specifically designed for this task. However, the exercise is still worthwhile because we know that the latency of our Asus monitor is between one to two frames, 16-33ms. Count the number of frames between the same image appearing on the Asus screen and its arrival on the Vita, add display latency and we have the total time taken for Remote Play to work its magic - though this does not include the input lag of the game itself.
The results are... well, inconsistent is the best description. What's clear is that there is a noticeable latency penalty to using Remote Play - and a baffling lack of solidity in its performance when we look at our results on a frame-by-frame basis. On our first run, we used the direct link functionality, which hooks up the PS4 and the Vita directly in much the same way that the Wii U interfaces with the GamePad. We tested Assassin's Creed 4, Battlefield 4, Killzone Shadow Fall and Resogun. In direct connection mode, we were playing just one metre away from the PS4 with the router turned off to ensure that traffic could only travel directly between the Vita and the PS4, while in router mode, we ran everything in a 5m x 3m room, with the PS4 connected to the router via a LAN cable, as Sony recommends.
"Remote Play sees frame-rate halved to 30fps - fine for the likes of Need for Speed: Rivals and Assassin's Creed 4, but definitely impactful when playing Battlefield 4, Resogun and others."
In our initial test run, Killzone Shadow Fall fared worse of all with Remote Play taking an average of 160ms to respond. Now, we've visited Guerrilla Games, talked to the developers and we know that input lag - which is not factored into the equation - comes in at around 80ms. That results in a deeply disappointing total end to end latency of 240ms. Weirdly - by switching to the router (where we would expect latency to increase), we shaved that down to an average of 180ms - playable, but not great. Moving to Battlefield 4, we saw the opposite. Via the direct connection, Remote Play latency (not factoring in input lag) arrived at 100ms, creeping up to 133ms when we hooked up via the router. At the granular level, we saw a variance in response even between the same tests repeated multiple times, and on all games. Killzone just got the luck of the draw in producing the worst response during our testing.
Looking across the breadth of our single measurements (not the averages) we found that at its absolute best, Vita Remote Play latency came in at around 90ms. At its nadir, we saw a single measurement - repeated several times over in the same session - of around 160ms. Add input lag on top of that and the experience you get compares poorly with the standard Dual Shock 4. The curious thing is that Remote Play is apparently based on Gaikai technology, and while our tests of the cloud service were a mixed bag, at its best, we saw complete end-to-end latency (including input lag) on Bulletstorm of just 133ms - significantly faster than Vita Remote Play.
Now, latency perception is a funny thing - we all feel it different ways, and there seem to be a lot of people out there very happy with Remote Play, so we embarked on a range of additional tests in order to verify our results and to weed out any potential weaknesses in the pipeline. First of all, we tried another PS4 and got a similar range of inconsistent metrics. Next, we swapped out our video-output modified launch Vita for a brand new slim and light PCH-2000, and nothing changed - although we did note that the LCD screen produced a brighter presentation, something that is more helpful than you might think. On the launch model, the image is very dark, necessitating the brightness settings to be ramped up to the max, while the PCH-2000 looked fine on default settings.
Finally, although it would in no way explain the inconsistent results from the direct PS4 to Vita connection, we even swapped routers, exchanging a standard UK Sky Fibre 802.11n router made by Sagem for a rather swish Asus RT-N66U - a beautiful piece of kit if networking hardware is your thing. Again, we saw no appreciable difference. We even cleared the room of all WiFi devices and the router, then tested again with the direct connection active - the two consoles the only wireless devices in the room. Again, we saw no appreciable difference. It's also worth pointing out that our bargain basement Sky Fibre router was the same box we used to test Nvidia Shield back in the day, with good results. In short, we pushed as hard as possible to get the best possible signal, but nothing seemed to help our latencies to any appreciable degree. In terms of range, we tested up to ten metres with no further impact to performance, and we even ran the PS4 from another building via powerline ethernet adaptors hooked up to the router, with much the same response.
One interesting result we did find from our testing is that if you disregard the Vita display and simply use the handheld as another controller for local gameplay, viewing on the HDTV directly attached to the PS4, the net result is that latency in this scenario isn't that bad at all. It's not quite up to Dual Shock 4 standards, but yes - you can use Vita just like a local controller and it works quite nicely in both router and direct connection modes. This indicates to us that the encoding/transmission part of the pipeline is perhaps where the hold-ups are occurring.
PlayStation Vita Remote Play: the Digital Foundry verdict
It was perhaps a bit much to hope that Vita Remote Play could match up to the Wii U GamePad, or even Nvidia Shield's performance. Nintendo built its system around its controller - ultra low latency was built into the design, while Vita Remote Play simply doesn't have that priority for Sony. With regards Shield, as details come to light on the Kepler GPU's onboard video encode architecture, we see a design built from the ground-up for low-latency image acquisition, along with high-speed hardware h.264 encoding - a system that has attracted plenty of praise from luminaries in the real-time capture field (Unwinder is the man behind MSI Afterburner). This is technology derived from GeForce GRID, Nvidia's cloud gaming system - so just like the Wii U GamePad, it's built for precisely this task.
"As it stands, PS4's Vita Remote Play functionality is a really cool value-added extra, but while fun, it can't really be compared with the Wii U GamePad experience."
The hardware encoders and surrounding pipelines built into both Xbox One and PlayStation 4 seem to be less sophisticated - absolutely fine for the clip-sharing and live-streaming functions they were primarily designed for, but with results falling short for ultra-low latency real-time transmission that Vita Remote Play demands for optimal performance.
The metrics aren't as good as we were hoping for, but we do have to stress that what feels laggy for one person could feel absolutely fine for another - what we've attempted to do here is to back our observations with hard numbers using proven methodology. The experience will change from game to game too: the lower the input lag on the title, the better the Remote Play experience is likely to be. Call of Duty prides itself on its low latency controls and the game felt OK when we played it via Vita, but Resogun just felt wrong in all of our testing across multiple connections, PS4s, Vitas and routers. On the flipside, Assassin's Creed 4 is generally quite latency tolerant anyway, and that worked fine for the most part, though once you see the lag, it's really difficult to "unsee". Rarely has the phrase "your mileage may vary" felt so apt.
However, whether we're talking about image quality, latency, frame-rate - or the simple fact that 1080p gaming doesn't always translate well to a five-inch screen - we couldn't help but feel that we were getting a less than ideal experience, and that nothing compares to just picking up the Dual Shock 4 and playing PS4 games in the conventional way. Perhaps - like the mooted Nvidia Shield enhancements - we'll see significant improvements to the functionality over time. However, in the here and now, Vita Remote Play is a cute function that's useful for some games but not for others - and right now we definitely see it as a cool value-added extra rather than the saviour of Sony's brilliant - but beleaguered - handheld.