AI upscaling the classic Metal Gear Solid 2 E3 2000 trailer
Can we remaster one of the best trailers of all time to 4K resolution?
It's a total coincidence of course, but on the day that Metal Gear Solid creator Hideo Kojima muses on Twitter that AI will eventually take over the remastering of old games, Digital Foundry publishes a video that uses machine learning to freshen up the classic MGS2 reveal from E3 2000. For my money the original is one of the greatest game trailers of all-time - and for a number of reasons, opportunities to see this video the way it was meant to be seen are limited, so digging out the best archival version there is and AI-upscaling it was irresistible.
The trailer itself is simply incredible, even today - and for a number of reasons. First of all, it was a genuine real-time showcase. This was PlayStation 2 producing visuals of quality we'd never seen before - genuine 'next-gen' stuff was on display before the machine was even released in Europe. Secondly, the technology on display wasn't just about graphics, it was about systems. Across the nine minutes of intense action, we got to see environmental destruction systems, AI, stealth, body part-specific damage and much more. And finally, there was direction, with cinematic cameras set-up to capture the action: the MGS trailer had shades of James Cameron and Michael Bay in its presentation and likely many more of Kojima's influences. Yes, there were hints of the story in there too but over and above that, Kojima wasn't afraid to insert humour too. The trailer blew us away with its content, then left us with a smile on our faces at the end.
Of course, the fact that the trailer was released in the summer of 2000 means that distribution of a decent quality version of it was problematic, certainly online. The internet didn't have any kind of decent video playback platform at the turn of the century - and YouTube is packed with horrific low resolution, low frame-rate renditions of the original asset to this very day. I worked on a PlayStation magazine at the time and Konami did share a Beta-SP version of the trailer with us for our cover-mounted DVDs, but the tape was returned and a PAL converted rendition is all we have left.
However, thankfully, a good quality version of the asset does exist, albeit in very limited quantities. Konami itself realised it was onto something special with this trailer and released it on DVD in Japan, so finally we could appreciate it at its original 60fps. My colleague John Linneman lovingly processed the asset and shared it on YouTube in 2017 - and that DVD is the basis for the AI upscaled version.
Topaz Video Enhance AI is the tool I used for the job, and the task facing it is daunting. We have a 480i source, meaning that to create a 4K60 version, the deep learning model needs to work with just 240 lines of source image per frame. Recent renditions of the tool have included processing models designed for interlaced sources - two of them, in fact. One is designed for interlaced material from low quality sources like VHS cassettes while the other is aimed at DVDs and 1080i tapes.
The results are certainly intriguing: text is easily upscaled, as are the HUD elements seen when Snake uses his tranquiliser pistol. Most dramatic at all is the model's treatment of Yoji Shinkawa's iconic artwork, showcased later on in the trailer. Elsewhere, the results are mixed but always interesting. The internal areas shown in the trailer are deftly handled, and it's actually quite remarkable to see how the external shots in the storm see detail dramatically increased - perhaps too much, to the point of over-sharpening at some points. Elements like the soldiers' fatigues gaining clarity and definition are easy to process, but the way that the AI model finds detail within the storm can beggar belief. The biggest limitation comes down to the fact that at its heart, MGS2 at the time was field-rendered - only ever delivering 240 lines per frame. The AI model isn't trained to smooth out the most egregious jaggies and in some scenes, there is little gain at all - just an extra clarity.
However, as Kojima says, the future is AI and tools like Topaz Video Enhance AI are already in use throughout the industry, most notably deployed for gaming remasters. Textures are being 'remastered' with AI, while full-motion video sequences rendered at a lower resolution gain substantially from AI upscaling. The cleaner the original source, the more an AI is able to do with it. But there is a certain AI look to the material. Outputs can be hit or miss too: in my experiments what I've found to be most limiting is that actual tweaking of the output picture is limited to non-existent, and the models themselves are like 'black boxes'. As a user we have no understanding of how it does what it does and therefore how to improve its outputs when it gets something wrong. In the case of my efforts here, I can't help but wonder how much better the results would be if the training data came entirely from video games.
Ultimately, the technology is hugely exciting - but we are still at the early days and there is a long road ahead. Deep learning has some way to go not just in terms of intelligence but also in terms of speed: the full trailer is around nine minutes long but took about three hours to process. That's with an AI studying material with no additional help. Nvidia DLSS has shown that a more focused application tied directly into a game engine produces results that are orders of magnitude faster and better - and in the short to medium term, that's where we're likely to see the most interesting and dramatic examples of how deep learning can radically improve the quality of the games we play.