Tech Interview: LucasArts' frame-rate upscaler

The basic idea is to build an extra frame based of the previous one [using] new information available for the current, new frame, and present it in the middle of 30FPS rendering while still working on the current frame. This way, technically, we are seeing an effective 60FPS with all the advantages of rendering at 30FPS.

Well, one of the points that I was trying to make with this presentation is that quite complex things can be actually very simple once you start thinking about them. Working things out by analogy, by trying to see why something happens. All those have indirect influence and that's why it is harder to see a simpler solution. Very often people ask, "Why are you doing this or that? It has nothing to do with the problem whatsoever." The point is that it does. You might not find the solution to the original problem, but very often you find some other interesting things that can lead your curiosity further, and this way you can find unexpected solutions to various problems.

Human eye and the whole visual system is one big inspiration to me. It makes you think, it makes you wonder. A while ago, after reading a book called "On Intelligence" by Jeff Hawkins, I prototyped a few neural networks with feedback connections, simulating the missing input from the blind spot. In fact, the network can predict what you would see as well as respond to certain optical illusions the way the real visual system does.

I also did a lot of experimenting, trying to put different patterns around the blind spot and try to observe what happens. That gave an idea that it is definitely pattern-based, it is localised but not around the "edge" and it is not too wide. Then I thought about what it would look like in motion and the fact that it has to not violate predictions our brain makes about the image. We would tend to notice things that change rather than things that don't change.

It is a set of ideas like this that are used to our advantage.

This is not necessary in general. As I discussed in comments, all the characters could be re-rendered at 60FPS, or could be rendered separately from the environment. It happens that they can move any way they want. So the easiest way is to have a version of the environment without the characters, and you use that version whenever we detect an artefact.

Now, with forward rendering it is easy to render the environment first, store it somewhere and then render characters on top of it. But it becomes very difficult to do the same thing with the use of deferred techniques. So when I was working on a console implementation I didn't want even to think about redoing most of our deferred rendering pipeline. It is already crazily optimised with different sorts of tricks. That's why I thought it would be easier to just somehow remove them from the existing frame and then use that to remove the artifacts.

But I must note that it is more than just a character removal. It is also to remove all other problematic regions. In the demo that you've seen it is only used for characters though. This is what I mean by saying that the talk should not be understood literally.