Can sports data analysis figure out exactly how much Mario Kart screws you?
When maths and Mario Kart meet.
"Much of what we experience in life results from a combination of skill and luck," writes esteemed analyst Michael Mauboussin. And while he doesn't name-check Mario Kart explicitly, I think we all know what he's getting at.
Mauboussin's 2012 book The Success Equation is all about figuring out the relative values of skill and luck. The former head of global financial strategies at Credit Suisse even managed to work out a method for plotting the big US team sports along a 'skill-luck continuum', where pure luck activities like roulette live at one end, and a pure skill endeavour like chess lives at the other. Basketball, apparently, lies much closer to the skill end than ice hockey, despite huge reserves of talent exhibited in both sports - it all comes down to factors like the number of scoring opportunities per game, and the relative impact of each team's best player. But it doesn't stop at the NBA or NHL - it can broadly be applied to almost anything. Like Mario Kart, for example.
It was my first thought, anyway. I looked over his complex mathematics, pieced together from techniques used by the sports data analysis 'moneyball' crowd and applied to things like high-risk investing and business strategies, and I thought: finally. Finally, I can figure out exactly how much Mario Kart 8 screws me over.
Wouldn't that be nice? To know precisely how unfair that final lap blue shell is in some quantifiable way, and to find a zen-like acceptance in the knowledge that however perfectly you take the corners, luck will always play a greater role in your destiny than skill? Or perhaps the numbers would reveal that the cream really does always rise to the top, and that with sufficient prowess you can always overcome the random race-hampering factors. Either way, applying these half-understood statistical analysis techniques to a bunch of Mario Kart 8 race results is the next important step for the committed racer, I decided.
Mauboussin asserts that you can quickly decide whether a given activity is about pure luck based on whether it's possible to lose on purpose. You can't throw a game of roulette, for example, or ensure a set of losing lottery numbers, therefore they're pure luck activities. In Mario Kart 8, it's simple to lose on purpose. Just don't do anything when the race starts: guaranteed 12th place. Therefore, there's some skill involved. But how much?
In order to work that out, the methodology gets a bit more complicated. Mauboussin's formula was created to analyse US team sports, and that doesn't make it easy to apply to a power-up-laden kart racer, for a few reasons. Firstly, it relies on the below formula:
Var(observed) = Var(true) + Var(random)
In other words, the variance of observed results is equal to the variance of skill, versus the variance of luck. It's essentially the Pythagorean theorem of statistics:
Var(X±Y) = Var(X) + Var(Y)
...but tweaked for purpose. It can't be applied verbatim to Nintendo's premier racing franchise-cum-impotent fury machine because unlike a game of basketball, baseball, or the other team sports Mauboussin discusses, it has more than two outcomes. Russell Westbrook can only win or lose a game; Dry Bowser can finish anywhere between first and twelfth position each race. The working in Mauboussin's example also uses the season records for 30 teams and their subsequent winning percentage as the basis for the first part of that formula, the variance of observed results. In other words, it's designed to measure the values of luck and skill when teams face each other, not to determine those values when an individual competes against the sport itself, as is the case in Mario Kart 8.
But with a little modification, perhaps that sports data analysis technique can still be used to gain some insight into this cruelest of kart racers. Logging my results in each of the 48 races, and in the process creating a spreadsheet that would provoke genuine worry from any onlookers, I gathered my own observed data. Quantifying luck requires a binary outcome - win or lose - so nuances like exact placings would be jettisoned in favour of a simple win percentage. That, in turn, could be used to compare theoretical win percentages in all-luck and all-skill scenarios. Using F1 2016 as a counterpoint, a rather more sensible and predictable racing game, I'd plot both titles on the skill-luck continuum and... retire on the groundbreaking nature of my results? Offer my services to The Oakland Athletics? It's not always clear what drives these endeavours. Perhaps they'd at least offer some insight into how we perceive luck and skill in games, and how developers gently massage those perceptions with a whole host of under-the-hood tricks.
Firstly, what would a Mario Kart 8 race look like if its outcome were determined purely by chance? It's already been established that it isn't, but does Baby Luigi really have just as much chance of winning a race as the player? When we spoke to director Kosuke Yabuki recently, we asked him how the team cultivates that perception of luck in every race. "We do collect and analyse play data," he says. "Is the same person coming first all the time? In a given race, what was the distance or time difference between first place and last place? Ultimately we make a judgement [based] on: does this feel fun?" That's fun for the player, mind you. Baby Luigi's enjoyment is not paramount.
Therefore there must, you'd imagine, be a few factors going on behind the scenes to give you a more than equal chance of taking victory than your AI opponents, but for the purposes of this formula let's remove those factors and say every racer has a one in twelve chance of winning, or 0.083 expressed as a decimal. In F1 2016 there are 22 drivers on each starting grid, so the chance of winning drops to 0.045. So in this pure luck fantasy world, in which Lakitu simply draws the results out of his cloud and a cabal of F1 conspirators decide classifications by tombola, you're much more likely to win in Mario Kart 8 than in F1 2016. With skill removed from the results, a given competitor would win one in every twelve races in Mario Kart 8, and 1 in every 22 races in F1 2016.
Those decimals provide part of the picture: on the skill-luck continuum, the closer that figure is is to zero, the nearer to a pure luck result it is. With those figures in hand, what would these two games be like if luck played no role at all, and skill always proved the deciding factor? In that instance you assume a competitor with better results will always beat one with poorer results. Look, I don't mean to be a jerk about it, but over the course of my 48-race Mario Kart 8 experiment, I was that better competitor. My win percentage stood at 54.2%, and the fact my percentage edged over 50% is significant. As NFL analyst Brian Burke posited in his own work on luck in sports, when you're talking about a theoretical all-skill scenario, the competitor with the best individual record enjoys a 0% probability that they're up against better opposition in subsequent matchups. My win percentage in the all-skill fantasy world therefore shoots up to... well, 100%.
So that's a 1.0 chance of winning in an all-skill world, and a 0.083 chance in an all-luck world. Converting my actual win rate to a decimal to plot on the skill-luck continuum, Mario Kart 8 sits roughly in the middle at 0.542. The maths that created it are hugely simplified versus Mauboussin's own equation, and the sabermetrician Tom Tango who inspired him, but it's something. And its central, edging-toward-skill position silences the voice that wants to call B.S. in every race. I'd been quite put out a number of times during the course of those races, writing notes after I'd been hit by what I deemed an inordinate volume of red shells, or "hit by absolutely nothing", or when Yoshi seemed to have limitless mushrooms. But in the end, it seemed, skill - or lack thereof - outweighs random chance.
In F1 2016 it's a different story. Over a 21-race season my win rate was a rather ignominious 10.2%. Both Mercedes drivers, and Vettel, landed more wins than me, which means in an all-skill scenario there's... er, no chance of winning the race. At all. Hamilton had the best record, ergo he goes on to win every race in the all-skill simulation. As I mentioned, there are problems with applying this thinking to non-binary outcomes. Converting that win rate to a decimal of 0.10 plots F1 2016 much closer to the luck end of the continuum than Mario Kart. The maths are skewing it, obviously, but the rags-to-riches-to-rags mechanics of Nintendo's game have been out-randomised by clinical, predictable F1 this time.
If I wasn't feeling frustrated by the ethereal Mario Kart's refusal to adhere to a luck/skill equation before, Kosuke Yabuki guarantees it by revealing that "There isn't actually a defined formula" for power-ups.
"It's not a set algorithm which determines how often you'd get something like a blue shell," he says. "We're always playing it and trying it... With the goal being to keep it enjoyable for everyone, if there was a set formula which you could analyse and understand, I think it would make it less enjoyable." It's almost vindicating, that: there's no set formula for chance in the game, so how can there be one for the results it produces?
However much stock you take in the sabermetrician approach to sports, and to games about plumbers tossing bananas out of moving vehicles, the endeavour does confirm what, hand on heart, we all know is going on in Mario Kart: it's a game of skill that presents us with excuses for failure.
It isn't the randomness of results that seems unfair (my own results ranged only from first to sixth places), but the tacit message that first place is the correct result, and that anything less is a loss. It doesn't throw insurmountable obstacles that can't be controlled, either. In 48 races I was hit by an average of 2.91 items each race, and in only eight of those races did I avoid projectiles altogether. However in races in which I recorded no mistakes or off-track excursions (it was, as I mentioned, a worryingly comprehensive spreadsheet), that average dropped to 1.38. That isn't revelatory, either. You know driving well minimises your chances of being hit, but it's hard to parse that while screaming into a sofa cushion. In other words, the game isn't picking on you because it hates you. Even though it has every right to, frankly, after the things you've called it.
Mauboussin describes luck as "the events or circumstances that operate for or against an individual. Luck, he says, "is above and beyond skill". Is there anything in Mario Kart - in gaming - that honestly can't be mitigated by a bit of player skill? COD killstreaks feel like a profound injustice from the receiving end, but they happen because someone used their skill to generate them. Your Hearthstone hand might suck, but a bit more deckbuilding nous might have remedied it. And certain racing games might prove infuriating enough to drive someone towards statistical analysis for answers and excuses, but the right lines can negate even the most inopportune blue shell.