Music Week: The many ways of making music in games (and what they mean)
Welcome to a week of articles about music and games.
Hello! This week we're going to be celebrating the intersection of music and games in many of its forms, and we're calling it Music Week. Kicking things off is Malindy and a piece about the pleasure of making stuff.
A while back now, Google Arts and Culture released Blob Opera, a virtual choir designed by Google's artist in residence, David Li. This virtual choir isn't just something you listen to, it's a toy with the tactile charm of a Google doodle. On screen, you have four blobs that with their round eyes and big mouths remind me of Rayman's best friend Globox. Each of them represents a voice in a choir - bass, tenor, mezzo-soprano and soprano. While I felt the slight of this choir not including a contralto voice, I was soothed by how fun it was to play with. Making music is easy - drag a blob upwards and it stretches like taffy, changing pitch. Drag it forward and it sings a different vowel sound. You can only control one blob at a time, the rest will automatically follow suit. It's a deceptively simple setup, but it allows you to make music with little more than your finger or your mouse. You can even record it and share it with others.
The story behind Blob Opera makes everything about it even better, because it's a genuine feat of technology, rather than a cute toy. Blob Opera is showcase of machine learning - four performers taught a computer music by singing to it, and what you hear when you use the opera yourself is a machine's interpretation of music, and more importantly, how several people are supposed to sound together, rather than just pre-recorded samples by artists that you can play around with.
I find this way to make music fascinating because I instinctively want to dislike it. Whenever we talk about the limits of machine learning, we like to use the arts as a prime example of all the things humans will always beat computers at. We assume no computer will ever write a poem like a human, or make music like one, because only humans have that special sauce called emotion, the real ingredient of any worthwhile piece of art. Humans, we like to think, have the element of surprise on their side, because we recognise how something is supposed to work and have the ability to deliberately defy expectations.
To be honest, I couldn't tell that the Blob Opera wasn't a choir of humans at all. Something about it sounds slightly off, but like anything that is implemented mechanically, there are a thousand reasons for this barely perceptible weirdness other than a literal computer using human voices to sing.
Harmonix's game Fuser is another great example of a machine interpreting music. Here's a free demo for Switch. In this game, you play a festival DJ, and you make music with a set of turntables and tracks from popular songs, which you can mix and match to create something new. You can also use an in-game sampling pad to create loops to use. Fuser works on the basis of an algorithm that adjusts pitch and speed on the fly, so that tracks fit together seamlessly. While Fuser has a lot more moving parts than Google's mini opera, the overall end goal is the same - to create harmony. To that end, song snippets have been pre-selected for maximum compatibility with each other. Still, the best part of Fuser is when things work that you didn't expect to work - who would've thought the bass from The Clash's Rock the Casbah is a great accompaniment to Cardi B's Bodak Yellow?
You might argue that there are enough people that could in fact have anticipated unlikely songs working in genres outside their own - that's how you get great cover songs, for example. But the ability to tell what goes together well is a learned skill, one Harmonix suggests people can learn from playing Fuser. It's a game that encourages experimentation, and not everything will sound good, but to find the limits of what's possible is often half the fun in games. You can learn more about what makes something sound good to you simply by trial and error.
In a way, that's how humans invented music. They hit things and later pressed keys or plucked strings until pleasant sounds emerged. Making music is like playing a game, so it's not surprising they both share the same verb of action. Making music is also like designing a game, in that there are rules, both to follow and to break, when creating something, and that there is a lot of the aforementioned trial-and-error involved in both.
Music making has made its way into games in many forms, which shows how irresistible it is. You don't even need it to be the focus of the whole experience. When I was a teen for example, I was fascinated by a mini-game in The Curse of Monkey Island. In it, Guybrush has to beat a pirate at banjo playing to make him a member of his seafaring crew. All you do is watch what note your opponent plays last and mirror him, but I found it amazing that you could make your character emulate the act of playing an instrument.
A more recent, and much more sophisticated example is Ellie playing guitar in The Last of Us 2. The game has an interesting way to translate guitar playing to controller outputs, which feels like learning the controls to an entirely new game, but allows you to play a wide variety of songs faithfully. Last year, Emma played Christmas carols in Rust by playing a real keyboard and then making do with the game's version of a piano.
For the upcoming Goodbye Volcano High, Jacob Blommestein created a whole new music tool that you can try out on the game's Discord. It's deceptively simple but another great showcase of the many different ways you can bang out a quick tune similar to the way you create your island theme in Animal Crossing: New Horizons.
I like all these different approaches to music because they highlight the fun aspect of making something. In just about anything you make, as well as in gaming, there can be a bit of an inhibition threshold to overcome. Mastery is very inspiring but also intimidating. We see someone who's good at something, and we want to get where they are, but to do so, we need to develop our skills, and that's daunting. Musicians have had their version of the FromSoftware difficulty debate for years, if you want. Don't get me wrong, mastery can be greatly satisfying and give you a goal to work towards. But not everyone is made for or interested in a challenge.
These attempts to make music in games, be it via Rust's piano or the Blob Choir, to me showcase that sometimes all you want or need is a playground, something to noodle around with. They're about satisfaction in the seemingly simple. With their help, you'll end up creating something, sometimes by deliberately bending the rules. Most importantly, to me these things exist not so much for sophistication or experimentation - they're there for fun.