Game Audio Innovation: Can Games Go That Little Bit Further?

There are many technical advancements that are being studied which will change the way game audio is approached. Ideas such as procedural audio and generative music are areas surrounding this research into the advancement of game audio. They look into creating real time audio assets whilst the game is running, using analysis and physical modeling as a way to recreate the sounds. Although this leads to a list of positives (non-repetition, reduced memory footprint and more control), it also comes with its negatives (CPU usage, difficult to debug). This is why it is currently being looked into, as a way to get around the cons and begin implementing this approach into games. We can see the generative music approach already happening in some games, one example being Spore, where renowned composer Brian Eno has implemented a generative and reactive music system which cooperates with the games events.

Creative Innovation

Although technology is one area that guarantees to innovate the way games operate, I want to discuss how games can be furthered aesthetically and inspire people on an emotional level. Game audio/games themselves have experienced a substantial increase in opportunities due to new developments that allow complete freedom in approach. As new technologies arise, so do new approaches to sound in games. It is evident that many games are leaning towards the Hollywood style of production, and with the industry growing bigger and bigger each year, hiring the London Symphony Orchestra to play the score seems a reasonable idea. But then again, is it entirely necessary? Can games instead go artistically further than those forms of media they base themselves upon?

It takes a lot to create something so sonically innovative in games, that it can change our musical preferences and expectations. Typically, people presume that the more time and funds a sound designer or composer has, the closer the audio will reside to these assumptions. This is usually true in most cases, as more money equals more opportunities to enhance the production (Such as hiring musicians and orchestras). However, I personally like to think opposite of this approach can yield greater results, when approached carefully….

Limbo

Limbo (2010), an arcade game from Playdead studios is renowned for it’s innovative and refreshing approach to both game and audio design. The Audio Director for the game is ‘Martin Stig Anderson’ who prior to Limbo, worked in the field of acousmatic music. His experience in this genre of music is present in the game, he utilises the audio in a unique way to fortify the aesthetics of the game. By only using samples usually associated with sound design, alongside the software on his computer, the unique score for limbo was born.

‘For me it has a much bigger psychological impact when you turn a naturalistic soundscape into abstraction by making your sound effects play as music rather than adding some traditional background music.’

-Martin Stig Anderson

Limbo definitely boasts the idea of less is more. The plot itself is completely ambiguous and left for the player to make sense of; therefore this style of music goes hand in hand with the style of the game.

3mins40 –Example of musical approach

What really strikes me about its sonic approach is how the boundaries between music and sound design have been blurred to create a completely immersive atmosphere, as reiterated by Anderson:

‘What I found interesting in relation to audiovisual media was that soundscape and acousmatic music together embraces the entire continuum between representational and abstract sound, in this way dismissing the traditional dividing line between sound design and music’

-Martin Stig Anderson

This approach undoubtedly comes from principles associated with acousmatic music and when its incorporated into a game that shares similar intentions, it delivers outstanding results. It may be down to the nature of the game itself as to why it is so effective, but maybe the idea of merging the sound design and music into one medium could be taken into other games. In my opinion, it helps the game to feel like a finished product on a very professional scale.

7:12 – Sound Design/music

This approach to a soundtrack can also be seen in many independent and arthouse films, arguably due to the budget for production that they have. The blurring of sound design and music is musical device that can be used to create ambiguity within the on screen visuals. An example of this effect in PI:

Limbo is a recent game, like many others, that has reminded game developers that games have the power and resources to be regarded as artistic or something completely innovative. They can now communicate emotional ideas on a much larger scale than a film, creating a completely new kind of immersive media output that inspires both those who play games and also those who design them.

The Last of Us

Wow, this game looks cool. Liking the number of gameplay possibilities there appears to be, there's another playthrough which shows a different approach to gameplay here. The dialogue and events are different in each version, with the player able to interact with almost anything. It seems that the disk is ready to stream one of many animations/events at all times and it looks good!

Realistic audio and reverb, sounds good!

Adaptive Musical Scores as a Functional Device in Games

In response to 'The Year I Gained Courage to Ignore Video Game Music'

The above article, in a nutshell, discusses one mans personal opinion that suggests music in a video game can be seen as superfluous. The writer of the article talks about how his feelings towards game music changed in 2010, where he opted to multitask during his gaming sessions and alternatively listen to a podcast or his own music on his iPod, instead of the non-diegetic music featured in the game. Red Dead Redemption acts as the basis for discussion and it was the game that triggered realisation. He talked about how RDR's score, not that it is a bad one, can be viewed as 'an optional garnish' to the aesthetic beauty of the game. The writer does however demonstrate his appreciation of how the musical soundtrack enhances a particular moment in the game. The below clip is the moment in which the player first steps foot/hooves on Mexican soil.

The song is 'Far Away' By 'Jose Gonzalez', it is only heard the once in the entire game, and in my opinion is an effective and refreshing approach to utilising modern game soundtracks. The thought that a player may turn the music off in the game, or listen to audio from an alternative source provokes a slight cause for concern amongst game developers. The intended emotions and narrative direction can become lost without the music present to evoke them.


Music as a Functional Device

This idea that music in games is superfluous, can be looked at in an opposing light, in some cases the audio is a functional device to help guide the player through the game. Take the game L.A. Noire as an example, where its music acts as a device in the crime cases to guide the player towards succession of a case. Within the first mission in the game, the HUD displays comments about how the player should use sound to their advantage (around 6 minutes in).

As shown above, the game gives indicators to listen to the audio and all these was devised using FMOD. The chimes are used to indicate an area of interest for the player, so often when they step past a clue in the case, the chimes are triggered. The music also fades when the player has located all of the clues, which demonstrates its functional use in-game to support the narrative. Without the sound, the player would lose a sense of direction within the game and its intentions would appear to be disjunct and unclear.

Games that are based in an open world environment, such as LA Noire, can use music or sound to guide the player through the narrative and supply them with audible information that allows them navigate through the game successfully. The Assassins Creed series is a another prime example in which adaptive music also serves a functional purpose within the game. In a stealth game, where the NPC's state is not always clear, music can act to inform. When the player is roaming the environment freely, they are able to do whatever they like, for example at any time they may choose to attack an NPC. The music or absence thereof can be used to inform the player if they have been spotted by an enemy guard or other game events. Playing the game with the absence of music could easily be done, but the quality of gameplay would significantly decrease as the player isn't receiving the intended information planted by the creators of the game. Also worth mentioning is the multiplayer mode in AC3, the player relies on sound in order to locate their target and pursuer, whispers indicate your pursuer is nearby and heartbeats indicate that your target is close. Although these are more sound effects, they share the same purpose.

To wrap up, if the music in a game acts to aid the gameplay in an informative way, then it is a necessary garnish to include whilst playing. I can however see it from another point of view, I
if it doesn't serve as functionality and its repetitive in its nature, then it could ultimately lead to distraction and a negative disposition to the game. One possible solution to this problem could be the inclusion of a generative music system, so that the music heard is never the same twice.

The Sound of L.A. Noire

Good read about audio implementation in L.A. Noire complete with some picture maps outlining some interesting stuff!

The Sound of L.A Noire

TED Talks: Gaming

Two inspirational and interesting TEDtalks about gaming - Effects on human health and Reward Systems:

1. Jane Mcgonigal on 'The game that can give you 10 extra years of life'. 
www.ted.com/talks/jane_mcgonigal_the_game_that_can_give_you_10_extra_years_of_life.html

2. Tom Chatfield on '7 ways games reward the brain' 
www.ted.com/talks/tom_chatfield_7_ways_games_reward_the_brain.html

Cited from Talk two by Tom Chatfield:

We evolved over hundreds of thousands of years to be satisfied by the world in particular ways; and to be intensely satisfied as a species by learning and problem-solving. Perhaps the most amazing thing about the virtual arenas that games create is that we are now able to reverse-engineer that, and to produce environments that exist expressly to tick our evolutionary boxes and to engage us.

When it comes to games themselves, the “fun first” principle is an absolute: before anything and everything else, a game must be fun. Not everything can be made into a game, though; and it’s simply misleading to think of games as potential solutions to all our ills. So I’ve come up with seven larger ideas for motivating and engaging people on the basis of observing many games’ stunning power as engines of human engagement:

1. Using an experience system. This is something that Jesse Schell has talked about brilliantly in this last year, and that is actually being done in places like Indiana University. Don’t have grades, for example: give students an avatar or a profile that levels up steadily based on things like attendance and performance. Everything should count in some way towards this precisely-measured, steady individual progression: a far more intimate, involving and nuanced way of measuring progress over time than most conventional means. 

2. Multiple long and short-term aims. You break something down into many parallel tasks. You don’t just to say to someone, do 5,000 sums, or 100, or even 50: you create a whole spectrum of larger and smaller objectives that help people take take ownership of their progress, and keep them feeling they are progressing and succeeding – as well as targeting particular sets of skills. 

3. You reward for effort. People should be credited for everything they try and do. Don’t punish failure. Instead, reward and reinforce, and make everything count towards a clear measure of progress. As I’ve said elsewhere, one of the most profound transformations we can learn from games is how to turn the sense that someone has “failed” into the sense that they “haven’t succeeded yet.” 

4. Rapid, clear, frequent feedback. This is absolutely central to all forms of learning and engagement. With many of the most intractable problems in the world today, like global warming and pollution, it can be almost impossible to learn or understand something when consequences and feedback are distant from causes. Showing a clear link between things, and allowing people to experience this experimentally, allows learning to take place: you need to be shown and to experience exactly how an action plays out, what it caused, whether your attempt worked or not. 

5. Uncertainty. This is the real neurological gold mine so far as gaming is concerned. Dopamine elevates when you get a little prize for doing something, but what really lights up the brain is the unexpected reward: the one that couldn’t be predicted. And so the right amount of well-calibrated uncertainty can create intense engagement in all manner of tasks. 

6. Windows of enhanced attention. This is about using the emerging field of neurological modelling to identify those moments when attention and memory are enhanced in the brain by an elevated dopamine level, and putting learning into them – literally dropping the nugget of fact into those few seconds when attention is elevated. It’s early days here, but the potential of the field is vast. 

7. Other people. If games should remind us of one crucial aspect of our evolutionary natures more than any other, it’s that reward is not just money or personal achievement points; and it’s not just solitary individuals slumped in front of screens: it’s the intense validation of doing something in comparison and in collaboration with others. 

Collective engagement can be transformed by the unprecedented laboratory that virtual worlds offer for observing group psychology and motivation; from analyzing Guild structures in games to exploring how the public visibility of participants’ levels of achievement can encourage both competition and collaboration. This is, for me, perhaps the most thrilling area of all.

Sound propagation and the 'Stealth' Genre

'Sound propagation' is the term used to explain the behavior of sound in an environment. When creating a game, we want to accurately reproduce the acoustics of the virtual environment as it would sound in reality. This is no only so that the player can have an understanding as they relate it to the real world, but also to help increase immersion into the game. Accurate reproduction of sound propagation in games consists of multiple aspects, such as Reflection (Reverberation), Refraction (Re-direction of wave) and Attenuation/spatialisation (Where the sound is situated and its drop-off properties).

Reflection and Refraction
See the link below and select 'sound from air to water' to get a basic idea of how it propagates:

www.phys.hawaii.edu/~teb/java/ntnujava/propagation/propagation.html

Essentially, this behavior can be reproduced through using the correct reverb. In UDK, it would require too much CPU to run an accurate propagation model within a map. So to prevent this problem, it provides pre calculated reverb effects in which you can choose from. You can apply one of these presets which seems appropriate using the [Reverb Volume].

Attenuation and spatialisation

Spatiailisation is basically the realisation of the sound in a 3D environment, it allows a sound to emanate from 3D space. A lot of the things we hear in games are off screen due to our field of vision and spatialisation of sounds helps inform us of our surroundings. Attenuation and sometimes filtering when designing games can give a sense of distance from the object to the player. There are many types of distance algorithms to consider when designing attenuation and in games, these are:

• Linear - Unrealistic in the real world, but can be used for crossfading between looped audio and room tones.
• Logarithmic - For sounds more suited to be heard at a closer distance.
• Inverse - Good for loud and distant objects. Quiet at max radius but can be heard from far away.
• LogReverse - Loud at the max radius.
• Natural Sound - Realistic attenuation curve.

In UDK, these algorithms can be selected using the 'Attenuation' node.

Sound Propagation in the stealth genre

Accurate reconstruction of sound propagation is crucial in more genres of gaming than others. One example of this is the stealth genre, in which the player relies on sound for the most part to receive information on NPC whereabouts and the geometry of the environment. 'Dishonored', a recent release from Bethesda Softworks looks into this idea carefully:

http://vimeo.com/45113835#at=0

The sound changes when the player is in another room and the NPC is behind a door, also when the NPC walks in to the same room, the sound informs the player of this through adding clarity to the audio.

Another game which springs to mind is 'Thief', a similar game to Dishonored which used the same idea of accurate and carefully considered sound propagation. It seems essential in stealth games to accurately reproduce the behavior of sound dependent on the geometry of the game.

A final example taken from a fun 2D game called 'Mark of the Ninja' demonstrates the importance of sound within the stealth genre:

Sound is so important to the flow of the game,that it is emphasized through the ability to see it.

Aspects taken from:
The Game Audio Tutorial - (Book by Richard Stevens and Dave Raybould)
The Propagation of Sound - http://www.jhu.edu/virtlab/ray/acoustic.htm
UDK Documentation -http://udn.epicgames.com/Three/SoundCueReference.html

Non Repetitive Design - Why it matters

Non-Repetitive Design – Why it matters

One of the main aspects that game developers set out to achieve when creating games is that the experience of playing it remains immersive throughout. The idea of immersion is that you can play the game without being distracted towards something that seems unnatural in its context. This is achieved not only through the game itself but also through the audio. With the audio, we ideally want realism and spontaneity in its design, much like the ‘real world’. Having minimal repetition with the audio in a game can help to indulge the gamer into complete immersion and fulfil their expectations, without interruption.

RAM budget and streaming allowance

Something that can be considered responsible for repetition within games, is the streaming allowance and RAM available for the audio content. As a sound designer in games, keeping to the correct memory usage whenever possible and ensuring you are saving space whenever possible can help to keep the size of the audio to a minimum.  A couple of ways of achieving this are as follows:

• Using audio compression to decrease the size of a file. (UDK has an in built compression available, named ‘Vorbis’.)
• Editing out silence in a sample. Silence takes up just as much space as audible sound does, so by carefully editing it out of samples you can save on file size.
• Being meticulous with Sample Rates and down-sampling files when necessary. A concept known as the ‘Nyquist Shannon Sampling Theorem’ states that you only need to sample a sound at twice its frequency in order to accurately reproduce it. For example, if the frequency content of a sound peaks at 22 khz then you would need to sample it at 44khz in order to ensure fully audibility of the sound. These method of memory saving can give promising results.

Non-repetitive design considerations

Once the above rules are adhered to, you can begin to look at ways in which you can create an unpredictable and non-repetitive soundscape for a game. The following are a couple of ideas that can be considered:

• You can use pitch shift to determine the weight or size of an object instead of using multiple samples. In most cases, a higher pitched sample can be used for a smaller object and vice versa. In terms of footsteps you can vary the pitch dependent on the surface or size of the character.
• Re-modelling a sound using pitch shift to create something else fit for another source. For example pitching a drum hit down to create a low frequency rumble.
• Pitch up/Play down – Pitch it up in your DAW and play it at half the speed in UDK to save half the size.
• Non Standard Randomization – Sound playback system that has a bias against recently triggered sounds.

In UDK, you can randomise sounds using [AmbientSoundNonLoop] or SoundCues. You can apply slight pitch, filter and volume variations to create many outcomes. Furthermore, if you deconstruct a sound of its various qualities, you can then split these qualities into individual categories, and this then allows more control over both sounds and they can be randomised and played back simultaneously. This significantly increases the combinations of playback and therefore decreases the repetition in the sound heard. An example of this could be proposed with the fundamental sounds within an explosion ; you have the sound of the initial explosion, followed by falling debris (and some other stuff). Consider that there are multiple samples of the explosion and multiple samples for the falling debris, each branched off into different randomised nodes. Using a concatenator the sounds can then be mixed and matched creating a numerous array of combinations to significantly decrease repetition.

Middleware can be used in this nature aswell, to randomly playback sounds to create different combinations. This FMOD tutoral video demonstrates on a basic scale how you can get sounds to playback differently each time to prevent repetition, in this case an explosion debris sample.

References:

-  R. Stevens, D Raybould (2011). The Game Audio tutorial. Oxford: ELSEVIER

Fallout: New Vegas - The Mojave Soundscape

CRIMES Function - Orientation

One aspect that strikes me about the soundtrack for Fallout: New Vegas, is that the music, sound design and overall ambience in the game all blend seamlessly in between one another to create a unique and fully immersive environment for the player. By working in unison so effectively, the resultant audio helps to express the desolate and post-apocalyptic qualities that the Mojave Wasteland sets out to exhibit whilst reacting to the landscape as accurately as it can.

Through further research into the implementation of the audio in the game, it became apparent that the audio team had thought hard about the music before the game was produced. They had devised a music system, which consisted of 5 'types' of music, as a way to aid the interactive outcome of the game. These five types of music are categorised as 'scripted', 'incidental', 'location', 'battle' and 'hostile'. One of the types in which I found of great interest is the 'location music', which operates in relation to where the player is situated on the map and thier distance from an art object within the game. There are usually three layers of music radiating outwards from an art object and these layers help to blend the music into the virtual atmosphere. See below.

This process within a game relates to the idea of 'Orientation', where music is used as a navigational device to direct you towards points of interest (in this case a town). 

An example of this system can be heard in 'Goodsprings', the first town you encounter. Arranged music can be heard in its full entity when situated inside the heart of the town, this can be referred to as Layer 1. As you move the character away from the town and into the Wasteland, you hear the music becoming more sparse in its consistency and less audible via seamless transitions through Layers 2 and 3. However, If the player is deep inside the Mojave Wasteland and isolated from civilisation, 'incidental music' can be heard instead, which is basically short segments of the music played randomly. The incidental music integrates with the ambience of the wasteland, blurring the boundaries between the music and sound design, whilst creating an unpredictable and ambiguous atmosphere to suit the game itself. This approach to adaptive/interactive music aids the natural flow of the game and toys with the player's subconscious, subtly informing them of locations and other things nearby. Using music sparsely also keeps the soundtrack fresh and prevents it from becoming too repetitive. 

The following youtube clip demonstrates the layers idea in detail:

A detailed article by Scott Lawlor (Audio Director for Bethesda) which looks closely into the implementation of the musical systems within the game can be found here.