PS5 could adjust game music on the fly — and read your emotions

The PS5 could use AI to dynamically adjust game audio based on how you're playing — and how you're feeling.

Sony’s ‘Dynamic Music Creation in Gaming’ patent spotted by GameRant, explains how machine learning algorithms could analyse different parts of a musical score and deliver variations of the music in reaction to how a game is being played. The idea is the music will further trigger feelings within a player and help make for a more engaging experience.

PS5 vs. Xbox Series X: Which new console will win?
I'm picking Xbox Series X over PS5 — here's why
Plus: PS5 DualSense leak may have just revealed PS5 release date

The patent explains that the AI system will analyse both a musical score and a game’s scenes to generate metadata for both to effectively map out which bits could trigger certain emotions. Those emotions include tension, joy, wonder, tenderness, nostalgia, sadness, fear and transcendence.

After that, the system would then map the music motifs based on the emotions certain “game vectors” (action, characters, locations, and the player’s character) are expected to trigger. The end result is expected to deliver game music in a more dynamic fashion that fits how someone is playing a game and not necessarily with what’s going on in the game’s environment.

Sony AI audio patent — (Image credit: Patent Scope)

In essence, if a certain part of a game would normally have tense music but the player wasn’t really reacting to it, then Sony’s system could ramp up the music tempo or make the music more emotional rather than just tense. A lot of games do something similar, but that only happens in pre-configured circumstances; Sony’s patent envisions a much more dynamic approach.

Sony didn’t give an example of this in action but we could hazard a guess as to how it might work. For example, if there was a particularly action-packed or ominous situation in Assassin’s Creed Valhalla or Cyberpunk 2077, the system could not only ramp up the tempo in the music but also add in an ominous discordant violin sound or a deeper baseline to make the soundtrack have more emotional heft.

Analysing emotions

This all sound like some pretty smart stuff, but Sony wants to go another step further and try and figure out a players emotions and adapt the game’s music accordingly. Sony plans to do this by analysing a whole lot of data, including matching comments in reviews with areas of the game where some form of emotional impact was expected and then timestamping that section in an electronic score so that it can be adjusted to better evoke certain emotions.

The patent also mentions other data analysis that could fuel the AI system, including tweaking a soundtrack based on who the player is. Sony could use personal data, if granted the permission to do so, to work out if a player is young or old, or introvert or extrovert, and what their personality might be gleaned from social media information.

Horizon: Zero Dawn — (Image credit: Sony)

Believe it or not, Sony might not stop there, as the patent also mentions a way to use biometric devices to read a players pulse, respiration and blood pressure, and other physical information. The PS5 could use that data to better inform the AI system on how the player might be feeling. This could then be used, for example, to ramp up the music to something tenser and dramatic if a player has a low heart rate and doesn’t seem interested in a section of a game that’s meant to be exciting.

Such a system might trigger privacy concerns, as some people might want to game without the feeling that they are being watched and analyzed by a Sony AI. But it’s also an interesting way to add something new into future games that don’t just involve fancier graphics or higher resolutions.

The PS5 wasn’t explicitly mentioned in the patent, but Sony has championed how its next-generation console will have a 3D audio processing unit to deliver more immersive sound. So there’s a chance that this AI tech could find its way into the PS5. And having such audio tech could be one way to put Microsoft’s Xbox Series X on the back foot.

Currently, the Xbox Series X has the advantage in power, but the PS5 has a faster SSD storage system. But an audio system that could transform games beyond just faster frame rates and slicker effects could be an ace up Sony’s sleeve that might see the PS5 win the upcoming next-generation console war.

Roland Moore-Colyer a Managing Editor at Tom’s Guide with a focus on news, features and opinion articles. He often writes about gaming, phones, laptops and other bits of hardware; he’s also got an interest in cars. When not at his desk Roland can be found wandering around London, often with a look of curiosity on his face.