Nvidia rolls out new AI model which creates audio from text

Nvidia

Continuing its charge in AI innovation, Nvidia has rolled out a new AI model which can create audio from text prompts. Called Fugatto, (Foundational Generative Audio Transformer Opus), it generates or transforms any mix of music, voices and sounds described with prompts using any combination of text and audio files. “While some AI models can compose a song or modify a voice, none have the dexterity of the new offering,” said Nvidia in a blog post.

How will the model work?

Fugatto can create a music snippet based on a text prompt, remove or add instruments from an existing song, change the accent or emotion in a voice. Furthermore, it can even let people produce sounds never heard before.

“We wanted to create a model that understands and generates sound like humans do,” said Rafael Valle, a manager of applied audio research at Nvidia.

Nvidia said that Fugatto was made by a diverse group of people from around the world, including India, Brazil, China, Jordan and South Korea. “Their collaboration made Fugatto’s multi-accent and multilingual capabilities stronger,” said the company.

Who can use Fugatto?

According to Nvidia, music producers could use Fugatto to prototype or edit an idea for a song, trying out different styles, voices and instruments. They could also add effects and enhance the overall audio quality of an existing track.

Video game developers could use the model to modify prerecorded assets in their title to fit the changing action as users play the game. Also, Nvidia believes ad agencies could apply Fugatto to target an existing campaign for multiple regions or situations, applying different accents and emotions to voiceovers.