With The Witcher 3: Wild Hunt, developers at CD Projekt RED felt pleased with the quality of lip-synced dialogue animations for its characters. Believing there was still room for improvement, though, the studio began searching for a way to raise the bar in Cyberpunk 2077. Most importantly, CD Projekt wanted better quality lip-sync with “muscle-driven emotional expressions” for every character in the open-world. Raising the stakes even further was the goal to achieve this feat across 10 different languages. Since facial motion capture of this scale was out of the question, CD Projekt turned to another solution–special AI technology.
In a Siggraph 2020 talk, “JALI Driven Expressive Facial Animation and Multilingual Speech in Cyberpunk 2077,” the title’s Lead Character Technical Director, Mateusz Popławskl, revealed how the novel AI technology found its way into the RPG. In 2016, Popławskl said, he stumbled across a technical paper from JALI Research regarding lip-synchronization.
Apparently, the paper described everything CD Projekt RED needed for Cyberpunk 2077, including how to approach various speech types like whispering and shouting. By contacting JALI, Popławskl found a match made in Heaven. One company needed advanced lip-sync tech, the other had the tech but needed a game to test it on.
A description of the AI’s function is as follows:
Cyberpunk 2077 Speech Animation Workflow: An audio performance and tagged animation transcript from Canada’s JALI Inc. is used to generate expressive high fidelity, multi-lingual speech at unprecedented scale.
The full talk lasts just under 20 minutes long and walks viewers through the tech’s minutiae with impressive demonstrations. Popławskl leads the discussion, alongside JALI Research’s Pif Edwards and Chris Landreth. Check it out in the video linked below:
Cyberpunk 2077 hits stores next month on November 19th for the PS4, PC, and Xbox One. Those who pick up a current-gen copy will gain free access to a next-gen version.
[Source: JALI Research Inc via DSOGaming]
