Portal de Periódicos da CAPES

Exploring Realtime Conversational Virtual Characters

2022; Volume: 131; Issue: 3 Linguagem: Inglês

10.5594/jmi.2022.3153646

2160-2492

Ha Nguyen, Aansh Malik, Michael Zink,

Human Motion and Animation

Advancements in artificial intelligence (AI) such as speech to text (STT), language understanding models, language generation models, and text to speech (TTS) enable various types of applications, one of which is realtime conversational virtual characters. Building an end-to-end framework with the right AI technology components enables relatable and multidimensional virtual characters, who can naturally converse in creatively controlled domains, while consistently maintaining their state and personality in predetermined narratives. In this work, we designed such a conversational framework with interchangeable and loosely coupled components to support granular creative details in character performance, efficiency in mass creation of virtual characters, and flexibility to embrace future improvements of each component in the fields. We then evaluated the robustness and modularity of the framework by creating Melodie, a virtual character who is fond of music and is a fan and promoter of the Eurovision Song Contest. With Melodie, we went through the full cycle from processing a speaker’s audio signals, to generating a proper response using a natural language generation model, to synthesizing the response in a character’s voice font, to finally synchronizing the synthesized response with corresponding body and facial movements to produce a coherent and believable character performance. Testing and analyzing the implementation of Melodie brought forth areas of improvement and ethical considerations that are, and continue to be, essential to the design of our future applications involving virtual characters .