Portal de Periódicos da CAPES

Human-Computer Interaction Approach with Empathic Conversational Agent and Computer Vision

2024; Springer Science+Business Media; Linguagem: Inglês

10.1007/978-3-031-61140-7_41

1611-3349

Rafael Pereira, Carla Mendes, Nuno Costa, Luís Frazão, Antonio Fernández‐Caballero, Ántónio Pereira,

Human Pose and Action Recognition

The integration of empathy in Human-Computer Interaction (HCI) is essential for enhancing user experiences. Current HCI systems often overlook users' emotional states, limiting interaction quality. This research examines the integration of Multimodal Emotion Recognition (MER) into empathic generative-based conversational agents, encompassing facial, body, and speech emotion recognition, along with sentiment analysis. These elements are fused and incorporated into Large Language Models (LLMs) to continuously comprehend and respond to users empathically. This paper highlights the advantages of this multimodal approach over traditional unimodal systems in recognizing complex human emotions. Additionally, it provides a well-structured background on the addressed topics. The findings include an overview of deep learning in HCI, a review of methods used for emotion recognition and conversational agents, and the proposal of an HCI architecture that integrates facial, body, and speech emotion recognition and sentiment analysis into a fusion model that is fed into an LLM making an empathic conversational agent. This research contributes to the field of HCI by providing an architecture to guide the development of more realistic and meaningful HCIs through MER and a conversational agent.