Etudiant : EL HAMDAOUI JAMAL

Filière : Master Big Data Analytics & Smart Systems (BDSaS)

Encadrant : Pr. NFAOUI EL HABIB

Annèe : 2024

Résumé : In recent years, emotion recognition has become a focal point of extensive research, particularly in the field of exploiting multimodal data. According to our studies, we have identified three principal limitations: (i) Limited applicability in real-world scenarios due to the assumption of pre-aligned multimodal data. (ii) Inadequate utilization of long-term dependencies across modalities. (iii) Insufficient exploitation of correlations among emotion labels. In this work, we started by adopting a pseudo-alignment algorithm (PAA), as a solution for the first limitation. We used PAA because our focus was on unaligned data. Subsequently, we proposed a Multi-Modal Data Interaction Process (MDIP) architecture that integrates text, audio, and video sequences and exploits long-term dependencies, and a Deep Reinforcement Learning Emotion Detection DRLED model that learns the relationships between labels. Extensive experiments conducted on the public benchmark dataset IEMOCAP demonstrate that our architecture yields promising results in emotion detection using multimodal data without pre-alignment.