Analysis of human emotion is very important as the field of social robotics where a new generation of humanoids and other smart devices will interact with humans. Emotional expression is a universal language for interaction with humans. Understanding human emotions is a necessary and important step for human-computer interaction. Human emotion is expressed as a complex combination of facial expressions, speech (including silence) and gestures postures, various limb-motions, gaze, and blinking. Multiple research models have been developed for limited facial expression analysis, speech based emotion analysis, limited models for gesture analysis and their limited integration. However, such analysis is limited to single frame analysis time-efficiency, limited handling of occlusion, notion of colors in facial expression analysis, lack of exploitation of symmetry, lack of dynamic change in assigning weight between the modalities based upon environmental requirement and six basic emotions.
This research develops a convolutional neural network based deep learning model that recognizes human facial expressions exploiting a combination of symmetrical representation to handle occlusion; a unified model based upon transforming facial muscle motion to geometric feature points; fusion of multiple modalities and fast hashing techniques for real-time emotion recognition. It also proposes a new model for recognition of mixed-emotion in real-time.