In this talk I will present our work on modeling the interplay of emotional components during conversation. Initial work focuses on predicting emotional experience from facial behavior and physiological responses. Quantitative models for emotion processes are desirable for suggesting psychological theory, and for applications to HCI such as as on-line education systems and robots which need to interact sensibly with humans.

Human emotion is a complex multi-component system that includes emotional experience, behaviors, and physiological responses. Prior work has mainly focused on recognizing facial expressions posed by actors who may or may not experience any particular emotion while doing so. By contrast, we have an interesting psychology data set collected during an emotional conversation between two women who have just met. It includes facial/vocal behaviors captured with video cameras and physiological measurements as well as a self-reported quantitative measure of feelings that are spontaneously induced in conversation. In this study, we are developing models for statistically linking relevant facial behaviors, physiological responses, and emotional experience (within and between partners) in this noisy data set.

I will first describe an Active-Appearance-Model based approach to track faces in large pose change and self occlusion. I will then describe a simple but effective way of extracting pose-invariant facial features. Finally, I will discuss how we link facial and physiological features to emotional experience by using a statistical model. Our preliminary results suggest that physiological responses are more reliable cues for predicting self reported positive/negative experience and that facial behaviors can differ from experience significantly if one attempts to suppress emotional expression (one of the manipulations in the experiment).

Joint work with Kobus Barnard, Emily Butler, and James Gross.