Machine Perception for Understanding Social Behavior

This research concerns the computational tools we build with the explicit intent of understanding human social behavior. In one project, we perform automatic analysis of ultrasound data of the human tongue during speech. In another project, we perform automatic analysis of human facial expressions to make predictions about a variety of behaviors, such as whether or not a person is showing true or fake pain, or true or fake joy, determining whether or not they understand a lecture, or determining if they are about to fall asleep while driving.

The machine perception primitives we develop are useful in two ways. First, they enable analysis of massive amounts of human behavioral data that were previously unheard of among psychologists. For instance, through analysis of several hours of subjects either faking or truly experiencing their arm being submerged in ice water, our group determined that certain facial action units that had not previously been predicted by psychologists were highly predictive of true vs. false pain. In addition, the machine perception algorithms themselves can help understand how humans process this data. For instance, in the ultrasound speech study, we are testing motor theories of speech understanding in humans by determining if and how information about the articulatory apparatus is used in machine learning-based speech recognition algorithms. Finally, we are performing studies of humans interacting with robots in order to understand how to build robots that can interact better with people, largely by studying how humans change their behaviors and how they respond to different perceptual features when interacting with "intelligent" appearing machines.