Derya Ozkan, Kenji Sagae, Louis-Philippe Morency: “Latent Mixture of Discriminative Experts for Multimodal Prediction Modeling”

August 23, 2010 | Beijing, China

Speaker: Derya Ozkan, Kenji Sagae, Louis-Philippe Morency
Host: 23rd International Conference on Computational Linguistics (COLING 2010)

During face-to-face conversation, people naturally integrate speech, gestures and higher level language interpretations to predict the right time to start talking or to give backchannel feedback. In this paper we introduce a new model called Latent Mixture of Discriminative Experts which addresses some of the key issues with multimodal language processing: (1) temporal synchrony/asynchrony between modalities, (2) micro dynamics and (3) integration of different levels of interpretation. We present an empirical evaluation on listener nonverbal feedback prediction (e.g., head nod), based on observable behaviors of the speaker. We confirm the importance of combining four types of multimodal features: lexical, syntactic structure, eye gaze, and prosody. We show that our Latent Mixture of Discriminative Experts model outperforms previous approaches based on Conditional Random Fields (CRFs) and Latent-Dynamic CRFs.