Iwan de Kok, Derya Ozkan, Dirk Heylen, Louis-Philippe Morency: “Learning and Evaluating Response Prediction Models using Parallel Listener Consensus”

November 9, 2010 | Beijing, China

Speaker: Iwan de Kok, Derya Ozkan, Dirk Heylen, Louis-Philippe Morency
Host: 12th International Conference on Multimodal Interfaces and 7th Workshop on Machine Learning for Multimodal Interaction

Traditionally listener response prediction models are learned from pre-recorded dyadic interactions. Because of individual di erences in behavior, these recordings do not capture the complete ground truth. Where the recorded listener did not respond to an opportunity provided by the speaker, another listener would have responded or vice versa. In this paper, we introduce the concept of parallel listener consen- sus where the listener responses from multiple parallel interactions are combined to better capture di erences and similarities between individuals. We show how parallel listener consensus can be used for both learning and evaluating probabilistic prediction models of listener responses. To improve the learning performance, the parallel consensus helps identifying better negative samples and reduces outliers in the positive samples. We propose a new error measurement called fConsensus which exploits the parallel consensus to better de ne the concepts of exactness (mislabels) and completeness (missed labels) for prediction models. We present a series of experiments using the MultiLis Corpus where three listeners were tricked into believing that they had a oneon- one conversation with a speaker, while in fact they were recorded in parallel in interaction with the same speaker. In this paper we show that using parallel listener consensus can improve learning performance and represent better evaluation criteria for predictive models.