Nonverbal Communication and Computation

When people interact with each other, it is common to see indications of acknowledgment given with a simple head gesture or explicit turn-taking with eye gaze shifts. People use visual feedback-visual information transferred during interaction-to communicate relevant information and to synchronize rhythm between participants. Novel computer interfaces should also have the ability to recognize visual feedback when interacting with a user. For example, a robot or virtual agent should be able to recognize when a user is confused or when the user is busy with another task, understand who is talking to whom in a multi-party conversation, and be aware of what physical objects in an environment a user is referring to. The recognition of visual feedback is a key component of human communication, and novel multimodal interfaces need to recognize and analyze these visual cues to facilitate more natural human-computer interaction. Visual feedback for multimodal interfaces is a multi-disciplinary research topic that overlays the fields of computer vision, human-computer interaction, machine learning and artificial intelligence, and has many practical applications in areas as diverse as robotics, education, and entertainment.

This project differs from others in the following ways:

Sensing and recognizing nonverbal behaviors is an important research topic in computer vision. The main novelty of our approach is that we want to understand nonverbal behaviors in a multimodal and context-based fashion. Recognizing visual gesture should not only be based on the video images from the camera but should also take into account the other input modalities (e.g., spoken words and prosody from the same person) as well as the context of the interaction (e.g., actions and gestures from other people or virtual agents). To be able to really understand gestures, we need to build a recognition framework that can analyze the relation between these contextual cues and multimodal input signals.

Tags: communication, human, interaction, non-verbal, virtual

View All Projects »

This video requires the latest version of Adobe Flash Player

Team

Louis-Philippe Morency

Contact

Louis-Philippe Morency,

Goals

Enabling this form of interaction in machine interfaces requires both advances in the understanding of visual feedback as naturally performed by people when interacting with computer systems-identifying the most relevant cues that machines can perceive-and most importantly, the development of efficient and robust algorithms to recognize these visual gestures. When recognizing visual feedback, people use more than their visual perception; knowledge about the current topic and expectations from previous utterances help guide recognition of visual cues such as head nodding and facial expressions. Since many of these visual gestures are subtle, the use of contextual information helps people to disambiguate between voluntary and non-voluntary gestures.

Virtual humans
Integrated Virtual Humans
Cognition and Emotion
Tactical questioning
Sensing nonverbal behaviors

Institute for Creative Technologies