Stefan Scherer, Louis-Philippe Morency and Albert “Skip” Rizzo: “Multisense and SimSensei — A Multimodal Research Platform for Real-time Assessment of Distress Indicators”

October 19, 2012 | Arlington, VA

Speaker: Stefan Scherer, Louis-Philippe Morency and Albert "Skip" Rizzo

Overview: USC Institute for Creative Technologies (ICT) is a leader in basic research and advanced technology development of virtual humans who think and behave like real people. ICT brings together experts in clinical psychology, cognitive science, computer vision, speech processing and artificial intelligence.

As part of our recent DARPA-funded project Detection and Computational Analysis of Psychological Signals (DCAPS), we perfected our expertise in automatic human behavior analysis to identify indicators of psychological distress such as depression, anxiety and PTSD. We developed two technologies which are particularly relevant to the Predicting Suicide Intent program:

  • MultiSense automatically tracks and analyzes in real-time facial expressions, body posture, acoustic features, linguistic patterns and higher-level behavior descriptors (e.g. attention, fidgeting). MultiSense infers indicators of psychological distress from these signals and behaviors, and sends this information to healthcare providers or the virtual human system.
  • SimSensei is a virtual human platform specifically designed for healthcare support and is based on the 10+ years of expertise at ICT in virtual human research and development. The platform enables an engaging face-to-face interaction where the virtual human automatically reacts with its own speech and gestures to the perceived user state and intent.

In a recent project with Cincinnati Children’s Hospital, these sensing technologies were used to automatically identify the characteristics of prosody and voice quality of individuals who had attempted suicide in the past.