Publications
Search
281.
Parra, Federico; Miljkovitch, Raphaële; Persiaux, Gwenaelle; Morales, Michelle; Scherer, Stefan
The Multimodal Assessment of Adult Attachment Security: Developing the Biometric Attachment Test Journal Article
In: Journal of Medical Internet Research, vol. 19, no. 4, pp. e100, 2017, ISSN: 1438-8871.
@article{parra_multimodal_2017,
title = {The Multimodal Assessment of Adult Attachment Security: Developing the Biometric Attachment Test},
author = {Federico Parra and Raphaële Miljkovitch and Gwenaelle Persiaux and Michelle Morales and Stefan Scherer},
url = {http://www.jmir.org/2017/4/e100/},
doi = {10.2196/jmir.6898},
issn = {1438-8871},
year = {2017},
date = {2017-03-01},
journal = {Journal of Medical Internet Research},
volume = {19},
number = {4},
pages = {e100},
abstract = {Background: Attachment theory has been proven essential for mental health, including psychopathology, development, and interpersonal relationships. Validated psychometric instruments to measure attachment abound but suffer from shortcomings common to traditional psychometrics. Recent developments in multimodal fusion and machine learning pave the way for new automated and objective psychometric instruments for adult attachment that combine psychophysiological, linguistic, and behavioral analyses in the assessment of the construct.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Background: Attachment theory has been proven essential for mental health, including psychopathology, development, and interpersonal relationships. Validated psychometric instruments to measure attachment abound but suffer from shortcomings common to traditional psychometrics. Recent developments in multimodal fusion and machine learning pave the way for new automated and objective psychometric instruments for adult attachment that combine psychophysiological, linguistic, and behavioral analyses in the assessment of the construct.
282.
Chang, Jonathan; Scherer, Stefan
LEARNING REPRESENTATIONS OF EMOTIONAL SPEECH WITH DEEP CONVOLUTIONAL GENERATIVE ADVERSARIAL NETWORKS Proceedings Article
In: Proceedings of the 42nd IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2017, IEEE, New Orleans, LA, 2017.
@inproceedings{chang_learning_2017,
title = {LEARNING REPRESENTATIONS OF EMOTIONAL SPEECH WITH DEEP CONVOLUTIONAL GENERATIVE ADVERSARIAL NETWORKS},
author = {Jonathan Chang and Stefan Scherer},
url = {https://arxiv.org/pdf/1705.02394.pdf},
year = {2017},
date = {2017-03-01},
booktitle = {Proceedings of the 42nd IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2017},
publisher = {IEEE},
address = {New Orleans, LA},
abstract = {Automatically assessing emotional valence in human speech has historically been a difficult task for machine learning algorithms. The subtle changes in the voice of the speaker that are indicative of positive or negative emotional states are often ”overshadowed” by voice characteristics relating to emotional intensity or emotional activation. In this work we explore a representation learning approach that automatically derives discriminative representations of emotional speech. In particular, we investigate two machine learning strategies to improve classifier performance: (1) utilization of unlabeled data using a deep convolutional generative adversarial network (DCGAN), and (2) multitask learning. Within our extensive experiments we leverage a multitask annotated emotional corpus as well as a large unlabeled meeting corpus (around 100 hours). Our speaker-independent classification experiments show that in particular the use of unlabeled data in our investigations improves performance of the classifiers and both fully supervised baseline approaches are outperformed considerably. We improve the classification of emotional valence on a discrete 5-point scale to 43.88% and on a 3-point scale to 49.80%, which is competitive to state-of-the-art performance. Index Terms— Machine Learning, Affective Computing, Semisupervised Learning, Deep Learning},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Automatically assessing emotional valence in human speech has historically been a difficult task for machine learning algorithms. The subtle changes in the voice of the speaker that are indicative of positive or negative emotional states are often ”overshadowed” by voice characteristics relating to emotional intensity or emotional activation. In this work we explore a representation learning approach that automatically derives discriminative representations of emotional speech. In particular, we investigate two machine learning strategies to improve classifier performance: (1) utilization of unlabeled data using a deep convolutional generative adversarial network (DCGAN), and (2) multitask learning. Within our extensive experiments we leverage a multitask annotated emotional corpus as well as a large unlabeled meeting corpus (around 100 hours). Our speaker-independent classification experiments show that in particular the use of unlabeled data in our investigations improves performance of the classifiers and both fully supervised baseline approaches are outperformed considerably. We improve the classification of emotional valence on a discrete 5-point scale to 43.88% and on a 3-point scale to 49.80%, which is competitive to state-of-the-art performance. Index Terms— Machine Learning, Affective Computing, Semisupervised Learning, Deep Learning
283.
Nouri, Elnaz; Georgila, Kallirroi; Traum, David
Culture-specific models of negotiation for virtual characters: multi-attribute decision-making based on culture-specific values Journal Article
In: AI & SOCIETY, vol. 32, no. 1, pp. 51–63, 2017, ISSN: 0951-5666, 1435-5655.
@article{nouri_culture-specific_2017,
title = {Culture-specific models of negotiation for virtual characters: multi-attribute decision-making based on culture-specific values},
author = {Elnaz Nouri and Kallirroi Georgila and David Traum},
url = {http://link.springer.com/10.1007/s00146-014-0570-7},
doi = {10.1007/s00146-014-0570-7},
issn = {0951-5666, 1435-5655},
year = {2017},
date = {2017-02-01},
journal = {AI & SOCIETY},
volume = {32},
number = {1},
pages = {51–63},
abstract = {We posit that observed differences in negotiation performance across cultures can be explained by participants trying to optimize across multiple values, where the relative importance of values differs across cultures. We look at two ways for specifying weights on values for different cultures: one in which the weights of the model are hand-crafted, based on intuition interpreting Hofstede dimensions for the cultures, and one in which the weights of the model are learned from data using inverse reinforcement learning (IRL). We apply this model to the Ultimatum Game and integrate it into a virtual human dialog system. We show that weights learned from IRL surpass both a weak baseline with random weights and a strong baseline considering only one factor for maximizing gain in own wealth in accounting for the behavior of human players from four different cultures. Wealso show that the weights learned with our model for one culture outperform weights learned for other cultures when playing against opponents of the first culture.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
We posit that observed differences in negotiation performance across cultures can be explained by participants trying to optimize across multiple values, where the relative importance of values differs across cultures. We look at two ways for specifying weights on values for different cultures: one in which the weights of the model are hand-crafted, based on intuition interpreting Hofstede dimensions for the cultures, and one in which the weights of the model are learned from data using inverse reinforcement learning (IRL). We apply this model to the Ultimatum Game and integrate it into a virtual human dialog system. We show that weights learned from IRL surpass both a weak baseline with random weights and a strong baseline considering only one factor for maximizing gain in own wealth in accounting for the behavior of human players from four different cultures. Wealso show that the weights learned with our model for one culture outperform weights learned for other cultures when playing against opponents of the first culture.
284.
Lucas, Gale M.; Gratch, Jonathan; Malandrakis, Nikolaos; Szablowski, Evan; Fessler, Eli; Nichols, Jeffrey
GOAALLL!: Using Sentiment in the World Cup to Explore Theories of Emotion Journal Article
In: Image and Vision Computing, 2017, ISSN: 02628856.
@article{lucas_goaalll_2017,
title = {GOAALLL!: Using Sentiment in the World Cup to Explore Theories of Emotion},
author = {Gale M. Lucas and Jonathan Gratch and Nikolaos Malandrakis and Evan Szablowski and Eli Fessler and Jeffrey Nichols},
url = {http://linkinghub.elsevier.com/retrieve/pii/S0262885617300148},
doi = {10.1016/j.imavis.2017.01.006},
issn = {02628856},
year = {2017},
date = {2017-01-01},
journal = {Image and Vision Computing},
abstract = {Sporting events evoke strong emotions amongst fans and thus act as natural laboratories to explore emotions and how they unfold in the wild. Computational tools, such as sentiment analysis, provide new ways to examine such dynamic emotional processes. In this article we use sentiment analysis to examine tweets posted during 2014 World Cup. Such analysis gives insight into how people respond to highly emotional events, and how these emotions are shaped by contextual factors, such as prior expectations, and how these emotions change as events unfold over time. Here we report on some preliminary analysis of a World Cup twitter corpus using sentiment analysis techniques. After performing initial tests of validation for sentiment analysis on data in this corpus, we show these tools can give new insights into existing theories of what makes a sporting match exciting. This analysis seems to suggest that, contrary to assumptions in sports economics, excitement relates to expressions of negative emotion. The results are discussed in terms of innovations in methodology and understanding the role of emotion for “tuning in” to real world events. We also discuss some challenges that such data present for existing sentiment analysis techniques and discuss future analysis.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Sporting events evoke strong emotions amongst fans and thus act as natural laboratories to explore emotions and how they unfold in the wild. Computational tools, such as sentiment analysis, provide new ways to examine such dynamic emotional processes. In this article we use sentiment analysis to examine tweets posted during 2014 World Cup. Such analysis gives insight into how people respond to highly emotional events, and how these emotions are shaped by contextual factors, such as prior expectations, and how these emotions change as events unfold over time. Here we report on some preliminary analysis of a World Cup twitter corpus using sentiment analysis techniques. After performing initial tests of validation for sentiment analysis on data in this corpus, we show these tools can give new insights into existing theories of what makes a sporting match exciting. This analysis seems to suggest that, contrary to assumptions in sports economics, excitement relates to expressions of negative emotion. The results are discussed in terms of innovations in methodology and understanding the role of emotion for “tuning in” to real world events. We also discuss some challenges that such data present for existing sentiment analysis techniques and discuss future analysis.
285.
Woolley, J. D.; Chuang, B.; Fussell, C.; Scherer, S.; Biagianti, B.; Fulford, D.; Mathalon, D. H.; Vinogradov, S.
Intranasal oxytocin increases facial expressivity, but not ratings of trustworthiness, in patients with schizophrenia and healthy controls Journal Article
In: Psychological Medicine, pp. 1–12, 2017, ISSN: 0033-2917, 1469-8978.
@article{woolley_intranasal_2017,
title = {Intranasal oxytocin increases facial expressivity, but not ratings of trustworthiness, in patients with schizophrenia and healthy controls},
author = {J. D. Woolley and B. Chuang and C. Fussell and S. Scherer and B. Biagianti and D. Fulford and D. H. Mathalon and S. Vinogradov},
url = {https://www.cambridge.org/core/product/identifier/S0033291716003433/type/journal_article},
doi = {10.1017/S0033291716003433},
issn = {0033-2917, 1469-8978},
year = {2017},
date = {2017-01-01},
journal = {Psychological Medicine},
pages = {1–12},
abstract = {Blunted facial affect is a common negative symptom of schizophrenia. Additionally, assessing the trustworthiness of faces is a social cognitive ability that is impaired in schizophrenia. Currently available pharmacological agents are ineffective at improving either of these symptoms, despite their clinical significance. The hypothalamic neuropeptide oxytocin has multiple prosocial effects when administered intranasally to healthy individuals and shows promise in decreasing negative symptoms and enhancing social cognition in schizophrenia. Although two small studies have investigated oxytocin's effects on ratings of facial trustworthiness in schizophrenia, its effects on facial expressivity have not been investigated in any population. We investigated the effects of oxytocin on facial emotional expressivity while participants performed a facial trustworthiness rating task in 33 individuals with schizophrenia and 35 age-matched healthy controls using a double-blind, placebo-controlled, cross-over design. Participants rated the trustworthiness of presented faces interspersed with emotionally evocative photographs while being video-recorded. Participants’ facial expressivity in these videos was quantified by blind raters using a well-validated manualized approach (i.e. the Facial Expression Coding System; FACES). While oxytocin administration did not affect ratings of facial trustworthiness, it significantly increased facial expressivity in individuals with schizophrenia (Z = −2.33},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Blunted facial affect is a common negative symptom of schizophrenia. Additionally, assessing the trustworthiness of faces is a social cognitive ability that is impaired in schizophrenia. Currently available pharmacological agents are ineffective at improving either of these symptoms, despite their clinical significance. The hypothalamic neuropeptide oxytocin has multiple prosocial effects when administered intranasally to healthy individuals and shows promise in decreasing negative symptoms and enhancing social cognition in schizophrenia. Although two small studies have investigated oxytocin's effects on ratings of facial trustworthiness in schizophrenia, its effects on facial expressivity have not been investigated in any population. We investigated the effects of oxytocin on facial emotional expressivity while participants performed a facial trustworthiness rating task in 33 individuals with schizophrenia and 35 age-matched healthy controls using a double-blind, placebo-controlled, cross-over design. Participants rated the trustworthiness of presented faces interspersed with emotionally evocative photographs while being video-recorded. Participants’ facial expressivity in these videos was quantified by blind raters using a well-validated manualized approach (i.e. the Facial Expression Coding System; FACES). While oxytocin administration did not affect ratings of facial trustworthiness, it significantly increased facial expressivity in individuals with schizophrenia (Z = −2.33
286.
Woo, Simon; Kaiser, Elsi; Artstein, Ron; Mirkovic, Jelena
Life-experience passwords (LEPs) Proceedings Article
In: Proceedings of the 32nd Annual Conference on Computer Security Applications, pp. 113–126, ACM Press, Los Angeles, CA, 2016, ISBN: 978-1-4503-4771-6.
@inproceedings{woo_life-experience_2016,
title = {Life-experience passwords (LEPs)},
author = {Simon Woo and Elsi Kaiser and Ron Artstein and Jelena Mirkovic},
url = {http://dl.acm.org/citation.cfm?doid=2991079.2991107},
doi = {10.1145/2991079.2991107},
isbn = {978-1-4503-4771-6},
year = {2016},
date = {2016-12-01},
booktitle = {Proceedings of the 32nd Annual Conference on Computer Security Applications},
pages = {113–126},
publisher = {ACM Press},
address = {Los Angeles, CA},
abstract = {Passwords are widely used for user authentication, but they are often difficult for a user to recall, easily cracked by automated programs and heavily reused. Security questions are also used for secondary authentication. They are more memorable than passwords, but are very easily guessed. We propose a new authentication mechanism, called "life-experience passwords (LEPs)," which outperforms passwords and security questions, both at recall and at security. Each LEP consists of several facts about a user-chosen past experience, such as a trip, a graduation, a wedding, etc. At LEP creation, the system extracts these facts from the user's input and transforms them into questions and answers. At authentication, the system prompts the user with questions and matches her answers with the stored ones. In this paper we propose two LEP designs, and evaluate them via user studies. We further compare LEPs to passwords, and find that: (1) LEPs are 30–47 bits stronger than an ideal, randomized, 8-character password, (2) LEPs are up to 3x more memorable, and (3) LEPs are reused half as often as passwords. While both LEPs and security questions use personal experiences for authentication, LEPs use several questions, which are closely tailored to each user. This increases LEP security against guessing attacks. In our evaluation, only 0.7% of LEPs were guessed by friends, while prior research found that friends could guess 17–25% of security questions. LEPs also contained a very small amount of sensitive or fake information. All these qualities make LEPs a promising, new authentication approach.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Passwords are widely used for user authentication, but they are often difficult for a user to recall, easily cracked by automated programs and heavily reused. Security questions are also used for secondary authentication. They are more memorable than passwords, but are very easily guessed. We propose a new authentication mechanism, called "life-experience passwords (LEPs)," which outperforms passwords and security questions, both at recall and at security. Each LEP consists of several facts about a user-chosen past experience, such as a trip, a graduation, a wedding, etc. At LEP creation, the system extracts these facts from the user's input and transforms them into questions and answers. At authentication, the system prompts the user with questions and matches her answers with the stored ones. In this paper we propose two LEP designs, and evaluate them via user studies. We further compare LEPs to passwords, and find that: (1) LEPs are 30–47 bits stronger than an ideal, randomized, 8-character password, (2) LEPs are up to 3x more memorable, and (3) LEPs are reused half as often as passwords. While both LEPs and security questions use personal experiences for authentication, LEPs use several questions, which are closely tailored to each user. This increases LEP security against guessing attacks. In our evaluation, only 0.7% of LEPs were guessed by friends, while prior research found that friends could guess 17–25% of security questions. LEPs also contained a very small amount of sensitive or fake information. All these qualities make LEPs a promising, new authentication approach.
287.
Chollet, Mathieu; Prendinger, Helmut; Scherer, Stefan
Native vs. Non-native Language Fluency Implications on Multimodal Interaction for Interpersonal Skills Training Proceedings Article
In: Proceedings of the 18th ACM International Conference on Multimodal Interaction, pp. 386–393, ACM Press, Tokyo, Japan, 2016, ISBN: 978-1-4503-4556-9.
@inproceedings{chollet_native_2016,
title = {Native vs. Non-native Language Fluency Implications on Multimodal Interaction for Interpersonal Skills Training},
author = {Mathieu Chollet and Helmut Prendinger and Stefan Scherer},
url = {http://dl.acm.org/citation.cfm?doid=2993148.2993196},
doi = {10.1145/2993148.2993196},
isbn = {978-1-4503-4556-9},
year = {2016},
date = {2016-11-01},
booktitle = {Proceedings of the 18th ACM International Conference on Multimodal Interaction},
pages = {386–393},
publisher = {ACM Press},
address = {Tokyo, Japan},
abstract = {New technological developments in the eld of multimodal interaction show great promise for the improvement and assessment of public speaking skills. However, it is unclear how the experience of non-native speakers interacting with such technologies di ers from native speakers. In particular, nonnative speakers could bene t less from training with multimodal systems compared to native speakers. Additionally, machine learning models trained for the automatic assessment of public speaking ability on data of native speakers might not be performing well for assessing the performance of non-native speakers. In this paper, we investigate two aspects related to the performance and evaluation of multimodal interaction technologies designed for the improvement and assessment of public speaking between a population of English native speakers and a population of non-native English speakers. Firstly, we compare the experiences and training outcomes of these two populations interacting with a virtual audience system designed for training public speaking ability, collecting a dataset of public speaking presentations in the process. Secondly, using this dataset, we build regression models for predicting public speaking performance on both populations and evaluate these models, both on the population they were trained on and on how they generalize to the second population.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
New technological developments in the eld of multimodal interaction show great promise for the improvement and assessment of public speaking skills. However, it is unclear how the experience of non-native speakers interacting with such technologies di ers from native speakers. In particular, nonnative speakers could bene t less from training with multimodal systems compared to native speakers. Additionally, machine learning models trained for the automatic assessment of public speaking ability on data of native speakers might not be performing well for assessing the performance of non-native speakers. In this paper, we investigate two aspects related to the performance and evaluation of multimodal interaction technologies designed for the improvement and assessment of public speaking between a population of English native speakers and a population of non-native English speakers. Firstly, we compare the experiences and training outcomes of these two populations interacting with a virtual audience system designed for training public speaking ability, collecting a dataset of public speaking presentations in the process. Secondly, using this dataset, we build regression models for predicting public speaking performance on both populations and evaluate these models, both on the population they were trained on and on how they generalize to the second population.
288.
Artstein, Ron; Traum, David; Boberg, Jill; Gainer, Alesia; Gratch, Jonathan; Johnson, Emmanuel; Leuski, Anton; Nakano, Mikio
Niki and Julie: A Robot and Virtual Human for Studying Multimodal Social Interaction Proceedings Article
In: Proceedings of the 18th ACM International Conference on Multimodal Interaction, pp. 402–403, ACM Press, Tokyo, Japan, 2016, ISBN: 978-1-4503-4556-9.
@inproceedings{artstein_niki_2016,
title = {Niki and Julie: A Robot and Virtual Human for Studying Multimodal Social Interaction},
author = {Ron Artstein and David Traum and Jill Boberg and Alesia Gainer and Jonathan Gratch and Emmanuel Johnson and Anton Leuski and Mikio Nakano},
url = {http://dl.acm.org/citation.cfm?doid=2993148.2998532},
doi = {10.1145/2993148.2998532},
isbn = {978-1-4503-4556-9},
year = {2016},
date = {2016-11-01},
booktitle = {Proceedings of the 18th ACM International Conference on Multimodal Interaction},
pages = {402–403},
publisher = {ACM Press},
address = {Tokyo, Japan},
abstract = {We demonstrate two agents, a robot and a virtual human, which can be used for studying factors that impact social influence. The agents engage in dialogue scenarios that build familiarity, share information, and attempt to influence a human participant. The scenarios are variants of the classical “survival task,” where members of a team rank the importance of a number of items (e.g., items that might help one survive a crash in the desert). These are ranked individually and then re-ranked following a team discussion, and the difference in ranking provides an objective measure of social influence. Survival tasks have been used in psychology, virtual human research, and human-robot interaction. Our agents are operated in a “Wizard-of-Oz” fashion, where a hidden human operator chooses the agents’ dialogue actions while interacting with an experiment participant.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
We demonstrate two agents, a robot and a virtual human, which can be used for studying factors that impact social influence. The agents engage in dialogue scenarios that build familiarity, share information, and attempt to influence a human participant. The scenarios are variants of the classical “survival task,” where members of a team rank the importance of a number of items (e.g., items that might help one survive a crash in the desert). These are ranked individually and then re-ranked following a team discussion, and the difference in ranking provides an objective measure of social influence. Survival tasks have been used in psychology, virtual human research, and human-robot interaction. Our agents are operated in a “Wizard-of-Oz” fashion, where a hidden human operator chooses the agents’ dialogue actions while interacting with an experiment participant.
289.
Core, Mark G.; Georgila, Kallirroi; Nye, Benjamin D.; Auerbach, Daniel; Liu, Zhi Fei; DiNinni, Richard
Learning, Adaptive Support, Student Traits, and Engagement in Scenario-Based Learning Proceedings Article
In: Proceedings from the Interservice/Industry Training, Simulation and Education Conference (I/ITSEC) 2016, National Training and Simulation Association, Orlando, FL, 2016.
@inproceedings{core_learning_2016,
title = {Learning, Adaptive Support, Student Traits, and Engagement in Scenario-Based Learning},
author = {Mark G. Core and Kallirroi Georgila and Benjamin D. Nye and Daniel Auerbach and Zhi Fei Liu and Richard DiNinni},
url = {http://www.iitsecdocs.com/search},
year = {2016},
date = {2016-11-01},
booktitle = {Proceedings from the Interservice/Industry Training, Simulation and Education Conference (I/ITSEC) 2016},
publisher = {National Training and Simulation Association},
address = {Orlando, FL},
abstract = {Scenario-based training systems pose an especially difficult challenge for an intelligent tutoring system (ITS). In addition to the basic problems of deciding when to intervene and what guidance to provide, the ITS must decide whether to give guidance directly (e.g., a hint message), indirectly through positive/negative results in the scenario, or to delay guidance until a post-scenario review session. There are a number of factors that an adaptive ITS should consider and we use self-report survey instruments to investigate the relationship between traits, learning strategies, expectations, learner behaviors derived from log files, post-use perceptions of the system, and pre-test and post-test results. We use the ELITE Lite Counseling training system as a testbed for our experiments. This system uses virtual role players to allow learners to practice leadership counseling skills, and is in use at the United States Military Academy (USMA). This paper analyzes two data sets. We collected data from local university students, a non-military population of roughly the same age as USMA Cadets using the system. For these local participants, we could administer surveys and pre-tests and post-tests, and collect log files recording clicks made while using ELITE Lite. The second data set comes from USMA itself but is limited to log files. In both populations, the ITS’s hints are effective at boosting scenario performance, and for the university students, the overall experience promoted learning, and survey results suggest that higher levels of organization in study habits may lead to greater learning with ELITE Lite. For the USMA Cadets, ELITE Lite is part of their Military Leadership course rather than an experiment, which could explain why we found higher scenario performance on average than the non-military population, and more use of the post-scenario review feature.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Scenario-based training systems pose an especially difficult challenge for an intelligent tutoring system (ITS). In addition to the basic problems of deciding when to intervene and what guidance to provide, the ITS must decide whether to give guidance directly (e.g., a hint message), indirectly through positive/negative results in the scenario, or to delay guidance until a post-scenario review session. There are a number of factors that an adaptive ITS should consider and we use self-report survey instruments to investigate the relationship between traits, learning strategies, expectations, learner behaviors derived from log files, post-use perceptions of the system, and pre-test and post-test results. We use the ELITE Lite Counseling training system as a testbed for our experiments. This system uses virtual role players to allow learners to practice leadership counseling skills, and is in use at the United States Military Academy (USMA). This paper analyzes two data sets. We collected data from local university students, a non-military population of roughly the same age as USMA Cadets using the system. For these local participants, we could administer surveys and pre-tests and post-tests, and collect log files recording clicks made while using ELITE Lite. The second data set comes from USMA itself but is limited to log files. In both populations, the ITS’s hints are effective at boosting scenario performance, and for the university students, the overall experience promoted learning, and survey results suggest that higher levels of organization in study habits may lead to greater learning with ELITE Lite. For the USMA Cadets, ELITE Lite is part of their Military Leadership course rather than an experiment, which could explain why we found higher scenario performance on average than the non-military population, and more use of the post-scenario review feature.
290.
Rizzo, Albert; Scherer, Scherer; DeVault, David; Gratch, Jonathan; Artstein, Ronald; Hartholt, Arno; Lucas, Gale; Marsella, Stacy; Morbini, Fabrizio; Nazarian, Angela; Stratou, Giota; Traum, David; Wood, Rachel; Boberg, Jill; Morency, Louis Philippe
Detection and computational analysis of psychological signals using a virtual human interviewing agent Journal Article
In: Journal of Pain Management, pp. 311–321, 2016, ISSN: 1939-5914.
@article{rizzo_detection_2016,
title = {Detection and computational analysis of psychological signals using a virtual human interviewing agent},
author = {Albert Rizzo and Scherer Scherer and David DeVault and Jonathan Gratch and Ronald Artstein and Arno Hartholt and Gale Lucas and Stacy Marsella and Fabrizio Morbini and Angela Nazarian and Giota Stratou and David Traum and Rachel Wood and Jill Boberg and Louis Philippe Morency},
url = {http://www.icdvrat.org/2014/papers/ICDVRAT2014_S03N3_Rizzo_etal.pdf},
issn = {1939-5914},
year = {2016},
date = {2016-11-01},
journal = {Journal of Pain Management},
pages = {311–321},
abstract = {It has long been recognized that facial expressions, body posture/gestures and vocal parameters play an important role in human communication and the implicit signalling of emotion. Recent advances in low cost computer vision and behavioral sensing technologies can now be applied to the process of making meaningful inferences as to user state when a person interacts with a computational device. Effective use of this additive information could serve to promote human interaction with virtual human (VH) agents that may enhance diagnostic assessment. This paper will focus on our current research in these areas within the DARPA-funded "Detection and Computational Analysis of Psychological Signals" project, with specific attention to the SimSensei application use case. SimSensei is a virtual human interaction platform that is able to sense and interpret real-time audiovisual behavioral signals from users interacting with the system. It is specifically designed for health care support and leverages years of virtual human research and development at USC-ICT. The platform enables an engaging face-to-face interaction where the virtual human automatically reacts to the state and inferred intent of the user through analysis of behavioral signals gleaned from facial expressions, body gestures and vocal parameters. Akin to how non-verbal behavioral signals have an impact on human to human interaction and communication, SimSensei aims to capture and infer from user non-verbal communication to improve engagement between a VH and a user. The system can also quantify and interpret sensed behavioral signals longitudinally that can be used to inform diagnostic assessment within a clinical context.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
It has long been recognized that facial expressions, body posture/gestures and vocal parameters play an important role in human communication and the implicit signalling of emotion. Recent advances in low cost computer vision and behavioral sensing technologies can now be applied to the process of making meaningful inferences as to user state when a person interacts with a computational device. Effective use of this additive information could serve to promote human interaction with virtual human (VH) agents that may enhance diagnostic assessment. This paper will focus on our current research in these areas within the DARPA-funded "Detection and Computational Analysis of Psychological Signals" project, with specific attention to the SimSensei application use case. SimSensei is a virtual human interaction platform that is able to sense and interpret real-time audiovisual behavioral signals from users interacting with the system. It is specifically designed for health care support and leverages years of virtual human research and development at USC-ICT. The platform enables an engaging face-to-face interaction where the virtual human automatically reacts to the state and inferred intent of the user through analysis of behavioral signals gleaned from facial expressions, body gestures and vocal parameters. Akin to how non-verbal behavioral signals have an impact on human to human interaction and communication, SimSensei aims to capture and infer from user non-verbal communication to improve engagement between a VH and a user. The system can also quantify and interpret sensed behavioral signals longitudinally that can be used to inform diagnostic assessment within a clinical context.
291.
Lucas, Gale; Stratou, Giota; Lieblich, Shari; Gratch, Jonathan
Trust Me: Multimodal Signals of Trustworthiness Proceedings Article
In: Proceedings of the 18th ACM International Conference on Multimodal Interaction, pp. 5–12, ACM Press, Tokyo, Japan, 2016, ISBN: 978-1-4503-4556-9.
@inproceedings{lucas_trust_2016,
title = {Trust Me: Multimodal Signals of Trustworthiness},
author = {Gale Lucas and Giota Stratou and Shari Lieblich and Jonathan Gratch},
url = {http://dl.acm.org/citation.cfm?doid=2993148.2993178},
doi = {10.1145/2993148.2993178},
isbn = {978-1-4503-4556-9},
year = {2016},
date = {2016-11-01},
booktitle = {Proceedings of the 18th ACM International Conference on Multimodal Interaction},
pages = {5–12},
publisher = {ACM Press},
address = {Tokyo, Japan},
abstract = {This paper builds on prior psychological studies that identify signals of trustworthiness between two human negotiators. Unlike prior work, the current work tracks such signals automatically and fuses them into computational models that predict trustworthiness. To achieve this goal, we apply automatic trackers to recordings of human dyads negotiating in a multi-issue bargaining task. We identify behavioral indicators in different modalities (facial expressions, gestures, gaze, and conversational features) that are predictive of trustworthiness. We predict both objective trustworthiness (i.e., are they honest) and perceived trustworthiness (i.e., do they seem honest to their interaction partner). Our experiments show that people are poor judges of objective trustworthiness (i.e., objective and perceived trustworthiness are predicted by different indicators), and that multimodal approaches better predict objective trustworthiness, whereas people overly rely on facial expressions when judging the honesty of their partner. Moreover, domain knowledge (from the literature and prior analysis of behaviors) facilitates the model development process.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
This paper builds on prior psychological studies that identify signals of trustworthiness between two human negotiators. Unlike prior work, the current work tracks such signals automatically and fuses them into computational models that predict trustworthiness. To achieve this goal, we apply automatic trackers to recordings of human dyads negotiating in a multi-issue bargaining task. We identify behavioral indicators in different modalities (facial expressions, gestures, gaze, and conversational features) that are predictive of trustworthiness. We predict both objective trustworthiness (i.e., are they honest) and perceived trustworthiness (i.e., do they seem honest to their interaction partner). Our experiments show that people are poor judges of objective trustworthiness (i.e., objective and perceived trustworthiness are predicted by different indicators), and that multimodal approaches better predict objective trustworthiness, whereas people overly rely on facial expressions when judging the honesty of their partner. Moreover, domain knowledge (from the literature and prior analysis of behaviors) facilitates the model development process.
292.
Pestian, John P.; Sorter, Michael; Connolly, Brian; Cohen, Kevin Bretonnel; McCullumsmith, Cheryl; Gee, Jeffry T.; Morency, Louis-Philippe; Scherer, Stefan; Rohlfs, Lesley
A Machine Learning Approach to Identifying the Thought Markers of Suicidal Subjects: A Prospective Multicenter Trial Journal Article
In: Suicide and Life-Threatening Behavior, 2016, ISSN: 03630234.
@article{pestian_machine_2016,
title = {A Machine Learning Approach to Identifying the Thought Markers of Suicidal Subjects: A Prospective Multicenter Trial},
author = {John P. Pestian and Michael Sorter and Brian Connolly and Kevin Bretonnel Cohen and Cheryl McCullumsmith and Jeffry T. Gee and Louis-Philippe Morency and Stefan Scherer and Lesley Rohlfs},
url = {http://doi.wiley.com/10.1111/sltb.12312},
doi = {10.1111/sltb.12312},
issn = {03630234},
year = {2016},
date = {2016-11-01},
journal = {Suicide and Life-Threatening Behavior},
abstract = {Death by suicide demonstrates profound personal suffering and societal failure. While basic sciences provide the opportunity to understand biological markers related to suicide, computer science provides opportunities to understand suicide thought markers. In this novel prospective, multimodal, multicenter, mixed demographic study, we used machine learning to measure and fuse two classes of suicidal thought markers: verbal and nonverbal. Machine learning algorithms were used with the subjects’ words and vocal characteristics to classify 379 subjects recruited from two academic medical centers and a rural community hospital into one of three groups: suicidal, mentally ill but not suicidal, or controls. By combining linguistic and acoustic characteristics, subjects could be classified into one of the three groups with up to 85% accuracy. The results provide insight into how advanced technology can be used for suicide assessment and prevention.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Death by suicide demonstrates profound personal suffering and societal failure. While basic sciences provide the opportunity to understand biological markers related to suicide, computer science provides opportunities to understand suicide thought markers. In this novel prospective, multimodal, multicenter, mixed demographic study, we used machine learning to measure and fuse two classes of suicidal thought markers: verbal and nonverbal. Machine learning algorithms were used with the subjects’ words and vocal characteristics to classify 379 subjects recruited from two academic medical centers and a rural community hospital into one of three groups: suicidal, mentally ill but not suicidal, or controls. By combining linguistic and acoustic characteristics, subjects could be classified into one of the three groups with up to 85% accuracy. The results provide insight into how advanced technology can be used for suicide assessment and prevention.
293.
Neubauer, Catherine; Woolley, Joshua; Khooshabeh, Peter; Scherer, Stefan
Getting to know you: a multimodal investigation of team behavior and resilience to stress Proceedings Article
In: Proceedings of the 18th ACM International Conference on Multimodal Interaction, pp. 193–200, ACM Press, Tokyo, Japan, 2016, ISBN: 978-1-4503-4556-9.
@inproceedings{neubauer_getting_2016,
title = {Getting to know you: a multimodal investigation of team behavior and resilience to stress},
author = {Catherine Neubauer and Joshua Woolley and Peter Khooshabeh and Stefan Scherer},
url = {http://dl.acm.org/citation.cfm?doid=2993148.2993195},
doi = {10.1145/2993148.2993195},
isbn = {978-1-4503-4556-9},
year = {2016},
date = {2016-11-01},
booktitle = {Proceedings of the 18th ACM International Conference on Multimodal Interaction},
pages = {193–200},
publisher = {ACM Press},
address = {Tokyo, Japan},
abstract = {Team cohesion has been suggested to be a critical factor in emotional resilience following periods of stress. Team cohesion may depend on several factors including emotional state, communication among team members and even psychophysiological response. The present study sought to employ several multimodal techniques designed to investigate team behavior as a means of understanding resilience to stress. We recruited 40 subjects to perform a cooperative-task in gender-matched, two-person teams. They were responsible for working together to meet a common goal, which was to successfully disarm a simulated bomb. This high-workload task requires successful cooperation and communication among members. We assessed several behaviors that relate to facial expression, word choice and physiological responses (i.e., heart rate variability) within this scenario. A manipulation of an â€oeice breaker†condition was used to induce a level of comfort or familiarity within the team prior to the task. We found that individuals in the â€oeice breaker†condition exhibited better resilience to subjective stress following the task. These individuals also exhibited more insight and cognitive speech, more positive facial expressions and were also able to better regulate their emotional expression during the task, compared to the control.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Team cohesion has been suggested to be a critical factor in emotional resilience following periods of stress. Team cohesion may depend on several factors including emotional state, communication among team members and even psychophysiological response. The present study sought to employ several multimodal techniques designed to investigate team behavior as a means of understanding resilience to stress. We recruited 40 subjects to perform a cooperative-task in gender-matched, two-person teams. They were responsible for working together to meet a common goal, which was to successfully disarm a simulated bomb. This high-workload task requires successful cooperation and communication among members. We assessed several behaviors that relate to facial expression, word choice and physiological responses (i.e., heart rate variability) within this scenario. A manipulation of an â€oeice breaker†condition was used to induce a level of comfort or familiarity within the team prior to the task. We found that individuals in the â€oeice breaker†condition exhibited better resilience to subjective stress following the task. These individuals also exhibited more insight and cognitive speech, more positive facial expressions and were also able to better regulate their emotional expression during the task, compared to the control.
294.
Traum, David
Using Dialogue System Technology to Support Interactive History Learning Journal Article
In: Journal of Japan Society of Artificial Intelligence (JSAI), vol. 31, no. 6, pp. 806, 2016.
@article{traum_using_2016,
title = {Using Dialogue System Technology to Support Interactive History Learning},
author = {David Traum},
url = {http://www.ai-gakkai.or.jp/en/en/vol31_no6/},
year = {2016},
date = {2016-11-01},
journal = {Journal of Japan Society of Artificial Intelligence (JSAI)},
volume = {31},
number = {6},
pages = {806},
abstract = {We describe the use of spoken dialogue technology to enhance informal history learning. We describe several uses for this technology, including allowing learners to engage in natural interactions at a historical site, allowing learners to talk with recreations of historical figures, and using oral history recordings of a witness to create a dialogue experience. Two projects are highlighted, one to give a guided experience of a historical location, and another, New Dimensions in Testimony, that allows an experience similar to face to face conversation with a Holocaust survivor. These techniques allow many of the bene ts of an intimate connection to historical places and people, through direct interaction and user initiative, but can also be delivered to a mass audience, formerly only reachable by broadcast, non-interactive media.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
We describe the use of spoken dialogue technology to enhance informal history learning. We describe several uses for this technology, including allowing learners to engage in natural interactions at a historical site, allowing learners to talk with recreations of historical figures, and using oral history recordings of a witness to create a dialogue experience. Two projects are highlighted, one to give a guided experience of a historical location, and another, New Dimensions in Testimony, that allows an experience similar to face to face conversation with a Holocaust survivor. These techniques allow many of the bene ts of an intimate connection to historical places and people, through direct interaction and user initiative, but can also be delivered to a mass audience, formerly only reachable by broadcast, non-interactive media.
295.
Narang, Sahil; Best, Andrew; Randhavane, Tanmay; Shapiro, Ari; Manocha, Dinesh
PedVR: simulating gaze-based interactions between a real user and virtual crowds Proceedings Article
In: Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology, pp. 91–100, ACM Press, Munich, Germany, 2016, ISBN: 978-1-4503-4491-3.
@inproceedings{narang_pedvr_2016,
title = {PedVR: simulating gaze-based interactions between a real user and virtual crowds},
author = {Sahil Narang and Andrew Best and Tanmay Randhavane and Ari Shapiro and Dinesh Manocha},
url = {http://dl.acm.org/citation.cfm?id=2993378},
doi = {10.1145/2993369.2993378},
isbn = {978-1-4503-4491-3},
year = {2016},
date = {2016-11-01},
booktitle = {Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology},
pages = {91–100},
publisher = {ACM Press},
address = {Munich, Germany},
abstract = {We present a novel interactive approach, PedVR, to generate plausible behaviors for a large number of virtual humans, and to enable natural interaction between the real user and virtual agents. Our formulation is based on a coupled approach that combines a 2D multi-agent navigation algorithm with 3D human motion synthesis. The coupling can result in plausible movement of virtual agents and can generate gazing behaviors, which can considerably increase the believability. We have integrated our formulation with the DK-2 HMD and demonstrate the benefits of our crowd simulation algorithm over prior decoupled approaches. Our user evaluation suggests that the combination of coupled methods and gazing behavior can considerably increase the behavioral plausibility.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
We present a novel interactive approach, PedVR, to generate plausible behaviors for a large number of virtual humans, and to enable natural interaction between the real user and virtual agents. Our formulation is based on a coupled approach that combines a 2D multi-agent navigation algorithm with 3D human motion synthesis. The coupling can result in plausible movement of virtual agents and can generate gazing behaviors, which can considerably increase the believability. We have integrated our formulation with the DK-2 HMD and demonstrate the benefits of our crowd simulation algorithm over prior decoupled approaches. Our user evaluation suggests that the combination of coupled methods and gazing behavior can considerably increase the behavioral plausibility.
296.
Zadeh, Amir; Zellers, Rowan; Pincus, Eli; Morency, Louis-Philippe
Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages Journal Article
In: IEEE Intelligent Systems, vol. 31, no. 6, pp. 82–88, 2016, ISSN: 1541-1672.
@article{zadeh_multimodal_2016,
title = {Multimodal sentiment intensity analysis in videos: Facial gestures and verbal messages},
author = {Amir Zadeh and Rowan Zellers and Eli Pincus and Louis-Philippe Morency},
url = {http://ieeexplore.ieee.org/abstract/document/7742221/},
doi = {10.1109/MIS.2016.94},
issn = {1541-1672},
year = {2016},
date = {2016-11-01},
journal = {IEEE Intelligent Systems},
volume = {31},
number = {6},
pages = {82–88},
abstract = {People share their opinions, stories, and reviews through online video sharing websites every day. The automatic analysis of these online opinion videos is bringing new or understudied research challenges to the field of computational linguistics and multimodal analysis. Among these challenges is the fundamental question of exploiting the dynamics between visual gestures and verbal messages to be able to better model sentiment. This article addresses this question in four ways: introducing the first multimodal dataset with opinion-level sentiment intensity annotations; studying the prototypical interaction patterns between facial gestures and spoken words when inferring sentiment intensity; proposing a new computational representation, called multimodal dictionary, based on a language-gesture study; and evaluating the authors' proposed approach in a speaker-independent paradigm for sentiment intensity prediction. The authors' study identifies four interaction types between facial gestures and verbal content: neutral, emphasizer, positive, and negative interactions. Experiments show statistically significant improvement when using multimodal dictionary representation over the conventional early fusion representation (that is, feature concatenation).},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
People share their opinions, stories, and reviews through online video sharing websites every day. The automatic analysis of these online opinion videos is bringing new or understudied research challenges to the field of computational linguistics and multimodal analysis. Among these challenges is the fundamental question of exploiting the dynamics between visual gestures and verbal messages to be able to better model sentiment. This article addresses this question in four ways: introducing the first multimodal dataset with opinion-level sentiment intensity annotations; studying the prototypical interaction patterns between facial gestures and spoken words when inferring sentiment intensity; proposing a new computational representation, called multimodal dictionary, based on a language-gesture study; and evaluating the authors' proposed approach in a speaker-independent paradigm for sentiment intensity prediction. The authors' study identifies four interaction types between facial gestures and verbal content: neutral, emphasizer, positive, and negative interactions. Experiments show statistically significant improvement when using multimodal dictionary representation over the conventional early fusion representation (that is, feature concatenation).
297.
Bernardet, Ulysses; Chollet, Mathieu; DiPaola, Steve; Scherer, Stefan
An Architecture for Biologically Grounded Real-Time Reflexive Behavior Book Section
In: Intelligent Virtual Agents, vol. 10011, pp. 295–305, Springer International Publishing, Cham, Switzerland, 2016, ISBN: 978-3-319-47664-3 978-3-319-47665-0.
@incollection{bernardet_architecture_2016,
title = {An Architecture for Biologically Grounded Real-Time Reflexive Behavior},
author = {Ulysses Bernardet and Mathieu Chollet and Steve DiPaola and Stefan Scherer},
url = {http://download.springer.com/static/pdf/224/chp%253A10.1007%252F978-3-319-47665-0_26.pdf?originUrl=http%3A%2F%2Flink.springer.com%2Fchapter%2F10.1007%2F978-3-319-47665-0_26&token2=exp=1485296780 acl=%2Fstatic%2Fpdf%2F224%2Fchp%25253A10.1007%25252F978-3-319-47665-0_26.pdf%3ForiginUrl%3Dhttp%253A%252F%252Flink.springer.com%252Fchapter%252F10.1007%252F978-3-319-47665-0_26* hmac=1bf37d11eda93937fedd36843994ffdaf645ebda569c86edbcf61ca905942f89},
isbn = {978-3-319-47664-3 978-3-319-47665-0},
year = {2016},
date = {2016-10-01},
booktitle = {Intelligent Virtual Agents},
volume = {10011},
pages = {295–305},
publisher = {Springer International Publishing},
address = {Cham, Switzerland},
abstract = {In this paper, we present a reflexive behavior architecture, that is geared towards the application in the control of the non-verbal behavior of the virtual humans in a public speaking training system. The model is organized along the distinction between behavior triggers that are internal (endogenous) to the agent, and those that origin in the environment (exogenous). The endogenous subsystem controls gaze behavior, triggers self-adaptors, and shifts between different postures, while the exogenous system controls the reaction towards auditory stimuli with different temporal and valence characteristics. We evaluate the different components empirically by letting participants compare the output of the proposed system to valid alternative variations.},
keywords = {},
pubstate = {published},
tppubtype = {incollection}
}
In this paper, we present a reflexive behavior architecture, that is geared towards the application in the control of the non-verbal behavior of the virtual humans in a public speaking training system. The model is organized along the distinction between behavior triggers that are internal (endogenous) to the agent, and those that origin in the environment (exogenous). The endogenous subsystem controls gaze behavior, triggers self-adaptors, and shifts between different postures, while the exogenous system controls the reaction towards auditory stimuli with different temporal and valence characteristics. We evaluate the different components empirically by letting participants compare the output of the proposed system to valid alternative variations.
298.
Lucas, Gale; Szablowski, Evan; Gratch, Jonathan; Feng, Andrew; Huang, Tiffany; Boberg, Jill; Shapiro, Ari
The effect of operating a virtual doppleganger in a 3D simulation Proceedings Article
In: Proceedings of the 9th International Conference on Motion in Games, pp. 167–174, ACM Press, Burlingame, CA, 2016, ISBN: 978-1-4503-4592-7.
@inproceedings{lucas_effect_2016,
title = {The effect of operating a virtual doppleganger in a 3D simulation},
author = {Gale Lucas and Evan Szablowski and Jonathan Gratch and Andrew Feng and Tiffany Huang and Jill Boberg and Ari Shapiro},
url = {http://dl.acm.org/citation.cfm?doid=2994258.2994263},
doi = {10.1145/2994258.2994263},
isbn = {978-1-4503-4592-7},
year = {2016},
date = {2016-10-01},
booktitle = {Proceedings of the 9th International Conference on Motion in Games},
pages = {167–174},
publisher = {ACM Press},
address = {Burlingame, CA},
abstract = {Recent advances in scanning technology have enabled the widespread capture of 3D character models based on human subjects. Intuition suggests that, with these new capabilities to create avatars that look like their users, every player should have his or her own avatar to play video games or simulations. We explicitly test the impact of having one’s own avatar (vs. a yoked control avatar) in a simulation (i.e., maze running task with mines). We test the impact of avatar identity on both subjective (e.g., feeling connected and engaged, liking avatar’s appearance, feeling upset when avatar’s injured, enjoying the game) and behavioral variables (e.g., time to complete task, speed, number of mines triggered, riskiness of maze path chosen). Results indicate that having an avatar that looks like the user improves their subjective experience, but there is no significant effect on how users perform in the simulation.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Recent advances in scanning technology have enabled the widespread capture of 3D character models based on human subjects. Intuition suggests that, with these new capabilities to create avatars that look like their users, every player should have his or her own avatar to play video games or simulations. We explicitly test the impact of having one’s own avatar (vs. a yoked control avatar) in a simulation (i.e., maze running task with mines). We test the impact of avatar identity on both subjective (e.g., feeling connected and engaged, liking avatar’s appearance, feeling upset when avatar’s injured, enjoying the game) and behavioral variables (e.g., time to complete task, speed, number of mines triggered, riskiness of maze path chosen). Results indicate that having an avatar that looks like the user improves their subjective experience, but there is no significant effect on how users perform in the simulation.
299.
Valstar, Michel; Gratch, Jonathan; Schuller, Björn; Ringeval, Fabien; Lalanne, Denis; Torres, Mercedes Torres; Scherer, Stefen; Stratou, Giota; Cowie, Roddy; Pantic, Maja
AVEC 2016: Depression, Mood, and Emotion Recognition Workshop and Challenge Proceedings Article
In: Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge, pp. 3–10, ACM Press, Amsterdam, The Netherlands, 2016, ISBN: 978-1-4503-4516-3.
@inproceedings{valstar_avec_2016,
title = {AVEC 2016: Depression, Mood, and Emotion Recognition Workshop and Challenge},
author = {Michel Valstar and Jonathan Gratch and Björn Schuller and Fabien Ringeval and Denis Lalanne and Mercedes Torres Torres and Stefen Scherer and Giota Stratou and Roddy Cowie and Maja Pantic},
url = {http://dl.acm.org/citation.cfm?id=2988258},
doi = {10.1145/2988257.2988258},
isbn = {978-1-4503-4516-3},
year = {2016},
date = {2016-10-01},
booktitle = {Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge},
pages = {3–10},
publisher = {ACM Press},
address = {Amsterdam, The Netherlands},
abstract = {The Audio/Visual Emotion Challenge and Workshop (AVEC 2016) "Depression, Mood and Emotion" will be the sixth competition event aimed at comparison of multimedia processing and machine learning methods for automatic audio, visual and physiological depression and emotion analysis, with all participants competing under strictly the same conditions. The goal of the Challenge is to provide a common benchmark test set for multi-modal information processing and to bring together the depression and emotion recognition communities, as well as the audio, video and physiological processing communities, to compare the relative merits of the various approaches to depression and emotion recognition under well-defined and strictly comparable conditions and establish to what extent fusion of the approaches is possible and beneficial. This paper presents the challenge guidelines, the common data used, and the performance of the baseline system on the two tasks.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
The Audio/Visual Emotion Challenge and Workshop (AVEC 2016) "Depression, Mood and Emotion" will be the sixth competition event aimed at comparison of multimedia processing and machine learning methods for automatic audio, visual and physiological depression and emotion analysis, with all participants competing under strictly the same conditions. The goal of the Challenge is to provide a common benchmark test set for multi-modal information processing and to bring together the depression and emotion recognition communities, as well as the audio, video and physiological processing communities, to compare the relative merits of the various approaches to depression and emotion recognition under well-defined and strictly comparable conditions and establish to what extent fusion of the approaches is possible and beneficial. This paper presents the challenge guidelines, the common data used, and the performance of the baseline system on the two tasks.
300.
Joshi, Himanshu; Rosenbloom, Paul S.; Ustun, Volkan
Continuous phone recognition in the Sigma cognitive architecture Journal Article
In: Biologically Inspired Cognitive Architectures, vol. 18, pp. 23–32, 2016, ISSN: 2212683X.
@article{joshi_continuous_2016,
title = {Continuous phone recognition in the Sigma cognitive architecture},
author = {Himanshu Joshi and Paul S. Rosenbloom and Volkan Ustun},
url = {http://linkinghub.elsevier.com/retrieve/pii/S2212683X16300652},
doi = {10.1016/j.bica.2016.09.001},
issn = {2212683X},
year = {2016},
date = {2016-10-01},
journal = {Biologically Inspired Cognitive Architectures},
volume = {18},
pages = {23–32},
abstract = {Spoken language processing is an important capability of human intelligence that has hitherto been unexplored by cognitive architectures. This reflects on both the symbolic and sub-symbolic nature of the speech problem, and the capabilities provided by cognitive architectures to model the latter and its rich interplay with the former. Sigma has been designed to leverage the state-of-the-art hybrid (discrete + continuous) mixed (symbolic + probabilistic) capability of graphical models to provide in a uniform non-modular fashion effective forms of, and integration across, both cognitive and sub-cognitive behavior. In this article, previous work on speaker dependent isolated word recognition has been extended to demonstrate Sigma’s feasibility to process a stream of fluent audio and recognize phones, in an online and incremental manner with speaker independence. Phone recognition is an important step in integrating spoken language processing into Sigma. This work also extends the acoustic front-end used in the previous work in service of speaker independence. All of the knowledge used in phone recognition was added supraarchitecturally – i.e. on top of the architecture – without requiring the addition of new mechanisms to the architecture.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Spoken language processing is an important capability of human intelligence that has hitherto been unexplored by cognitive architectures. This reflects on both the symbolic and sub-symbolic nature of the speech problem, and the capabilities provided by cognitive architectures to model the latter and its rich interplay with the former. Sigma has been designed to leverage the state-of-the-art hybrid (discrete + continuous) mixed (symbolic + probabilistic) capability of graphical models to provide in a uniform non-modular fashion effective forms of, and integration across, both cognitive and sub-cognitive behavior. In this article, previous work on speaker dependent isolated word recognition has been extended to demonstrate Sigma’s feasibility to process a stream of fluent audio and recognize phones, in an online and incremental manner with speaker independence. Phone recognition is an important step in integrating spoken language processing into Sigma. This work also extends the acoustic front-end used in the previous work in service of speaker independence. All of the knowledge used in phone recognition was added supraarchitecturally – i.e. on top of the architecture – without requiring the addition of new mechanisms to the architecture.
Filter
2010
Roque, Antonio; Georgila, Kallirroi; Artstein, Ron; Sagae, Kenji; Traum, David
Natural language processing for joint fire observer training Proceedings Article
In: Proceedings of the 27th Army Science Conference, Orlando, FL, 2010.
Abstract | Links | BibTeX | Tags: Virtual Humans
@inproceedings{roque_natural_2010,
title = {Natural language processing for joint fire observer training},
author = {Antonio Roque and Kallirroi Georgila and Ron Artstein and Kenji Sagae and David Traum},
url = {http://ict.usc.edu/pubs/Natural%20language%20processing%20for%20joint%20fire%20observer%20training.pdf},
year = {2010},
date = {2010-12-01},
booktitle = {Proceedings of the 27th Army Science Conference},
address = {Orlando, FL},
abstract = {We describe recent research to enhance a training system which interprets Call for Fire (CFF) radio artillery requests. The research explores the feasibility of extending the system to also understand calls for Close Air Support (CAS). This work includes automated analysis of complex language behavior in CAS missions, evaluation of speech recognition performance, and simulation of speech recognition errors.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
We describe recent research to enhance a training system which interprets Call for Fire (CFF) radio artillery requests. The research explores the feasibility of extending the system to also understand calls for Close Air Support (CAS). This work includes automated analysis of complex language behavior in CAS missions, evaluation of speech recognition performance, and simulation of speech recognition errors.
Pütten, Astrid M.; Krämer, Nicole; Gratch, Jonathan; Kang, Sin-Hwa
"It doesn't matter what you are!": Explaining social effects of agents and avatars Journal Article
In: Computers in Human Behavior, vol. 26, no. 6, 2010.
Abstract | Links | BibTeX | Tags: Virtual Humans
@article{von_der_putten_it_2010,
title = {"It doesn't matter what you are!": Explaining social effects of agents and avatars},
author = {Astrid M. Pütten and Nicole Krämer and Jonathan Gratch and Sin-Hwa Kang},
url = {http://ict.usc.edu//pubs/It%20doesnt%20matter%20what%20you%20are-Explaining%20social%20effects%20of%20agents%20and%20avatars.pdf},
year = {2010},
date = {2010-11-01},
journal = {Computers in Human Behavior},
volume = {26},
number = {6},
abstract = {Empirical studies have repeatedly shown that autonomous artificial entities, so-called embodied conver- sational agents, elicit social behavior on the part of the human interlocutor. Various theoretical approaches have tried to explain this phenomenon: According to the Threshold Model of Social Influence (Blascovich et al., 2002), the social influence of real persons who are represented by avatars will always be high, whereas the influence of an artificial entity depends on the realism of its behavior. Conversely, the Ethopoeia concept (Nass & Moon, 2000) predicts that automatic social reactions are triggered by sit- uations as soon as they include social cues. The presented study evaluates whether participants Ì belief in interacting with either an avatar (a virtual representation of a human) or an agent (autonomous virtual person) lead to different social effects. We used a 2 ô°£ 2 design with two levels of agency (agent or avatar) and two levels of behavioral realism (showing feedback behavior versus showing no behavior). We found that the belief of interacting with either an avatar or an agent barely resulted in differences with regard to the evaluation of the virtual character or behavioral reactions, whereas higher behavioral realism affected both. It is discussed to what extent the results thus support the Ethopoeia concept.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {article}
}
Empirical studies have repeatedly shown that autonomous artificial entities, so-called embodied conver- sational agents, elicit social behavior on the part of the human interlocutor. Various theoretical approaches have tried to explain this phenomenon: According to the Threshold Model of Social Influence (Blascovich et al., 2002), the social influence of real persons who are represented by avatars will always be high, whereas the influence of an artificial entity depends on the realism of its behavior. Conversely, the Ethopoeia concept (Nass & Moon, 2000) predicts that automatic social reactions are triggered by sit- uations as soon as they include social cues. The presented study evaluates whether participants Ì belief in interacting with either an avatar (a virtual representation of a human) or an agent (autonomous virtual person) lead to different social effects. We used a 2 ô°£ 2 design with two levels of agency (agent or avatar) and two levels of behavioral realism (showing feedback behavior versus showing no behavior). We found that the belief of interacting with either an avatar or an agent barely resulted in differences with regard to the evaluation of the virtual character or behavioral reactions, whereas higher behavioral realism affected both. It is discussed to what extent the results thus support the Ethopoeia concept.
Kenny, Patrick G.; Parsons, Thomas D.; Garrity, Pat
Virtual Patients for Virtual Sick Call Medical Training Proceedings Article
In: Interservice/Industry Training, Simulation and Education Conference (I/ITSEC), 2010.
Abstract | Links | BibTeX | Tags: MedVR, Virtual Humans
@inproceedings{kenny_virtual_2010,
title = {Virtual Patients for Virtual Sick Call Medical Training},
author = {Patrick G. Kenny and Thomas D. Parsons and Pat Garrity},
url = {http://ict.usc.edu/pubs/Virtual%20Patients%20for%20Virtual%20Sick%20Call%20Medical%20Training.pdf},
year = {2010},
date = {2010-11-01},
booktitle = {Interservice/Industry Training, Simulation and Education Conference (I/ITSEC)},
abstract = {Training military clinicians and physicians to treat Soldiers directly impacts their mental and physical health and may even affect their survival. Developing skills such as: patient interviewing, interpersonal interaction and diagnosis can be difficult and is severely lacking in hands-on-training due to the cost and availability of trained standardized patients. A solution to this problem is in using computer generated virtual patient avatars that exhibit the mental and physiologically accurate symptoms of their particular illness; such physical indicators as sweating, blushing and breathing due to discomfort and matching conversational dialog for the disorder. These avatars are highly interactive with speech recognition, natural language understanding, non-verbal behavior, facial expressions and conversational skills. This paper will discuss the research, technology and the value of developing virtual patients. Previous work will be stated along with issues behind creating virtual characters and scenarios for the joint forces. It will then focus on subject testing that is being conducted with a Navy scenario at the Navy Independent Duty Corpsman (IDC) School at the Navy Medical Center in San Diego. The protocol involves pre and post tests with a 15 minute interview of the virtual patient. Analysis of the data will yield results in user interactions with the patient and discuss how the system can be used for training for future deployment of these systems for medical professionals. The Virtual Sick Call Project under the Joint Medical Simulation Technology Integrated Product Team (JMST IPT) seeks to push the state of the art in developing high fidelity virtual patients that will enable the caregiver to improve interpersonal skills for scenarios that require not only medical experience, but the ability to relate at an interpersonal level, with interviewing and diagnosis skills as patients can be hiding symptoms of post-traumatic stress disorder, suicide and domestic violence.},
keywords = {MedVR, Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
Training military clinicians and physicians to treat Soldiers directly impacts their mental and physical health and may even affect their survival. Developing skills such as: patient interviewing, interpersonal interaction and diagnosis can be difficult and is severely lacking in hands-on-training due to the cost and availability of trained standardized patients. A solution to this problem is in using computer generated virtual patient avatars that exhibit the mental and physiologically accurate symptoms of their particular illness; such physical indicators as sweating, blushing and breathing due to discomfort and matching conversational dialog for the disorder. These avatars are highly interactive with speech recognition, natural language understanding, non-verbal behavior, facial expressions and conversational skills. This paper will discuss the research, technology and the value of developing virtual patients. Previous work will be stated along with issues behind creating virtual characters and scenarios for the joint forces. It will then focus on subject testing that is being conducted with a Navy scenario at the Navy Independent Duty Corpsman (IDC) School at the Navy Medical Center in San Diego. The protocol involves pre and post tests with a 15 minute interview of the virtual patient. Analysis of the data will yield results in user interactions with the patient and discuss how the system can be used for training for future deployment of these systems for medical professionals. The Virtual Sick Call Project under the Joint Medical Simulation Technology Integrated Product Team (JMST IPT) seeks to push the state of the art in developing high fidelity virtual patients that will enable the caregiver to improve interpersonal skills for scenarios that require not only medical experience, but the ability to relate at an interpersonal level, with interviewing and diagnosis skills as patients can be hiding symptoms of post-traumatic stress disorder, suicide and domestic violence.
Rosenbloom, Paul
Implementing First-Order Variables in a Graphical Cognitive Architecture Proceedings Article
In: Proceedings of the First International Conference on Biologically Inspired Cognitive Architectures, IOS Press, Washington, DC, 2010.
Abstract | Links | BibTeX | Tags: CogArch, Cognitive Architecture, Virtual Humans
@inproceedings{rosenbloom_implementing_2010,
title = {Implementing First-Order Variables in a Graphical Cognitive Architecture},
author = {Paul Rosenbloom},
url = {http://ict.usc.edu/pubs/Implementing%20First-Order%20Variables%20in%20a%20Graphical%20Cognitive%20Architecture.pdf},
year = {2010},
date = {2010-11-01},
booktitle = {Proceedings of the First International Conference on Biologically Inspired Cognitive Architectures},
publisher = {IOS Press},
address = {Washington, DC},
abstract = {Graphical cognitive architectures implement their functionality through localized message passing among computationally limited nodes. First-order variables – particularly universally quantified ones – while critical for some potential architectural mechanisms, can be quite difficult to implement in such architectures. A new implementation strategy based on message decomposition in graphical models is presented that yields tractability while preserving key symmetries in the graphs concerning how quantified variables are represented and how symbols, probabilities and signals are processed.},
keywords = {CogArch, Cognitive Architecture, Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
Graphical cognitive architectures implement their functionality through localized message passing among computationally limited nodes. First-order variables – particularly universally quantified ones – while critical for some potential architectural mechanisms, can be quite difficult to implement in such architectures. A new implementation strategy based on message decomposition in graphical models is presented that yields tractability while preserving key symmetries in the graphs concerning how quantified variables are represented and how symbols, probabilities and signals are processed.
Morency, Louis-Philippe
Modeling Human Communication Dynamics Journal Article
In: IEEE Signal Processing Magazine, no. September 2010, pp. 112–116, 2010.
Abstract | Links | BibTeX | Tags: Virtual Humans
@article{morency_modeling_2010,
title = {Modeling Human Communication Dynamics},
author = {Louis-Philippe Morency},
url = {http://ict.usc.edu/pubs/Modeling%20Human%20Communication%20Dynamics.pdf},
year = {2010},
date = {2010-09-01},
journal = {IEEE Signal Processing Magazine},
number = {September 2010},
pages = {112–116},
abstract = {Face-to-face communication is a highly interactive process where participants mutually exchange and interpret verbal and nonverbal messages. Communication dynamics represent the temporal relationship between these communicative messages. Even when only one person speaks at a time, other participants exchange information continuously among themselves and with the speaker through gesture, gaze, posture, and facial expressions. The transactional view of human communication shows an important dynamic between communicative behaviors where each person serves simultaneously as speaker and listener. At the same time you send a message, you also receive messages from your own communications (individual dynamics) as well as from the reactions of the other person(s) (interpersonal dynamics).},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {article}
}
Face-to-face communication is a highly interactive process where participants mutually exchange and interpret verbal and nonverbal messages. Communication dynamics represent the temporal relationship between these communicative messages. Even when only one person speaks at a time, other participants exchange information continuously among themselves and with the speaker through gesture, gaze, posture, and facial expressions. The transactional view of human communication shows an important dynamic between communicative behaviors where each person serves simultaneously as speaker and listener. At the same time you send a message, you also receive messages from your own communications (individual dynamics) as well as from the reactions of the other person(s) (interpersonal dynamics).
Gratch, Jonathan
In: Doveling, Katrin; Scheve, Christian; Konijn, Elly A. (Ed.): The Routledge Handbook of Emotions and Mass Media, 2010, ISBN: 978-0-415-48160-1.
Links | BibTeX | Tags: Virtual Humans
@incollection{gratch_emotionally_2010,
title = {Emotionally resonant media: Advances in sensing, understanding and influencing human emotion through interactive media},
author = {Jonathan Gratch},
editor = {Katrin Doveling and Christian Scheve and Elly A. Konijn},
url = {http://ict.usc.edu/pubs/Emotionally%20resonant%20media.pdf},
isbn = {978-0-415-48160-1},
year = {2010},
date = {2010-09-01},
booktitle = {The Routledge Handbook of Emotions and Mass Media},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {incollection}
}
Morency, Louis-Philippe
The Role of Context in Affective Behavior Understanding Book Section
In: Social Emotions in nature and Artifact: Emotions in Human and Human-Computer Interaction, 2010.
Abstract | Links | BibTeX | Tags: Virtual Humans
@incollection{morency_role_2010,
title = {The Role of Context in Affective Behavior Understanding},
author = {Louis-Philippe Morency},
url = {http://ict.usc.edu/pubs/Role_of_Context_in_Afective_Behavior.pdf},
year = {2010},
date = {2010-09-01},
booktitle = {Social Emotions in nature and Artifact: Emotions in Human and Human-Computer Interaction},
abstract = {Face-to-face communication is highly interactive. Even when only one person speaks at the time, other participants exchange information continuously amongst themselves and with the speaker through gesture, gaze, posture and facial expressions. Such affective feedback is an essential and predictable aspect of natural conversation and its absence can significantly disrupt participants ability to communicate. During multiparty interactions such as in meetings, information is exchanged between participants using both audio and visual channels. Visual feedback can range from a simple eye glance to a large arm gesture or posture change. One important visual cue is head nod during conversation. Head nods are used for displaying agreement, grounding information or during turn-taking. Recognizing these affective gestures is important for understanding all the information exchanged during a meeting or conversation, and can be particularly crucial for identifying more subtle factors such as the effectiveness of communication, points of confusion, status relationships between participants, or the diagnosis social disorders. This chapter argues that it is possible to significantly improve state-of-the art recognition techniques by exploiting regularities in how people communicate. People do not provide affective feedback at random. Rather they react to the current topic, previous utterances and the speaker's current verbal and nonverbal behavior. For example, listeners are far more likely to nod or shake if the speaker has just asked them a question, and incorporating such dialogue context can improve recognition performance during human-robot interaction. More generally, speakers and listeners co-produce a range of lexical, prosodic, and nonverbal patterns. Our goal is to automatically discover these patterns using only easily observable features of human face-to-face interaction (e.g. prosodic features and eye gaze), and exploit them to improve recognition accuracy.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {incollection}
}
Face-to-face communication is highly interactive. Even when only one person speaks at the time, other participants exchange information continuously amongst themselves and with the speaker through gesture, gaze, posture and facial expressions. Such affective feedback is an essential and predictable aspect of natural conversation and its absence can significantly disrupt participants ability to communicate. During multiparty interactions such as in meetings, information is exchanged between participants using both audio and visual channels. Visual feedback can range from a simple eye glance to a large arm gesture or posture change. One important visual cue is head nod during conversation. Head nods are used for displaying agreement, grounding information or during turn-taking. Recognizing these affective gestures is important for understanding all the information exchanged during a meeting or conversation, and can be particularly crucial for identifying more subtle factors such as the effectiveness of communication, points of confusion, status relationships between participants, or the diagnosis social disorders. This chapter argues that it is possible to significantly improve state-of-the art recognition techniques by exploiting regularities in how people communicate. People do not provide affective feedback at random. Rather they react to the current topic, previous utterances and the speaker's current verbal and nonverbal behavior. For example, listeners are far more likely to nod or shake if the speaker has just asked them a question, and incorporating such dialogue context can improve recognition performance during human-robot interaction. More generally, speakers and listeners co-produce a range of lexical, prosodic, and nonverbal patterns. Our goal is to automatically discover these patterns using only easily observable features of human face-to-face interaction (e.g. prosodic features and eye gaze), and exploit them to improve recognition accuracy.
Swartout, William; Traum, David; Artstein, Ron; Noren, Dan; Debevec, Paul; Bronnenkant, Kerry; Williams, Josh; Leuski, Anton; Narayanan, Shrikanth; Piepol, Diane; Lane, H. Chad; Morie, Jacquelyn; Aggarwal, Priti; Liewer, Matt; Chiang, Jen-Yuan; Gerten, Jillian; Chu, Selina; White, Kyle
Ada and Grace: Toward Realistic and Engaging Virtual Museum Guides Proceedings Article
In: Proceedings of the 10th International Conference on Intelligent Virtual Agents (IVA 2010), Philadelphia, PA, 2010.
Abstract | Links | BibTeX | Tags: Graphics, Learning Sciences, Virtual Humans, Virtual Worlds
@inproceedings{swartout_ada_2010,
title = {Ada and Grace: Toward Realistic and Engaging Virtual Museum Guides},
author = {William Swartout and David Traum and Ron Artstein and Dan Noren and Paul Debevec and Kerry Bronnenkant and Josh Williams and Anton Leuski and Shrikanth Narayanan and Diane Piepol and H. Chad Lane and Jacquelyn Morie and Priti Aggarwal and Matt Liewer and Jen-Yuan Chiang and Jillian Gerten and Selina Chu and Kyle White},
url = {http://ict.usc.edu/pubs/ada%20and%20grace.pdf},
year = {2010},
date = {2010-09-01},
booktitle = {Proceedings of the 10th International Conference on Intelligent Virtual Agents (IVA 2010)},
address = {Philadelphia, PA},
abstract = {To increase the interest and engagement of middle school students in science and technology, the InterFaces project has created virtual museum guides that are in use at the Museum of Science, Boston. The characters use natural language interaction and have near photoreal appearance to increase and presents reports from museum staff on visitor reaction},
keywords = {Graphics, Learning Sciences, Virtual Humans, Virtual Worlds},
pubstate = {published},
tppubtype = {inproceedings}
}
To increase the interest and engagement of middle school students in science and technology, the InterFaces project has created virtual museum guides that are in use at the Museum of Science, Boston. The characters use natural language interaction and have near photoreal appearance to increase and presents reports from museum staff on visitor reaction
Lange, Belinda; Buckwalter, John Galen; Forbell, Eric; Kim, Julia; Sagae, Kenji; Williams, Josh; Difede, JoAnn; Rothbaum, Barbara O.; Reger, Greg; Parsons, Thomas D.; Kenny, Patrick G.
SimCoach: An Intelligent Virtual Human System for Providing Healthcare Information and Support Proceedings Article
In: Proceedings of the 8th International Conference on Disability, Virtual Reality and Associated Technology (ICDVRAT), Valparaiso, Chile, 2010.
Abstract | Links | BibTeX | Tags: MedVR, Virtual Humans
@inproceedings{lange_simcoach_2010,
title = {SimCoach: An Intelligent Virtual Human System for Providing Healthcare Information and Support},
author = {Belinda Lange and John Galen Buckwalter and Eric Forbell and Julia Kim and Kenji Sagae and Josh Williams and JoAnn Difede and Barbara O. Rothbaum and Greg Reger and Thomas D. Parsons and Patrick G. Kenny},
url = {http://ict.usc.edu/pubs/SimCoach-%20an%20intelligent%20virtual%20human%20system%20for%20providing%20healthcare%20information%20and%20support.pdf},
year = {2010},
date = {2010-09-01},
booktitle = {Proceedings of the 8th International Conference on Disability, Virtual Reality and Associated Technology (ICDVRAT)},
address = {Valparaiso, Chile},
abstract = {Over the last 15 years, a virtual revolution has taken place in the use of Virtual Reality simulation technology for clinical purposes. Recent shifts in the social and scientific landscape have now set the stage for the next major movement in Clinical Virtual Reality with the "birth" of intelligent virtual humans. This paper will present an overview of the SimCoach project that aims to develop virtual human support agents to serve as online guides for promoting access to psychological healthcare information and for assisting military personnel and family members in breaking down barriers to initiating care. While we believe that the use of virtual humans to serve the role of virtual therapists is still fraught with both technical and ethical concerns, the SimCoach project does not aim to become a "doc in box". Rather, the SimCoach experience is being designed to attract and engage military Service Members, Veterans and their significant others who might not otherwise seek help with a live healthcare provider. It is expected that this experience will motivate users to take the first step – to empower themselves to seek advice and information regarding their healthcare (e.g., psychological health, traumatic brain injury, addiction, etc.) and general personal welfare (i.e., other non-medical stressors such as economic or relationship issues) – and encourage them to take the next step towards seeking other, more formal resources if needed.},
keywords = {MedVR, Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
Over the last 15 years, a virtual revolution has taken place in the use of Virtual Reality simulation technology for clinical purposes. Recent shifts in the social and scientific landscape have now set the stage for the next major movement in Clinical Virtual Reality with the "birth" of intelligent virtual humans. This paper will present an overview of the SimCoach project that aims to develop virtual human support agents to serve as online guides for promoting access to psychological healthcare information and for assisting military personnel and family members in breaking down barriers to initiating care. While we believe that the use of virtual humans to serve the role of virtual therapists is still fraught with both technical and ethical concerns, the SimCoach project does not aim to become a "doc in box". Rather, the SimCoach experience is being designed to attract and engage military Service Members, Veterans and their significant others who might not otherwise seek help with a live healthcare provider. It is expected that this experience will motivate users to take the first step – to empower themselves to seek advice and information regarding their healthcare (e.g., psychological health, traumatic brain injury, addiction, etc.) and general personal welfare (i.e., other non-medical stressors such as economic or relationship issues) – and encourage them to take the next step towards seeking other, more formal resources if needed.
Kang, Sin-Hwa; Sidner, Candy; Gratch, Jonathan; Morency, Louis-Philippe
Turn-taking patterns in self-disclosure interactions with Virtual Agents Proceedings Article
In: Proceedings of the Conference on Intelligent Virtual Agents (IVA), Philadelphia, PA, 2010.
Links | BibTeX | Tags: Virtual Humans
@inproceedings{kang_turn-taking_2010,
title = {Turn-taking patterns in self-disclosure interactions with Virtual Agents},
author = {Sin-Hwa Kang and Candy Sidner and Jonathan Gratch and Louis-Philippe Morency},
url = {http://ict.usc.edu/pubs/Turn-taking%20patterns%20in%20self-disclosure%20interactions%20with%20Virtual%20Agents.pdf},
year = {2010},
date = {2010-09-01},
booktitle = {Proceedings of the Conference on Intelligent Virtual Agents (IVA)},
address = {Philadelphia, PA},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
Georgila, Kallirroi; Wang, Ning; Gratch, Jonathan
Cross-Domain Speech Disfluency Detection Proceedings Article
In: 11th SIGdial Workshop on Discourse and Dialogue, Tokyo, Japan, 2010.
Abstract | Links | BibTeX | Tags: Virtual Humans
@inproceedings{georgila_cross-domain_2010,
title = {Cross-Domain Speech Disfluency Detection},
author = {Kallirroi Georgila and Ning Wang and Jonathan Gratch},
url = {http://ict.usc.edu/pubs/Cross-Domain%20Speech%20Disfluency%20Detection.pdf},
year = {2010},
date = {2010-09-01},
booktitle = {11th SIGdial Workshop on Discourse and Dialogue},
address = {Tokyo, Japan},
abstract = {We build a model for speech disfluency detection based on conditional random fields (CRFs) using the Switchboard cor- pus. This model is then applied to a new domain without any adaptation. We show that a technique for detecting speech disfluencies based on Integer Linear Pro- gramming (ILP) (Georgila, 2009) signifi- cantly outperforms CRFs. In particular, in terms of F-score and NIST Error Rate the absolute improvement of ILP over CRFs exceeds 20% and 25% respectively. We conclude that ILP is an approach with great potential for speech disfluency detec- tion when there is a lack or shortage of in- domain data for training.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
We build a model for speech disfluency detection based on conditional random fields (CRFs) using the Switchboard cor- pus. This model is then applied to a new domain without any adaptation. We show that a technique for detecting speech disfluencies based on Integer Linear Pro- gramming (ILP) (Georgila, 2009) signifi- cantly outperforms CRFs. In particular, in terms of F-score and NIST Error Rate the absolute improvement of ILP over CRFs exceeds 20% and 25% respectively. We conclude that ILP is an approach with great potential for speech disfluency detec- tion when there is a lack or shortage of in- domain data for training.
Brusk, Jenny; Artstein, Ron; Traum, David
Don't tell anyone! Two Experiments on Gossip Conversations Proceedings Article
In: 11th SIGdial Workshop on Discourse and Dialogue, pp. 193–200, Tokyo, Japan, 2010.
Abstract | Links | BibTeX | Tags: Virtual Humans
@inproceedings{brusk_dont_2010,
title = {Don't tell anyone! Two Experiments on Gossip Conversations},
author = {Jenny Brusk and Ron Artstein and David Traum},
url = {http://ict.usc.edu/pubs/Don%E2%80%99t%20tell%20anyone%20Two%20Experiments%20on%20Gossip%20Conversations.pdf},
year = {2010},
date = {2010-09-01},
booktitle = {11th SIGdial Workshop on Discourse and Dialogue},
pages = {193–200},
address = {Tokyo, Japan},
abstract = {The purpose of this study is to get a working definition that matches people's intuitive notion of gossip and is sufficiently precise for computational implementation. We conducted two experiments investigating what type of conversations people intuitively understand and interpret as gossip, and whether they could identify three proposed constituents of gossip conversations: third person focus, pejorative evaluation and substantiating behavior. The results show that (1) conversations are very likely to be considered gossip if all elements are present, no intimate relationships exist between the participants, and the person in focus is unambiguous. (2) Conversations that have at most one gossip element are not considered gossip. (3) Conversations that lack one or two elements or have an ambiguous element lead to inconsistent judgments.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
The purpose of this study is to get a working definition that matches people's intuitive notion of gossip and is sufficiently precise for computational implementation. We conducted two experiments investigating what type of conversations people intuitively understand and interpret as gossip, and whether they could identify three proposed constituents of gossip conversations: third person focus, pejorative evaluation and substantiating behavior. The results show that (1) conversations are very likely to be considered gossip if all elements are present, no intimate relationships exist between the participants, and the person in focus is unambiguous. (2) Conversations that have at most one gossip element are not considered gossip. (3) Conversations that lack one or two elements or have an ambiguous element lead to inconsistent judgments.
Huang, Lixing; Morency, Louis-Philippe; Gratch, Jonathan
Learning Backchannel Prediction Model from Parasocial Consensus Sampling: A Subjective Evaluation Proceedings Article
In: The 10th International Conference on Intelligent Virtual Agents (IVA 2010), Philadelphia, PA, 2010.
Abstract | Links | BibTeX | Tags: Virtual Humans
@inproceedings{huang_learning_2010,
title = {Learning Backchannel Prediction Model from Parasocial Consensus Sampling: A Subjective Evaluation},
author = {Lixing Huang and Louis-Philippe Morency and Jonathan Gratch},
url = {http://ict.usc.edu/pubs/Learning%20Backchannel%20Prediction%20Model%20from%20Parasocial%20Consensus%20Sampling-%20A%20Subjective%20Evaluation.pdf},
year = {2010},
date = {2010-09-01},
booktitle = {The 10th International Conference on Intelligent Virtual Agents (IVA 2010)},
address = {Philadelphia, PA},
abstract = {Backchannel feedback is an important kind of nonverbal feedback within face-to-face interaction that signals a person's interest, attention and willingness to keep listening. Learning to predict when to give such feedback is one of the keys to creating natural and realistic virtual humans. Prediction models are traditionally learned from large corpora of annotated face-to-face interactions, but this approach has several limitations. Previously, we proposed a novel data collection method, Parasocial Consensus Sampling, which addresses these limitations. In this paper, we show that data collected in this manner can produce effective learned models. A subjective evaluation shows that the virtual human driven by the resulting probabilistic model significantly outperforms a previously published rule-based agent in terms of rapport, perceived accuracy and naturalness, and it is even better than the virtual human driven by real listeners' behavior in some cases.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
Backchannel feedback is an important kind of nonverbal feedback within face-to-face interaction that signals a person's interest, attention and willingness to keep listening. Learning to predict when to give such feedback is one of the keys to creating natural and realistic virtual humans. Prediction models are traditionally learned from large corpora of annotated face-to-face interactions, but this approach has several limitations. Previously, we proposed a novel data collection method, Parasocial Consensus Sampling, which addresses these limitations. In this paper, we show that data collected in this manner can produce effective learned models. A subjective evaluation shows that the virtual human driven by the resulting probabilistic model significantly outperforms a previously published rule-based agent in terms of rapport, perceived accuracy and naturalness, and it is even better than the virtual human driven by real listeners' behavior in some cases.
Melo, Celso M.; Carnevale, Peter; Gratch, Jonathan
The Influence of Emotions in Embodied Agents on Human Decision-Making Proceedings Article
In: Intelligent Virtual Agents, pp. 357–370, Philadelphia, PA, 2010.
Abstract | Links | BibTeX | Tags: Virtual Humans
@inproceedings{de_melo_influence_2010-1,
title = {The Influence of Emotions in Embodied Agents on Human Decision-Making},
author = {Celso M. Melo and Peter Carnevale and Jonathan Gratch},
url = {http://www.ict.usc.edu/pubs/The%20Influence%20of%20Emotions%20in%20Embodied%20Agents.pdf},
year = {2010},
date = {2010-09-01},
booktitle = {Intelligent Virtual Agents},
pages = {357–370},
address = {Philadelphia, PA},
abstract = {Acknowledging the social functions that emotions serve, there has been growing interest in the interpersonal effect of emotion in human decision making. Following the paradigm of experimental games from social psychology and experimental economics, we explore the interpersonal effect of emotions expressed by embodied agents on human decision making. The paper describes an experiment where participants play the iterated prisoner's dilemma against two different agents that play the same strategy (tit-for-tat), but communicate different goal orientations (cooperative vs. individualistic) through their patterns of facial displays. The results show that participants are sensitive to differences in the facial displays and cooperate significantly more with the cooperative agent. The data indicate that emotions in agents can influence human decision making and that the nature of the emotion, as opposed to mere presence, is crucial for these effects. We discuss the implications of the results for designing human-computer interfaces and understanding human-human interaction.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
Acknowledging the social functions that emotions serve, there has been growing interest in the interpersonal effect of emotion in human decision making. Following the paradigm of experimental games from social psychology and experimental economics, we explore the interpersonal effect of emotions expressed by embodied agents on human decision making. The paper describes an experiment where participants play the iterated prisoner's dilemma against two different agents that play the same strategy (tit-for-tat), but communicate different goal orientations (cooperative vs. individualistic) through their patterns of facial displays. The results show that participants are sensitive to differences in the facial displays and cooperate significantly more with the cooperative agent. The data indicate that emotions in agents can influence human decision making and that the nature of the emotion, as opposed to mere presence, is crucial for these effects. We discuss the implications of the results for designing human-computer interfaces and understanding human-human interaction.
Gandhe, Sudeep; Traum, David
I've said it before, and I'll say it again: An empirical investigation of the upper bound of the selection approach to dialogue Proceedings Article
In: 11th SIGdial Workshop on Discourse and Dialogue, pp. 245–248, Tokyo, Japan, 2010.
Abstract | Links | BibTeX | Tags: Virtual Humans
@inproceedings{gandhe_ive_2010,
title = {I've said it before, and I'll say it again: An empirical investigation of the upper bound of the selection approach to dialogue},
author = {Sudeep Gandhe and David Traum},
url = {http://ict.usc.edu//pubs/I%E2%80%99ve%20said%20it%20before,%20and%20I%E2%80%99ll%20say%20it%20again.pdf},
year = {2010},
date = {2010-09-01},
booktitle = {11th SIGdial Workshop on Discourse and Dialogue},
pages = {245–248},
address = {Tokyo, Japan},
abstract = {We perform a study of existing dialogue corpora to establish the theoretical max- imum performance of the selection ap- proach to simulating human dialogue be- havior in unseen dialogues. This maxi- mum is the proportion of test utterances for which an exact or approximate match exists in the corresponding training cor- pus. The results indicate that some do- mains seem quite suitable for a corpus- based selection approach, with over half of the test utterances having been seen before in the corpus, while other domains show much more novelty compared to previous dialogues.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
We perform a study of existing dialogue corpora to establish the theoretical max- imum performance of the selection ap- proach to simulating human dialogue be- havior in unseen dialogues. This maxi- mum is the proportion of test utterances for which an exact or approximate match exists in the corresponding training cor- pus. The results indicate that some do- mains seem quite suitable for a corpus- based selection approach, with over half of the test utterances having been seen before in the corpus, while other domains show much more novelty compared to previous dialogues.
Rosenbloom, Paul
Combining Procedural and Declarative Knowledge in a Graphical Architecture Proceedings Article
In: Proceedings of the 10th International Conference on Cognitive Modeling, Philadelphia, PA, 2010.
Abstract | Links | BibTeX | Tags: CogArch, Cognitive Architecture, Virtual Humans, Virtual Worlds
@inproceedings{rosenbloom_combining_2010,
title = {Combining Procedural and Declarative Knowledge in a Graphical Architecture},
author = {Paul Rosenbloom},
url = {http://ict.usc.edu/pubs/Combining%20Procedural%20and%20Declarative%20Knowledge%20in%20a%20Graphical%20Architecture.pdf},
year = {2010},
date = {2010-08-01},
booktitle = {Proceedings of the 10th International Conference on Cognitive Modeling},
address = {Philadelphia, PA},
abstract = {A prototypical cognitive architecture defines a memory architecture embodying forms of both procedural and declarative memory, plus their interaction. Reengineering such a dual architecture on a common foundation of graphical models enables a better understanding of both the substantial commonalities between procedural and declarative memory and the subtle differences that endow each with its own special character. It also opens the way towards blended capabilities that go beyond existing architectural memories.},
keywords = {CogArch, Cognitive Architecture, Virtual Humans, Virtual Worlds},
pubstate = {published},
tppubtype = {inproceedings}
}
A prototypical cognitive architecture defines a memory architecture embodying forms of both procedural and declarative memory, plus their interaction. Reengineering such a dual architecture on a common foundation of graphical models enables a better understanding of both the substantial commonalities between procedural and declarative memory and the subtle differences that endow each with its own special character. It also opens the way towards blended capabilities that go beyond existing architectural memories.
Leuski, Anton; Traum, David
Practical Language Processing for Virtual Humans Proceedings Article
In: Twenty-Second Annual Conference on Innovative Applications of Artificial Intelligence (IAAI-10), Atlanta, GA, 2010.
Abstract | Links | BibTeX | Tags: Virtual Humans
@inproceedings{leuski_practical_2010,
title = {Practical Language Processing for Virtual Humans},
author = {Anton Leuski and David Traum},
url = {http://ict.usc.edu/pubs/Practical%20Language%20Processing%20for%20Virtual%20Humans.pdf},
year = {2010},
date = {2010-07-01},
booktitle = {Twenty-Second Annual Conference on Innovative Applications of Artificial Intelligence (IAAI-10)},
address = {Atlanta, GA},
abstract = {NPCEditor is a system for building a natural language processing component for virtual humans capable of engaging a user in spoken dialog on a limited domain. It uses a statistical language classification technology for mapping from user's text input to system responses. NPCEditor provides a user-friendly editor for creating effective virtual humans quickly. It has been deployed as a part of various virtual human systems in several applications.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
NPCEditor is a system for building a natural language processing component for virtual humans capable of engaging a user in spoken dialog on a limited domain. It uses a statistical language classification technology for mapping from user's text input to system responses. NPCEditor provides a user-friendly editor for creating effective virtual humans quickly. It has been deployed as a part of various virtual human systems in several applications.
Rosenbloom, Paul S.
Speculations on Leveraging Graphical Models for Architectural Integration of Visual Representation and Reasoning Proceedings Article
In: Proceedings of the AAAI-10 Workshop on Visual Representations and Reasoning, Atlanta, GA, 2010.
Abstract | Links | BibTeX | Tags: CogArch, Cognitive Architecture, Virtual Humans
@inproceedings{rosenbloom_speculations_2010,
title = {Speculations on Leveraging Graphical Models for Architectural Integration of Visual Representation and Reasoning},
author = {Paul S. Rosenbloom},
url = {http://ict.usc.edu/pubs/Speculations%20on%20Leveraging%20Graphical%20Models%20for%20Architectural%20Integration%20of%20Visual%20Representation%20and%20Reasoning.pdf},
year = {2010},
date = {2010-07-01},
booktitle = {Proceedings of the AAAI-10 Workshop on Visual Representations and Reasoning},
address = {Atlanta, GA},
abstract = {Speculations on Leveraging Graphical Models for Architectural Integration of Visual Representation and Reasoning},
keywords = {CogArch, Cognitive Architecture, Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
Speculations on Leveraging Graphical Models for Architectural Integration of Visual Representation and Reasoning
Gerardi, Maryrose; Cukor, Judith; Difede, JoAnn; Rizzo, Albert; Rothbaum, Barbara O.
Virtual Reality Exposure Therapy for Post-Traumatic Stress Disorder and Other Anxiety Disorders Journal Article
In: Current Psychiatry Reports, vol. 12, no. 4, pp. 298–305, 2010.
Abstract | Links | BibTeX | Tags: MedVR, Virtual Humans
@article{gerardi_virtual_2010,
title = {Virtual Reality Exposure Therapy for Post-Traumatic Stress Disorder and Other Anxiety Disorders},
author = {Maryrose Gerardi and Judith Cukor and JoAnn Difede and Albert Rizzo and Barbara O. Rothbaum},
url = {http://ict.usc.edu/pubs/Virtual%20Reality%20Exposure%20Therapy%20for%20Post-Traumatic%20Stress%20Disorder%20and%20Other%20Anxiety%20Disorders.pdf},
year = {2010},
date = {2010-06-01},
journal = {Current Psychiatry Reports},
volume = {12},
number = {4},
pages = {298–305},
abstract = {Anxiety disorders, including phobias and post- traumatic stress disorder, are common and disabling disorders that often involve avoidance behavior. Cognitive-behavioral treatments, specifically imaginal and in vivo forms of exposure therapy, have been accepted and successful forms of treatment for these disorders. Virtual reality exposure therapy, an alternative to more traditional exposure-based therapies, involves immersion in a computer-generated virtual environment that minimizes avoidance and facilitates emo- tional processing. In this article, we review evidence on the application of virtual reality exposure therapy to the treatment of specific phobias and post-traumatic stress disorder and discuss its advantages and cautions.},
keywords = {MedVR, Virtual Humans},
pubstate = {published},
tppubtype = {article}
}
Anxiety disorders, including phobias and post- traumatic stress disorder, are common and disabling disorders that often involve avoidance behavior. Cognitive-behavioral treatments, specifically imaginal and in vivo forms of exposure therapy, have been accepted and successful forms of treatment for these disorders. Virtual reality exposure therapy, an alternative to more traditional exposure-based therapies, involves immersion in a computer-generated virtual environment that minimizes avoidance and facilitates emo- tional processing. In this article, we review evidence on the application of virtual reality exposure therapy to the treatment of specific phobias and post-traumatic stress disorder and discuss its advantages and cautions.
Sagae, Kenji; DeVault, David; Traum, David
Interpretation of Partial Utterances in Virtual Human Dialogue Systems Proceedings Article
In: 11th Annual Conference of the North American Chapter of the Association for Computational Linguistics, Los Angeles, CA, 2010.
Abstract | Links | BibTeX | Tags: Virtual Humans
@inproceedings{sagae_interpretation_2010,
title = {Interpretation of Partial Utterances in Virtual Human Dialogue Systems},
author = {Kenji Sagae and David DeVault and David Traum},
url = {http://ict.usc.edu/pubs/Interpretation%20of%20Partial%20Utterances%20in%20Virtual%20Human%20Dialogue%20Systems.pdf},
year = {2010},
date = {2010-06-01},
booktitle = {11th Annual Conference of the North American Chapter of the Association for Computational Linguistics},
address = {Los Angeles, CA},
abstract = {Dialogue systems typically follow a rigid pace of interaction where the system waits until the user has finished speaking before producing a response. Interpreting user utterances be- fore they are completed allows a system to display more sophisticated conversational be- havior, such as rapid turn-taking and appropri- ate use of backchannels and interruptions. We demonstrate a natural language understanding approach for partial utterances, and its use in a virtual human dialogue system that can often complete a user's utterances in real time.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
Dialogue systems typically follow a rigid pace of interaction where the system waits until the user has finished speaking before producing a response. Interpreting user utterances be- fore they are completed allows a system to display more sophisticated conversational be- havior, such as rapid turn-taking and appropri- ate use of backchannels and interruptions. We demonstrate a natural language understanding approach for partial utterances, and its use in a virtual human dialogue system that can often complete a user's utterances in real time.
Gerber, Matt; Gordon, Andrew S.; Sagae, Kenji
Open-domain Commonsense Reasoning Using Discourse Relations from a Corpus of Weblog Stories. Proceedings Article
In: Proceedings of the 1st International Workshop on Formalisms and Methodology for Learning by Reading (FAM-LbR) NAACL 2010 Workshop, Los Angeles, CA, 2010.
Abstract | Links | BibTeX | Tags: The Narrative Group, Virtual Humans
@inproceedings{gerber_open-domain_2010,
title = {Open-domain Commonsense Reasoning Using Discourse Relations from a Corpus of Weblog Stories.},
author = {Matt Gerber and Andrew S. Gordon and Kenji Sagae},
url = {http://www.ict.usc.edu/pubs/Open-domain%20Commonsense%20Reasoning%20Using%20Discourse%20Relations%20from%20a%20Corpus%20of%20Weblog%20Stories.pdf},
year = {2010},
date = {2010-06-01},
booktitle = {Proceedings of the 1st International Workshop on Formalisms and Methodology for Learning by Reading (FAM-LbR) NAACL 2010 Workshop},
address = {Los Angeles, CA},
abstract = {We present a method of extracting open- domain commonsense knowledge by apply- ing discourse parsing to a large corpus of per- sonal stories written by Internet authors. We demonstrate the use of a linear-time, joint syn- tax/discourse dependency parser for this pur- pose, and we show how the extracted dis- course relations can be used to generate open- domain textual inferences. Our evaluations of the discourse parser and inference models show some success, but also identify a num- ber of interesting directions for future work.},
keywords = {The Narrative Group, Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
We present a method of extracting open- domain commonsense knowledge by apply- ing discourse parsing to a large corpus of per- sonal stories written by Internet authors. We demonstrate the use of a linear-time, joint syn- tax/discourse dependency parser for this pur- pose, and we show how the extracted dis- course relations can be used to generate open- domain textual inferences. Our evaluations of the discourse parser and inference models show some success, but also identify a num- ber of interesting directions for future work.
Wang, Ning; Johnson, W. Lewis; Gratch, Jonathan
Facial Expressions and Politeness Effect in Foreign Language Training System Proceedings Article
In: Lecture Notes in Computer Science, pp. 165–173, Pittsburgh, PA, 2010.
Abstract | Links | BibTeX | Tags: Virtual Humans
@inproceedings{wang_facial_2010,
title = {Facial Expressions and Politeness Effect in Foreign Language Training System},
author = {Ning Wang and W. Lewis Johnson and Jonathan Gratch},
url = {http://www.ict.usc.edu/pubs/Facial%20Expressions%20and%20Politeness%20Effect%20in%20Foreign%20Language%20Training%20System.pdf},
doi = {10.1007/978-3-642-13388-6_21},
year = {2010},
date = {2010-06-01},
booktitle = {Lecture Notes in Computer Science},
volume = {6094},
pages = {165–173},
address = {Pittsburgh, PA},
abstract = {Previous studies on the Politeness Effect show that using politeness strategies in tutorial feedback can have a positive impact on learning (McLaren et al. 2010; Wang and Johnson 2008; Wang et al. 2005). While prior research efforts tried to uncover the mechanism through which the politeness strategies impact the learner, the results were inconclusive. Further, it is unclear how the politeness strategies should adapt over time. In this paper, we analyze the video tapes of participants' facial expression while interacting with a polite or direct tutor in a foreign language training system. The Facial Action Coding System was then used to analyze the facial expressions. Results show that as social distance decreases over time, polite feedback is received less favorably while the preference for direct feedback increases.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
Previous studies on the Politeness Effect show that using politeness strategies in tutorial feedback can have a positive impact on learning (McLaren et al. 2010; Wang and Johnson 2008; Wang et al. 2005). While prior research efforts tried to uncover the mechanism through which the politeness strategies impact the learner, the results were inconclusive. Further, it is unclear how the politeness strategies should adapt over time. In this paper, we analyze the video tapes of participants' facial expression while interacting with a polite or direct tutor in a foreign language training system. The Facial Action Coding System was then used to analyze the facial expressions. Results show that as social distance decreases over time, polite feedback is received less favorably while the preference for direct feedback increases.
Bunt, Harry; Alexandersson, Jan; Carletta, Jean; Choe, Jae-Woong; Fang, Alex Chengyu; Hasida, Koiti; Lee, Kiyong; Petukhova, Volha; Popescu-Belis, Andrei; Romary, Laurent; Soria, Claudia; Traum, David
Towards an ISO standard for dialogue act annotation Proceedings Article
In: International Conference on Language Resources and Evaluation (LREC), Valletta, Malta, 2010.
Abstract | Links | BibTeX | Tags: Virtual Humans
@inproceedings{bunt_towards_2010,
title = {Towards an ISO standard for dialogue act annotation},
author = {Harry Bunt and Jan Alexandersson and Jean Carletta and Jae-Woong Choe and Alex Chengyu Fang and Koiti Hasida and Kiyong Lee and Volha Petukhova and Andrei Popescu-Belis and Laurent Romary and Claudia Soria and David Traum},
url = {http://ict.usc.edu/pubs/towards%20an%20iso%20standard.pdf},
year = {2010},
date = {2010-05-01},
booktitle = {International Conference on Language Resources and Evaluation (LREC)},
address = {Valletta, Malta},
abstract = {This paper describes an ISO project developing an international standard for annotating dialogue with semantic information, in particular concerning the communicative functions of the utterances, the kind of content they address, and the dependency relations to what was said and done earlier in the dialogue. The project, registered as ISO 24617-2 Semantic annotation framework, Part 2: Dialogue acts�, is currently at DIS stage.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
This paper describes an ISO project developing an international standard for annotating dialogue with semantic information, in particular concerning the communicative functions of the utterances, the kind of content they address, and the dependency relations to what was said and done earlier in the dialogue. The project, registered as ISO 24617-2 Semantic annotation framework, Part 2: Dialogue acts�, is currently at DIS stage.
Yao, Xuchen; Bhutada, Pravin; Georgila, Kallirroi; Sagae, Kenji; Artstein, Ron; Traum, David
Practical Evaluation of Speech Recognizers for Virtual Human Dialogue Systems Proceedings Article
In: Proceedings of the Seventh International Conference on Language Resources and Evaluation, Malta, 2010.
Abstract | Links | BibTeX | Tags: Virtual Humans
@inproceedings{yao_practical_2010,
title = {Practical Evaluation of Speech Recognizers for Virtual Human Dialogue Systems},
author = {Xuchen Yao and Pravin Bhutada and Kallirroi Georgila and Kenji Sagae and Ron Artstein and David Traum},
url = {http://ict.usc.edu/pubs/Practical%20Evaluation%20of%20Speech%20Recognizers%20for%20Virtual%20Human%20Dialogue%20Systems.pdf},
year = {2010},
date = {2010-05-01},
booktitle = {Proceedings of the Seventh International Conference on Language Resources and Evaluation},
address = {Malta},
abstract = {We perform a large-scale evaluation of multiple off-the-shelf speech recognizers across diverse domains for virtual human dialogue systems. Our evaluation is aimed at speech recognition consumers and potential consumers with limited experience with readily available recognizers. We focus on practical factors to determine what levels of performance can be expected from different available recognizers in various projects featuring different types of conversational utterances. Our results show that there is no single recognizer that outperforms all other recognizers in all domains. The performance of each recognizer may vary significantly depending on the domain, the size and perplexity of the corpus, the out-of-vocabulary rate, and whether acoustic and language model adaptation has been used or not. We expect that our evaluation will prove useful to other speech recognition consumers, especially in the dialogue community, and will shed some light on the key problem in spoken dialogue systems of selecting the most suitable available speech recognition system for a particular application, and what impact training will have.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
We perform a large-scale evaluation of multiple off-the-shelf speech recognizers across diverse domains for virtual human dialogue systems. Our evaluation is aimed at speech recognition consumers and potential consumers with limited experience with readily available recognizers. We focus on practical factors to determine what levels of performance can be expected from different available recognizers in various projects featuring different types of conversational utterances. Our results show that there is no single recognizer that outperforms all other recognizers in all domains. The performance of each recognizer may vary significantly depending on the domain, the size and perplexity of the corpus, the out-of-vocabulary rate, and whether acoustic and language model adaptation has been used or not. We expect that our evaluation will prove useful to other speech recognition consumers, especially in the dialogue community, and will shed some light on the key problem in spoken dialogue systems of selecting the most suitable available speech recognition system for a particular application, and what impact training will have.
Andersson, Sebastian; Georgila, Kallirroi; Traum, David; Aylett, Matthew; Clark, Robert A. J.
Prediction and Realisation of Conversational Characteristics by Utilising Spontaneous Speech for Unit Selection Proceedings Article
In: Speech Prosody, Chicago, IL, 2010.
Abstract | Links | BibTeX | Tags: Virtual Humans
@inproceedings{andersson_prediction_2010,
title = {Prediction and Realisation of Conversational Characteristics by Utilising Spontaneous Speech for Unit Selection},
author = {Sebastian Andersson and Kallirroi Georgila and David Traum and Matthew Aylett and Robert A. J. Clark},
url = {http://ict.usc.edu/pubs/Prediction%20and%20Realisation%20of%20Conversational%20Characteristics%20by%20Utilising%20Spontaneous%20Speech%20for%20Unit%20Selection.pdf},
year = {2010},
date = {2010-05-01},
booktitle = {Speech Prosody},
address = {Chicago, IL},
abstract = {Unit selection speech synthesis has reached high levels of naturalness and intelligibility for neutral read aloud speech. However, synthetic speech generated using neutral read aloud data lacks all the attitude, intention and spontaneity associated with everyday conversations. Unit selection is heavily data de- pendent and thus in order to simulate human conversational speech, or create synthetic voices for believable virtual char- acters, we need to utilise speech data with examples of how people talk rather than how people read. In this paper we in- cluded carefully selected utterances from spontaneous conver- sational speech in a unit selection voice. Using this voice and by automatically predicting type and placement of lexical fillers and filled pauses we can synthesise utterances with conversa- tional characteristics. A perceptual listening test showed that it is possible to make synthetic speech sound more conversational without degrading naturalness.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
Unit selection speech synthesis has reached high levels of naturalness and intelligibility for neutral read aloud speech. However, synthetic speech generated using neutral read aloud data lacks all the attitude, intention and spontaneity associated with everyday conversations. Unit selection is heavily data de- pendent and thus in order to simulate human conversational speech, or create synthetic voices for believable virtual char- acters, we need to utilise speech data with examples of how people talk rather than how people read. In this paper we in- cluded carefully selected utterances from spontaneous conver- sational speech in a unit selection voice. Using this voice and by automatically predicting type and placement of lexical fillers and filled pauses we can synthesise utterances with conversa- tional characteristics. A perceptual listening test showed that it is possible to make synthetic speech sound more conversational without degrading naturalness.
Robinson, Susan; Roque, Antonio; Traum, David
Dialogues in Context: An Objective User-Oriented Evaluation Approach for Virtual Human Dialogue Proceedings Article
In: 7th International Conference on Language Resources and Evaluation, Valletta, Malta, 2010.
Abstract | Links | BibTeX | Tags: Virtual Humans
@inproceedings{robinson_dialogues_2010,
title = {Dialogues in Context: An Objective User-Oriented Evaluation Approach for Virtual Human Dialogue},
author = {Susan Robinson and Antonio Roque and David Traum},
url = {http://ict.usc.edu/pubs/Dialogues%20in%20Context.pdf},
year = {2010},
date = {2010-05-01},
booktitle = {7th International Conference on Language Resources and Evaluation},
address = {Valletta, Malta},
abstract = {As conversational agents are now being developed to encounter more complex dialogue situations it is increasingly difficult to find satisfactory methods for evaluating these agents. Task-based measures are insufficient where there is no clearly defined task. While user-based evaluation methods may give a general sense of the quality of an agent's performance, they shed little light on the relative quality or success of specific features of dialogue that are necessary for system improvement. This paper examines current dialogue agent evaluation practices and motivates the need for a more detailed approach for defining and measuring the quality of dialogues between agent and user. We present a framework for evaluating the dialogue competence of artificial agnets involved in complex and underspecified tasks when conversing with people. A multi-part coding scheme is proposed that provides a qualitative analysis of human utterances, and rates the appropriateness of the agent's responses to these utterances. The scheme is outlined, and then used to evaluate Staff Duty Officer Moleno, a virtual guide in Second Life.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
As conversational agents are now being developed to encounter more complex dialogue situations it is increasingly difficult to find satisfactory methods for evaluating these agents. Task-based measures are insufficient where there is no clearly defined task. While user-based evaluation methods may give a general sense of the quality of an agent's performance, they shed little light on the relative quality or success of specific features of dialogue that are necessary for system improvement. This paper examines current dialogue agent evaluation practices and motivates the need for a more detailed approach for defining and measuring the quality of dialogues between agent and user. We present a framework for evaluating the dialogue competence of artificial agnets involved in complex and underspecified tasks when conversing with people. A multi-part coding scheme is proposed that provides a qualitative analysis of human utterances, and rates the appropriateness of the agent's responses to these utterances. The scheme is outlined, and then used to evaluate Staff Duty Officer Moleno, a virtual guide in Second Life.
Traum, David; Morency, Louis-Philippe
Integration of Visual Perception in Dialogue Understanding for Virtual Humans in Multi-Party interaction Proceedings Article
In: International Conference on Autonomous Agents and Multiagent Systems (AAMAS), Toronto, Ontario, 2010.
Abstract | Links | BibTeX | Tags: Virtual Humans
@inproceedings{traum_integration_2010,
title = {Integration of Visual Perception in Dialogue Understanding for Virtual Humans in Multi-Party interaction},
author = {David Traum and Louis-Philippe Morency},
url = {http://ict.usc.edu/pubs/Integration%20of%20Visual%20Perception%20in%20Dialogue%20Understanding%20for%20Virtual%20Humans%20in%20Multi-Party%20interaction.pdf},
year = {2010},
date = {2010-05-01},
booktitle = {International Conference on Autonomous Agents and Multiagent Systems (AAMAS)},
address = {Toronto, Ontario},
abstract = {While the dialogue functions of speech in two-party dialogue have been extensively studied, there has been much less work on either multi-party communication, multimodal communication, and espe- cially vision in a multi-party face-to-face setting. In this paper we report on one such effort to apply state of the art real-time visual processing to enhance a dialogue model of multi-party communi- cation. We are concerned with situations in which there are at least three parties involved in conversation (at least one of whom is a human participant and at least one of whom is a virtual human). We focus on the visual behaviors of head orientation, head nods and head shakes, and examine how these behaviors influence sev- eral aspects of a multi-layer dialogue model, including addressee identification, turn-taking, referent identification, social affiliation, grounding, and question answering. We describe the extensions to the dialogue model and the implemented techniques for recogniz- ing these behaviors and their impact on the dialogue models in real time, in realistic conversational settings from people participating in dialogue with virtual humans. We present several case studies (with accompanying videos) of dialogue fragments of the virtual agents with and without the recognition of these behaviors. Future work involves detailed studies on both the context recognition rates for this task as well as overall subjective impact on user satisfaction and dialogue efficiency.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
While the dialogue functions of speech in two-party dialogue have been extensively studied, there has been much less work on either multi-party communication, multimodal communication, and espe- cially vision in a multi-party face-to-face setting. In this paper we report on one such effort to apply state of the art real-time visual processing to enhance a dialogue model of multi-party communi- cation. We are concerned with situations in which there are at least three parties involved in conversation (at least one of whom is a human participant and at least one of whom is a virtual human). We focus on the visual behaviors of head orientation, head nods and head shakes, and examine how these behaviors influence sev- eral aspects of a multi-layer dialogue model, including addressee identification, turn-taking, referent identification, social affiliation, grounding, and question answering. We describe the extensions to the dialogue model and the implemented techniques for recogniz- ing these behaviors and their impact on the dialogue models in real time, in realistic conversational settings from people participating in dialogue with virtual humans. We present several case studies (with accompanying videos) of dialogue fragments of the virtual agents with and without the recognition of these behaviors. Future work involves detailed studies on both the context recognition rates for this task as well as overall subjective impact on user satisfaction and dialogue efficiency.
Endrass, Birgit; André, Elisabeth; Huang, Lixing; Gratch, Jonathan
A data-driven approach to model culture-specific communication management styles for virtual agents Proceedings Article
In: Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems, Toronto, Canada, 2010, ISBN: 978-0-9826571-1-9.
Abstract | Links | BibTeX | Tags: Virtual Humans
@inproceedings{endrass_data-driven_2010,
title = {A data-driven approach to model culture-specific communication management styles for virtual agents},
author = {Birgit Endrass and Elisabeth André and Lixing Huang and Jonathan Gratch},
url = {http://ict.usc.edu/pubs/A%20data-driven%20approach%20to%20model%20culture-specific%20communication%20management%20styles%20for%20virtual%20agents.pdf},
isbn = {978-0-9826571-1-9},
year = {2010},
date = {2010-05-01},
booktitle = {Proceedings of the 9th International Conference on Autonomous Agents and Multiagent Systems},
volume = {1},
address = {Toronto, Canada},
abstract = {Virtual agents are a great opportunity in teaching inter- cultural competencies. Advantages, such as the repeatabil- ity of training sessions, emotional distance to virtual charac- ters, the opportunity to over-exaggerate or generalize behav- ior or simply to save the costs for human training-partners support that idea. Especially the way communication is co- ordinated varies across cultures. In this paper, we present our approach of simulating differences in the management of communication for the American and Arabic cultures. Therefore, we give an overview of behavioral tendencies de- scribed in the literature, pointing out differences between the two cultures. Grounding our expectations in empiri- cal data we analyzed a multi-modal corpora. These findings were integrated into a demonstrator using virtual agents and evaluated in a preliminary study.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
Virtual agents are a great opportunity in teaching inter- cultural competencies. Advantages, such as the repeatabil- ity of training sessions, emotional distance to virtual charac- ters, the opportunity to over-exaggerate or generalize behav- ior or simply to save the costs for human training-partners support that idea. Especially the way communication is co- ordinated varies across cultures. In this paper, we present our approach of simulating differences in the management of communication for the American and Arabic cultures. Therefore, we give an overview of behavioral tendencies de- scribed in the literature, pointing out differences between the two cultures. Grounding our expectations in empiri- cal data we analyzed a multi-modal corpora. These findings were integrated into a demonstrator using virtual agents and evaluated in a preliminary study.
Melo, Celso M.; Kenny, Patrick G.; Gratch, Jonathan
Real-time expression of affect through respiration Journal Article
In: Computer Animtion and Virtual Worlds, vol. 21, pp. 225+234, 2010.
Abstract | Links | BibTeX | Tags: MedVR, Virtual Humans
@article{de_melo_real-time_2010,
title = {Real-time expression of affect through respiration},
author = {Celso M. Melo and Patrick G. Kenny and Jonathan Gratch},
url = {http://ict.usc.edu/pubs/Real-time%20expression%20of%20affect%20through%20respiration.pdf},
year = {2010},
date = {2010-05-01},
journal = {Computer Animtion and Virtual Worlds},
volume = {21},
pages = {225+234},
abstract = {Affect has been shown to influence respiration in people. This paper takes this insight and proposes a real-time model to express affect through respiration in virtual humans. Fourteen affective states are explored: excitement, relaxation, focus, pain, relief, boredom, anger, fear, panic, disgust, surprise, startle, sadness, and joy. Specific respiratory patterns are described from the literature for each of these affective states. Then, a real-time model of respiration is proposed that uses morphing to animate breathing and provides parameters to control respiration rate, respiration depth and the respiration cycle curve. These parameters are used to implement the respiratory patterns. Finally, a within-subjects study is described where subjects are asked to classify videos of the virtual human expressing each affective state with or without the specific respiratory patterns. The study was presented to 41 subjects and the results show that the model improved perception of excitement, pain, relief, boredom, anger, fear, panic, disgust, and startle.},
keywords = {MedVR, Virtual Humans},
pubstate = {published},
tppubtype = {article}
}
Affect has been shown to influence respiration in people. This paper takes this insight and proposes a real-time model to express affect through respiration in virtual humans. Fourteen affective states are explored: excitement, relaxation, focus, pain, relief, boredom, anger, fear, panic, disgust, surprise, startle, sadness, and joy. Specific respiratory patterns are described from the literature for each of these affective states. Then, a real-time model of respiration is proposed that uses morphing to animate breathing and provides parameters to control respiration rate, respiration depth and the respiration cycle curve. These parameters are used to implement the respiratory patterns. Finally, a within-subjects study is described where subjects are asked to classify videos of the virtual human expressing each affective state with or without the specific respiratory patterns. The study was presented to 41 subjects and the results show that the model improved perception of excitement, pain, relief, boredom, anger, fear, panic, disgust, and startle.
Herrera, David; Novick, David; Jan, Dusan; Traum, David
The UTEP-ICT Cross-Cultural Multiparty Multimodal Dialog Corpus Proceedings Article
In: In proceedings of the Multimodal Corpora Workshop: Advances in Capturing, Coding and Analyzing Multimodality (MMC 2010), Malta, 2010.
Abstract | Links | BibTeX | Tags: Virtual Humans
@inproceedings{herrera_utep-ict_2010,
title = {The UTEP-ICT Cross-Cultural Multiparty Multimodal Dialog Corpus},
author = {David Herrera and David Novick and Dusan Jan and David Traum},
url = {http://ict.usc.edu/pubs/The%20UTEP-ICT%20Cross-Cultural%20Multiparty%20Multimodal%20Dialog%20Corpus.pdf},
year = {2010},
date = {2010-05-01},
booktitle = {In proceedings of the Multimodal Corpora Workshop: Advances in Capturing, Coding and Analyzing Multimodality (MMC 2010)},
address = {Malta},
abstract = {To help answer questions about conversational control behaviors across cultures, a collaborative team from the University of Texas at El Paso and the Institute for Creative Technologies collected and partially coded approximately ten hours of audiovisual multiparty interactions in three different cultures and languages. Groups of four native speakers of Arabic, American English and Mexican Spanish completed five tasks and were recorded from six angles. Excerpts of four of the tasks were coded for proxemics, gaze, and turn-taking; interrater reliability had a Kappa score of about 0.8. Lessons learned from the multiparty corpus are being applied to the recording and annotation of a complementary dyadic corpus.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
To help answer questions about conversational control behaviors across cultures, a collaborative team from the University of Texas at El Paso and the Institute for Creative Technologies collected and partially coded approximately ten hours of audiovisual multiparty interactions in three different cultures and languages. Groups of four native speakers of Arabic, American English and Mexican Spanish completed five tasks and were recorded from six angles. Excerpts of four of the tasks were coded for proxemics, gaze, and turn-taking; interrater reliability had a Kappa score of about 0.8. Lessons learned from the multiparty corpus are being applied to the recording and annotation of a complementary dyadic corpus.
Kang, Sin-Hwa; Gratch, Jonathan
The Effect of Avatar Realism of Virtual Humans on Self-Disclosure in Anonymous Social Interactions Proceedings Article
In: Proceedings of the 28th ACM Conference on Human Factors in Computing Systems, Atlanta, GA, 2010.
Abstract | Links | BibTeX | Tags: Virtual Humans
@inproceedings{kang_effect_2010,
title = {The Effect of Avatar Realism of Virtual Humans on Self-Disclosure in Anonymous Social Interactions},
author = {Sin-Hwa Kang and Jonathan Gratch},
url = {http://ict.usc.edu/pubs/The%20Effect%20of%20Avatar%20Realism%20of%20Virtual%20Humans%20on%20Self-Disclosure%20in%20Anonymous%20Social%20Interactions.pdf},
year = {2010},
date = {2010-04-01},
booktitle = {Proceedings of the 28th ACM Conference on Human Factors in Computing Systems},
address = {Atlanta, GA},
abstract = {n this paper, we illustrate progress in research designed to investigate interactants' self-disclosure when they communicate with virtual humans or real humans in computer-mediated interactions. We explored the effect of the combination of avatar realism and interactants' anticipated future interaction (AFI) on self-disclosure in emotionally engaged and synchronous interaction. We primarily aimed at exploring ways to promote interactants' self-disclosure while securing their visual anonymity, even with minimal cues of virtual humans, when interactants anticipate future interaction. The research examined interactants' self-disclosure through measuring their verbal behaviors. The preliminary findings indicated that interactants revealed greater intimate information about themselves in interactions with virtual humans than with real humans. However, interactants' AFI did not affect their self-disclosure, which does not correspond to the results of previous studies using text based interfaces.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
n this paper, we illustrate progress in research designed to investigate interactants' self-disclosure when they communicate with virtual humans or real humans in computer-mediated interactions. We explored the effect of the combination of avatar realism and interactants' anticipated future interaction (AFI) on self-disclosure in emotionally engaged and synchronous interaction. We primarily aimed at exploring ways to promote interactants' self-disclosure while securing their visual anonymity, even with minimal cues of virtual humans, when interactants anticipate future interaction. The research examined interactants' self-disclosure through measuring their verbal behaviors. The preliminary findings indicated that interactants revealed greater intimate information about themselves in interactions with virtual humans than with real humans. However, interactants' AFI did not affect their self-disclosure, which does not correspond to the results of previous studies using text based interfaces.
Melo, Celso M.; Kenny, Patrick G.; Gratch, Jonathan
The Influence of Autonomic Signals on Perception of Emotions in Embodied Agents Journal Article
In: Applied Artificial Intelligence Journal, vol. Applied Artificial Intelligence Journal, no. Special Issue, 2010.
Abstract | Links | BibTeX | Tags: MedVR, Virtual Humans
@article{de_melo_influence_2010,
title = {The Influence of Autonomic Signals on Perception of Emotions in Embodied Agents},
author = {Celso M. Melo and Patrick G. Kenny and Jonathan Gratch},
url = {http://ict.usc.edu/pubs/THE%20INFLUENCE%20OF%20AUTONOMIC%20SIGNALS%20ON%20PERCEPTION%20OF%20EMOTIONS%20IN%20EMBODIED%20AGENTS.pdf},
year = {2010},
date = {2010-04-01},
journal = {Applied Artificial Intelligence Journal},
volume = {Applied Artificial Intelligence Journal},
number = {Special Issue},
abstract = {Specific patterns of autonomic activity have been reported when people experience emotions. Typical autonomic signals that change with emotion are wrinkles, blushing, sweating, tearing and respiration. This article explores whether these signals can also influence the perception of emotion in embodied agents. The article first reviews the literature on specific autonomic signal patterns associated with certain affective states. Next, it proceeds to describe a realtime model for wrinkles, blushing, sweating, tearing and respiration that is capable of implementing those patterns. Two studies are then described: in the first, subjects compare surprise, sadness, anger, shame, pride and fear expressed in an agent with or without blushing, wrinkles, sweating or tears; in the second, subjects compare excitement, relaxation, focus, pain, relief, boredom, anger, fear, panic, disgust, surprise, startle, sadness and joy expressed in an agent with or without typical respiration patterns. The first study shows a statistically significant positive effect on perception of surprise, sadness, anger, shame and fear. The second study shows a statistically significant positive effect on perception of excitement, pain, relief, boredom, anger, fear, panic, disgust and startle. The relevance of these results is discussed for the fields of artificial intelligence and intelligent virtual agents.},
keywords = {MedVR, Virtual Humans},
pubstate = {published},
tppubtype = {article}
}
Specific patterns of autonomic activity have been reported when people experience emotions. Typical autonomic signals that change with emotion are wrinkles, blushing, sweating, tearing and respiration. This article explores whether these signals can also influence the perception of emotion in embodied agents. The article first reviews the literature on specific autonomic signal patterns associated with certain affective states. Next, it proceeds to describe a realtime model for wrinkles, blushing, sweating, tearing and respiration that is capable of implementing those patterns. Two studies are then described: in the first, subjects compare surprise, sadness, anger, shame, pride and fear expressed in an agent with or without blushing, wrinkles, sweating or tears; in the second, subjects compare excitement, relaxation, focus, pain, relief, boredom, anger, fear, panic, disgust, surprise, startle, sadness and joy expressed in an agent with or without typical respiration patterns. The first study shows a statistically significant positive effect on perception of surprise, sadness, anger, shame and fear. The second study shows a statistically significant positive effect on perception of excitement, pain, relief, boredom, anger, fear, panic, disgust and startle. The relevance of these results is discussed for the fields of artificial intelligence and intelligent virtual agents.
Metallinou, Angeliki; Busso, Carlos; Lee, Sungbok; Narayanan, Shrikanth
Visual Emotion Recognition Using Compact Facial Representations and Viseme Information Proceedings Article
In: Proceedings of IEEE ICASSP 2010, Dallas, TX, 2010.
Abstract | Links | BibTeX | Tags: Virtual Humans
@inproceedings{metallinou_visual_2010,
title = {Visual Emotion Recognition Using Compact Facial Representations and Viseme Information},
author = {Angeliki Metallinou and Carlos Busso and Sungbok Lee and Shrikanth Narayanan},
url = {http://ict.usc.edu/pubs/Visual%20Emotion%20Recognition%20Using%20Compact%20Facial%20Representations%20and%20Viseme%20Information.pdf},
year = {2010},
date = {2010-03-01},
booktitle = {Proceedings of IEEE ICASSP 2010},
address = {Dallas, TX},
abstract = {Emotion expression is an essential part of human interaction. Rich emotional information is conveyed through the human face. In this study, we analyze detailed motion-captured facial information of ten speakers of both genders during emotional speech. We derive compact facial representations using methods motivated by Principal Component Analysis and speaker face normalization. Moreover, we model emotional facial movements by conditioning on knowledge of speech-related movements (articulation). We achieve average classi ô€‚¿cation accuracies on the order of 75% for happiness, 50-60% for anger and sadness and 35% for neutrality in speaker independent experiments. We also ô€‚¿nd that dynamic modeling and the use of viseme information improves recognition accuracy for anger, happiness and sadness, as well as for the overall unweighted performance. Index Terms– Emotion recognition, Principal Component Analysis, Principal Feature Analysis, Fisher Criterion, visemes},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
Emotion expression is an essential part of human interaction. Rich emotional information is conveyed through the human face. In this study, we analyze detailed motion-captured facial information of ten speakers of both genders during emotional speech. We derive compact facial representations using methods motivated by Principal Component Analysis and speaker face normalization. Moreover, we model emotional facial movements by conditioning on knowledge of speech-related movements (articulation). We achieve average classi ô€‚¿cation accuracies on the order of 75% for happiness, 50-60% for anger and sadness and 35% for neutrality in speaker independent experiments. We also ô€‚¿nd that dynamic modeling and the use of viseme information improves recognition accuracy for anger, happiness and sadness, as well as for the overall unweighted performance. Index Terms– Emotion recognition, Principal Component Analysis, Principal Feature Analysis, Fisher Criterion, visemes
Morency, Louis-Philippe
Co-occurrence Graphs: Contextual Representation for Head Gesture Recognition during Multi-Party Interactions Proceedings Article
In: Workshop on Use of Context for Visual Processing, 2010.
Abstract | Links | BibTeX | Tags: Virtual Humans
@inproceedings{morency_co-occurrence_2010,
title = {Co-occurrence Graphs: Contextual Representation for Head Gesture Recognition during Multi-Party Interactions},
author = {Louis-Philippe Morency},
url = {http://ict.usc.edu/pubs/Co-occurrence%20Graphs-%20Contextual%20Representation%20for%20Head%20Gesture%20Recognition%20during%20Multi-Party%20Interactions.pdf},
year = {2010},
date = {2010-01-01},
booktitle = {Workshop on Use of Context for Visual Processing},
abstract = {Head pose and gesture offer several conversational grounding cues and are used extensively in face-to-face interaction among people. To accurately recognize visual feedback, humans often use contextual knowledge from previous and current events to anticipate when feedback is most likely to occur. In this paper we describe how contextual information from other participants can be used to predict visual feedback and improve recognition of head gestures in multiparty interactions (e.g., meetings). An important contribution of this paper is our data-driven representation, called co-occurrence graphs, which models co-occurrence between contextual cues such as spoken words and pauses, and visual head gestures. By analyzing these co-occurrence patterns we can automatically select relevant contextual features and predict when visual gestures are more likely. Using a discriminative approach to multi-modal integration, our contextual representation using co-occurrence graphs improves head gesture recognition performance on a publicly available dataset of multi-party interactions.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
Head pose and gesture offer several conversational grounding cues and are used extensively in face-to-face interaction among people. To accurately recognize visual feedback, humans often use contextual knowledge from previous and current events to anticipate when feedback is most likely to occur. In this paper we describe how contextual information from other participants can be used to predict visual feedback and improve recognition of head gestures in multiparty interactions (e.g., meetings). An important contribution of this paper is our data-driven representation, called co-occurrence graphs, which models co-occurrence between contextual cues such as spoken words and pauses, and visual head gestures. By analyzing these co-occurrence patterns we can automatically select relevant contextual features and predict when visual gestures are more likely. Using a discriminative approach to multi-modal integration, our contextual representation using co-occurrence graphs improves head gesture recognition performance on a publicly available dataset of multi-party interactions.
DeVault, David; Robinson, Susan; Traum, David
IORelator: A Graphical User Interface to Enable Rapid Semantic Annotation for Data-Driven Natural Language Understanding Proceedings Article
In: Proceedings of the 5th Joint ISO-ACL/SIGSEM Workshop on Interoperable Semantic Annotation (ISA-5), 2010.
Abstract | Links | BibTeX | Tags: Virtual Humans
@inproceedings{devault_iorelator_2010,
title = {IORelator: A Graphical User Interface to Enable Rapid Semantic Annotation for Data-Driven Natural Language Understanding},
author = {David DeVault and Susan Robinson and David Traum},
url = {http://ict.usc.edu/pubs/IORelator-%20A%20Graphical%20User%20Interface%20to%20Enable%20Rapid%20Semantic%20Annotation%20for%20Data-Driven%20Natural%20Language%20Understanding.pdf},
year = {2010},
date = {2010-01-01},
booktitle = {Proceedings of the 5th Joint ISO-ACL/SIGSEM Workshop on Interoperable Semantic Annotation (ISA-5)},
abstract = {This paper describes a new annotation GUI, called IORelator, which is designed to facilitate rapid semantic annotation in support of high-performance data-driven NLU for spoken dialogue systems. We summarize our requirements for rapid NLU annotation, and discuss how the GUI views and operations that IORelator provides meet these needs by enabling thousands of natural utterances to be quickly annotated with their correct semantics.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
This paper describes a new annotation GUI, called IORelator, which is designed to facilitate rapid semantic annotation in support of high-performance data-driven NLU for spoken dialogue systems. We summarize our requirements for rapid NLU annotation, and discuss how the GUI views and operations that IORelator provides meet these needs by enabling thousands of natural utterances to be quickly annotated with their correct semantics.
Rosenbloom, Paul S.
An Architectural Approach to Statistical Relational AI Proceedings Article
In: Proceedings of the AAAI-10 Workshop on Statistical Relational AI, Atlanta, GA, 2010.
Abstract | Links | BibTeX | Tags: CogArch, Cognitive Architecture, Virtual Humans
@inproceedings{rosenbloom_architectural_2010,
title = {An Architectural Approach to Statistical Relational AI},
author = {Paul S. Rosenbloom},
url = {http://ict.usc.edu/pubs/An%20Architectural%20Approach%20to%20Statistical%20Relational%20AI.pdf},
year = {2010},
date = {2010-01-01},
booktitle = {Proceedings of the AAAI-10 Workshop on Statistical Relational AI},
address = {Atlanta, GA},
abstract = {The architectural approach to AI focuses on the fixed structure underlying intelligence. Applying it to statistical relational AI could stimulate investigations of statistical relational approaches across AI, encourage understanding of commonalities and compatibilities across this range, and yield new architectures significantly beyond today's best.},
keywords = {CogArch, Cognitive Architecture, Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
The architectural approach to AI focuses on the fixed structure underlying intelligence. Applying it to statistical relational AI could stimulate investigations of statistical relational approaches across AI, encourage understanding of commonalities and compatibilities across this range, and yield new architectures significantly beyond today's best.
Heylen, Dirk; Bevacqua, Elisabetta; Pelachaud, Catherine; Poggi, Isabella; Gratch, Jonathan; Schröder, Marc
Generating Listening Behaviour Book Section
In: Handbook of Emotion-Oriented Technologies, 2010.
Abstract | Links | BibTeX | Tags: Virtual Humans
@incollection{heylen_generating_2010,
title = {Generating Listening Behaviour},
author = {Dirk Heylen and Elisabetta Bevacqua and Catherine Pelachaud and Isabella Poggi and Jonathan Gratch and Marc Schröder},
url = {http://ict.usc.edu/pubs/Generating%20Listening%20Behaviour.pdf},
year = {2010},
date = {2010-01-01},
booktitle = {Handbook of Emotion-Oriented Technologies},
abstract = {In face-to-face conversations listeners provide feedback and comments at the same time as speakers are uttering their words and sentence. This 'talk' in the backchannel provides speakers with information about reception and acceptance – or lack thereof – of their speech. Listeners, through short verbalisations and non-verbal signals, show how they are engaged in the dialogue. The lack of incremental, real-time processing has hampered the creation of conversational agents that can respond to the human interlocutor in real time as the speech is being produced. The need for such feedback in conversational agents is, however, undeniable for reasons of naturalism or believability, to increase the efficiency of communication and to show engagement and building of rapport. In this chapter, the joint activity of speakers and listeners that constitutes a conversation is more closely examined and the work that is devoted to the construction of agents that are able to show that they are listening is reviewed. Two issues are dealt with in more detail. The first is the search for appropriate responses for an agent to display. The second is the study of how listening responses may increase rapport between agents and their human partners in conversation.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {incollection}
}
In face-to-face conversations listeners provide feedback and comments at the same time as speakers are uttering their words and sentence. This 'talk' in the backchannel provides speakers with information about reception and acceptance – or lack thereof – of their speech. Listeners, through short verbalisations and non-verbal signals, show how they are engaged in the dialogue. The lack of incremental, real-time processing has hampered the creation of conversational agents that can respond to the human interlocutor in real time as the speech is being produced. The need for such feedback in conversational agents is, however, undeniable for reasons of naturalism or believability, to increase the efficiency of communication and to show engagement and building of rapport. In this chapter, the joint activity of speakers and listeners that constitutes a conversation is more closely examined and the work that is devoted to the construction of agents that are able to show that they are listening is reviewed. Two issues are dealt with in more detail. The first is the search for appropriate responses for an agent to display. The second is the study of how listening responses may increase rapport between agents and their human partners in conversation.
Putten, Astrid M.; Kramer, Nicole C.; Gratch, Jonathan
How Our Personality Shapes Our Interactions with Virtual Characters - Implications for Research and Development Proceedings Article
In: 10th International Conference on Intelligent Virtual Agents, Philadelphia, PA, 2010.
Abstract | Links | BibTeX | Tags: Social Simulation, Virtual Humans
@inproceedings{von_der_putten_how_2010,
title = {How Our Personality Shapes Our Interactions with Virtual Characters - Implications for Research and Development},
author = {Astrid M. Putten and Nicole C. Kramer and Jonathan Gratch},
url = {http://ict.usc.edu/pubs/How%20Our%20Personality%20Shapes%20Our%20Interactions%20with%20Virtual%20Characters%20-%20Implications%20for%20Research%20and%20Development.pdf},
year = {2010},
date = {2010-01-01},
booktitle = {10th International Conference on Intelligent Virtual Agents},
address = {Philadelphia, PA},
abstract = {here is a general lack of awareness for the influence of users' personality traits on human-agent-interaction (HAI). Numerous studies do not even consider explanatory variables like age and gender although they are easily accessible. The present study focuses on explaining the occurrence of social effects in HAI. Apart from the original manipulation of the study we assessed the users' traits. Results show that participants' personality traits those traits which relate to persistent behavioral patterns in social contact (agreeableness, extraversion, approach avoidance, self-efficacy in monitoring others, shyness, public self-consciousness) were found to be predictive, whereas other personality traits and gender and age did not affect the evaluation. Results suggest that personality traits are better predictors for the evaluation outcome than the actual behavior of the agent as it has been manipulated in the experiment. Implications for the research on and development of virtual agents are discussed.},
keywords = {Social Simulation, Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
here is a general lack of awareness for the influence of users' personality traits on human-agent-interaction (HAI). Numerous studies do not even consider explanatory variables like age and gender although they are easily accessible. The present study focuses on explaining the occurrence of social effects in HAI. Apart from the original manipulation of the study we assessed the users' traits. Results show that participants' personality traits those traits which relate to persistent behavioral patterns in social contact (agreeableness, extraversion, approach avoidance, self-efficacy in monitoring others, shyness, public self-consciousness) were found to be predictive, whereas other personality traits and gender and age did not affect the evaluation. Results suggest that personality traits are better predictors for the evaluation outcome than the actual behavior of the agent as it has been manipulated in the experiment. Implications for the research on and development of virtual agents are discussed.
Huang, Lixing; Morency, Louis-Philippe; Gratch, Jonathan
Parasocial Consensus Sampling: Combining Multiple Perspectives to Learn Virtual Human Behavior Proceedings Article
In: International Conference on Autonomous Agents and Multiagent Systems (AAMAS), Toronto, Canada, 2010.
Abstract | Links | BibTeX | Tags: Virtual Humans
@inproceedings{huang_parasocial_2010,
title = {Parasocial Consensus Sampling: Combining Multiple Perspectives to Learn Virtual Human Behavior},
author = {Lixing Huang and Louis-Philippe Morency and Jonathan Gratch},
url = {http://ict.usc.edu/pubs/Parasocial%20Consensus%20Sampling-%20Combining%20Multiple%20Perspectives%20to%20Learn%20Virtual%20Human%20Behavior.pdf},
year = {2010},
date = {2010-01-01},
booktitle = {International Conference on Autonomous Agents and Multiagent Systems (AAMAS)},
address = {Toronto, Canada},
abstract = {Virtual humans are embodied software agents that should not only be realistic looking but also have natural and realistic behaviors. Traditional virtual human systems learn these interaction behaviors by observing how individuals respond in face-to-face situations (i.e., direct interaction). In contrast, this paper introduces a novel methodological approach called parasocial consensus sampling (PCS) which allows multiple individuals to vicariously experience the same situation to gain insight on the typical (i.e., consensus view) of human responses in social interaction. This approach can help tease apart what is idiosyncratic from what is essential and help reveal the strength of cues that elicit social responses. Our PCS approach has several advantages over traditional methods: (1) it integrates data from multiple independent listeners interacting with the same speaker, (2) it associates probability of how likely feedback will be given over time, (3) it can be used as a prior to analyze and understand the face-to-face interaction data, (4) it facilitates much quicker and cheaper data collection. In this paper, we apply our PCS approach to learn a predictive model of listener backchannel feedback. Our experiments demonstrate that a virtual human driven by our PCS approach creates significantly more rapport and is perceived as more believable than the virtual human driven by face-to-face interaction data.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
Virtual humans are embodied software agents that should not only be realistic looking but also have natural and realistic behaviors. Traditional virtual human systems learn these interaction behaviors by observing how individuals respond in face-to-face situations (i.e., direct interaction). In contrast, this paper introduces a novel methodological approach called parasocial consensus sampling (PCS) which allows multiple individuals to vicariously experience the same situation to gain insight on the typical (i.e., consensus view) of human responses in social interaction. This approach can help tease apart what is idiosyncratic from what is essential and help reveal the strength of cues that elicit social responses. Our PCS approach has several advantages over traditional methods: (1) it integrates data from multiple independent listeners interacting with the same speaker, (2) it associates probability of how likely feedback will be given over time, (3) it can be used as a prior to analyze and understand the face-to-face interaction data, (4) it facilitates much quicker and cheaper data collection. In this paper, we apply our PCS approach to learn a predictive model of listener backchannel feedback. Our experiments demonstrate that a virtual human driven by our PCS approach creates significantly more rapport and is perceived as more believable than the virtual human driven by face-to-face interaction data.
Marsella, Stacy C.; Gratch, Jonathan; Petta, Paola
Computational Models of Emotion Book Section
In: Scherer, K. R.; Bänziger, T.; Roesch, (Ed.): A blueprint for an affectively competent agent: Cross-fertilization between Emotion Psychology, Affective Neuroscience, and Affective Computing, Oxford University Press, Oxford, 2010.
Abstract | Links | BibTeX | Tags: Social Simulation, Virtual Humans
@incollection{marsella_computational_2010,
title = {Computational Models of Emotion},
author = {Stacy C. Marsella and Jonathan Gratch and Paola Petta},
editor = {K. R. Scherer and T. Bänziger and Roesch},
url = {http://ict.usc.edu/pubs/Computational%20Models%20of%20Emotion.pdf},
year = {2010},
date = {2010-01-01},
booktitle = {A blueprint for an affectively competent agent: Cross-fertilization between Emotion Psychology, Affective Neuroscience, and Affective Computing},
publisher = {Oxford University Press},
address = {Oxford},
abstract = {Recent years have seen a significant expansion in research on computational models of human emotional processes, driven both by their potential for basic research on emotion and cognition as well as their promise for an ever increasing range of applications. This has led to a truly interdisciplinary, mutually beneficial partnership between emotion research in psychology and computational science, of which this volume is an exemplar. To understand this partnership and its potential for transforming existing practices in emotion research across disciplines and for disclosing important novel areas of research, we explore in this chapter the history of work in computational models of emotion including the various uses to which they have been put, the theoretical traditions that have shaped their development, and how these uses and traditions are reflected in their underlying architectures.},
keywords = {Social Simulation, Virtual Humans},
pubstate = {published},
tppubtype = {incollection}
}
Recent years have seen a significant expansion in research on computational models of human emotional processes, driven both by their potential for basic research on emotion and cognition as well as their promise for an ever increasing range of applications. This has led to a truly interdisciplinary, mutually beneficial partnership between emotion research in psychology and computational science, of which this volume is an exemplar. To understand this partnership and its potential for transforming existing practices in emotion research across disciplines and for disclosing important novel areas of research, we explore in this chapter the history of work in computational models of emotion including the various uses to which they have been put, the theoretical traditions that have shaped their development, and how these uses and traditions are reflected in their underlying architectures.
Olden, Megan; Cukor, Judith; Rizzo, Albert; Rothbaum, Barbara O.; Difede, JoAnn
House calls revisited: leveraging technology to overcome obstacles to veteran psychiatric care and improve treatment outcomes Journal Article
In: Annals of the New York Academy of Sciences (NYAS), vol. 1208, pp. 133–141, 2010.
Abstract | Links | BibTeX | Tags: MedVR, Virtual Humans
@article{olden_house_2010,
title = {House calls revisited: leveraging technology to overcome obstacles to veteran psychiatric care and improve treatment outcomes},
author = {Megan Olden and Judith Cukor and Albert Rizzo and Barbara O. Rothbaum and JoAnn Difede},
url = {http://ict.usc.edu/pubs/House%20calls%20revisited.pdf},
year = {2010},
date = {2010-01-01},
journal = {Annals of the New York Academy of Sciences (NYAS)},
volume = {1208},
pages = {133–141},
abstract = {Despite an increasing number of military service members in need of mental health treatment following deployment to Iraq and Afghanistan, numerous psychological and practical barriers limit access to care. Perceived stigma about admitting psychological difficulties as well as frequent long distances to treatment facilities reduce many veterans' willingness and ability to receive care. Telemedicine and virtual human technologies offer a unique potential to ex- pand services to those in greatest need. Telemedicine-based treatment has been used to address multiple psychiatric disorders, including posttraumatic stress disorder, depression, and substance use, as well as to provide suicide risk as- sessment and intervention. Clinician education and training has also been enhanced and expanded through the use of distance technologies, with trainees practicing clinical skills with virtual patients and supervisors connecting with clin- icians via videoconferencing. The use of these innovative and creative vehicles offers a significant and as yet unfulfilled promise to expand delivery of high-quality psychological therapies, regardless of clinician and patient location.},
keywords = {MedVR, Virtual Humans},
pubstate = {published},
tppubtype = {article}
}
Despite an increasing number of military service members in need of mental health treatment following deployment to Iraq and Afghanistan, numerous psychological and practical barriers limit access to care. Perceived stigma about admitting psychological difficulties as well as frequent long distances to treatment facilities reduce many veterans' willingness and ability to receive care. Telemedicine and virtual human technologies offer a unique potential to ex- pand services to those in greatest need. Telemedicine-based treatment has been used to address multiple psychiatric disorders, including posttraumatic stress disorder, depression, and substance use, as well as to provide suicide risk as- sessment and intervention. Clinician education and training has also been enhanced and expanded through the use of distance technologies, with trainees practicing clinical skills with virtual patients and supervisors connecting with clin- icians via videoconferencing. The use of these innovative and creative vehicles offers a significant and as yet unfulfilled promise to expand delivery of high-quality psychological therapies, regardless of clinician and patient location.
Kang, Sin-Hwa; Gratch, Jonathan
Virtual humans elicit socially anxious interactants' verbal self-disclosure Proceedings Article
In: Computer Animation and Virtual Worlds, Saint-Malo, France, 2010.
Abstract | Links | BibTeX | Tags: Virtual Humans
@inproceedings{kang_virtual_2010,
title = {Virtual humans elicit socially anxious interactants' verbal self-disclosure},
author = {Sin-Hwa Kang and Jonathan Gratch},
url = {http://ict.usc.edu/pubs/Virtual%20humans%20elicit%20socially%20anxious%20interactants%E2%80%99%20verbal%20self-disclosure.pdf},
year = {2010},
date = {2010-01-01},
booktitle = {Computer Animation and Virtual Worlds},
address = {Saint-Malo, France},
abstract = {We explored the relationship between interactants' social anxiety (shyness) and the interactional fidelity of virtual humans. We specifically addressed whether the contingent nonverbal feedback of virtual humans affects the association between interactants' social anxiety and their verbal self-disclosure. This subject was investigated across three experimental conditions where participants interacted with real humans and virtual humans in computer mediated interactions. The results demonstrated that more socially anxious people would reveal intimate, verbal information about themselves more than less socially anxious people, while exhibiting more fluent speech in the interview task when interacting with virtual humans than with real human video avatars. We discuss the implication of this association between the interactional fidelity of an embodied agent and social anxiety in a human interactant on the design of virtual humans for social skills training and psychotherapy, especially for shy populations.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
We explored the relationship between interactants' social anxiety (shyness) and the interactional fidelity of virtual humans. We specifically addressed whether the contingent nonverbal feedback of virtual humans affects the association between interactants' social anxiety and their verbal self-disclosure. This subject was investigated across three experimental conditions where participants interacted with real humans and virtual humans in computer mediated interactions. The results demonstrated that more socially anxious people would reveal intimate, verbal information about themselves more than less socially anxious people, while exhibiting more fluent speech in the interview task when interacting with virtual humans than with real human video avatars. We discuss the implication of this association between the interactional fidelity of an embodied agent and social anxiety in a human interactant on the design of virtual humans for social skills training and psychotherapy, especially for shy populations.
DeVault, David; Sagae, Kenji; Traum, David
Incremental Interpretation and Prediction of Utterance Meaning for Interactive Dialogue Journal Article
In: Dialogue and Discourse, vol. 2, no. 1, pp. 143–170, 2010.
Abstract | Links | BibTeX | Tags: Virtual Humans
@article{devault_incremental_2010,
title = {Incremental Interpretation and Prediction of Utterance Meaning for Interactive Dialogue},
author = {David DeVault and Kenji Sagae and David Traum},
url = {http://ict.usc.edu/pubs/Incremental%20Interpretation%20and%20Prediction%20of%20Utterance%20Meaning%20for%20Interactive%20Dialogue.pdf},
year = {2010},
date = {2010-01-01},
journal = {Dialogue and Discourse},
volume = {2},
number = {1},
pages = {143–170},
abstract = {We present techniques for the incremental interpretation and prediction of utterance meaning in dialogue systems. These techniques open possibilities for systems to initiate responsive overlap be- haviors during user speech, such as interrupting, acknowledging, or completing a user's utterance while it is still in progress. In an implemented system, we show that relatively high accuracy can be achieved in understanding of spontaneous utterances before utterances are completed. Further, we present a method for determining when a system has reached a point of maximal understanding of an ongoing user utterance, and show that this determination can be made with high precision. Finally, we discuss a prototype implementation that shows how systems can use these abilities to strategically initiate system completions of user utterances. More broadly, this framework facili- tates the implementation of a range of overlap behaviors that are common in human dialogue, but have been largely absent in dialogue systems.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {article}
}
We present techniques for the incremental interpretation and prediction of utterance meaning in dialogue systems. These techniques open possibilities for systems to initiate responsive overlap be- haviors during user speech, such as interrupting, acknowledging, or completing a user's utterance while it is still in progress. In an implemented system, we show that relatively high accuracy can be achieved in understanding of spontaneous utterances before utterances are completed. Further, we present a method for determining when a system has reached a point of maximal understanding of an ongoing user utterance, and show that this determination can be made with high precision. Finally, we discuss a prototype implementation that shows how systems can use these abilities to strategically initiate system completions of user utterances. More broadly, this framework facili- tates the implementation of a range of overlap behaviors that are common in human dialogue, but have been largely absent in dialogue systems.
Bosse, Tibor; Gratch, Jonathan; Hoorn, Johan F.; Pontier, Matthijs; Siddiqui, Ghazanfar F.
Comparing Three Computational Models of Affect Proceedings Article
In: 8th International Conference on Practical Applications of Agents and Multi-Agent Systems, Salamanca, Spain, 2010.
Abstract | Links | BibTeX | Tags: Virtual Humans
@inproceedings{bosse_comparing_2010,
title = {Comparing Three Computational Models of Affect},
author = {Tibor Bosse and Jonathan Gratch and Johan F. Hoorn and Matthijs Pontier and Ghazanfar F. Siddiqui},
url = {http://ict.usc.edu/pubs/Comparing%20%20Three%20Computational%20Models%20of%20Affect.pdf},
year = {2010},
date = {2010-01-01},
booktitle = {8th International Conference on Practical Applications of Agents and Multi-Agent Systems},
address = {Salamanca, Spain},
abstract = {In aiming for behavioral fidelity, artificial intelligence cannot and no longer ignores the formalization of human affect. Affect modeling plays a vital role in faithfully simulating human emotion and in emotionally-evocative technology that aims at being real. This paper offers a short expose about three models concerning the generation and regulation of affect: CoMERG, EMA and I-PEFiCADM, which each in their own right are successfully applied in the agent and robot domain. We argue that the three models partly overlap and where distinct, they complement one another. We provide an analysis of the theoretical concepts, and provide a blueprint of an integration, which should result in a more precise representation of affect simulation in virtual humans.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
In aiming for behavioral fidelity, artificial intelligence cannot and no longer ignores the formalization of human affect. Affect modeling plays a vital role in faithfully simulating human emotion and in emotionally-evocative technology that aims at being real. This paper offers a short expose about three models concerning the generation and regulation of affect: CoMERG, EMA and I-PEFiCADM, which each in their own right are successfully applied in the agent and robot domain. We argue that the three models partly overlap and where distinct, they complement one another. We provide an analysis of the theoretical concepts, and provide a blueprint of an integration, which should result in a more precise representation of affect simulation in virtual humans.
2009
Cukor, Judith; Spitalnick, Josh; Difede, JoAnn; Rizzo, Albert; Rothbaum, Barbara O.
Emerging Treatments for PTSD Journal Article
In: Clinical Psychology Review, 2009.
Abstract | Links | BibTeX | Tags: MedVR, Virtual Humans
@article{cukor_emerging_2009,
title = {Emerging Treatments for PTSD},
author = {Judith Cukor and Josh Spitalnick and JoAnn Difede and Albert Rizzo and Barbara O. Rothbaum},
url = {http://ict.usc.edu/pubs/Emerging%20treatments%20for%20PTSD.pdf},
year = {2009},
date = {2009-12-01},
journal = {Clinical Psychology Review},
abstract = {Recent innovations in posttraumatic stress disorder (PTSD) research have identified new treatments with significant potential, as well as novel enhancements to empirically-validated treatments. This paper reviews emerging psychotherapeutic and pharmacologic interventions for the treatment of PTSD. It examines the evidence for a range of interventions, from social and family-based treatments to technological-based treatments. It describes recent findings regarding novel pharmacologic approaches including propranolol, ketamine, prazosin, and methylenedioxymethamphetamine. Special emphasis is given to the description of virtual reality and D-cycloserine as enhancements to prolonged exposure therapy.},
keywords = {MedVR, Virtual Humans},
pubstate = {published},
tppubtype = {article}
}
Recent innovations in posttraumatic stress disorder (PTSD) research have identified new treatments with significant potential, as well as novel enhancements to empirically-validated treatments. This paper reviews emerging psychotherapeutic and pharmacologic interventions for the treatment of PTSD. It examines the evidence for a range of interventions, from social and family-based treatments to technological-based treatments. It describes recent findings regarding novel pharmacologic approaches including propranolol, ketamine, prazosin, and methylenedioxymethamphetamine. Special emphasis is given to the description of virtual reality and D-cycloserine as enhancements to prolonged exposure therapy.
Pütten, Astrid M.; Krämer, Nicole; Gratch, Jonathan
Who Is there? Can a Virtual Agent Really Elicit Social Presence? Proceedings Article
In: The 12th Annual International Workshop on Presence, Los Angeles, CA, 2009.
Abstract | Links | BibTeX | Tags: Virtual Humans
@inproceedings{von_der_puitten_who_2009,
title = {Who Is there? Can a Virtual Agent Really Elicit Social Presence?},
author = {Astrid M. Pütten and Nicole Krämer and Jonathan Gratch},
url = {http://ict.usc.edu/pubs/whosthere.pdf},
year = {2009},
date = {2009-11-01},
booktitle = {The 12th Annual International Workshop on Presence},
address = {Los Angeles, CA},
abstract = {This study investigates whether humans perceive a higher degree of social presence when interacting with an animated character that displays natural as opposed to no listening behaviors and whether this interacts with people's believe that they are interacting with an agent or an avatar. In a 2x2 between subjects experimental design 83 participants were either made believe that they encounter an agent, or that they communicate with another participant mediated by an avatar. In fact, in both conditions the communication partner was an autonomous agent that either exhibited high or low behavioral realism. We found that participants experienced equal amounts of presence, regardless of interacting with an agent or an avatar. Behavioral realism, however, had an impact on the subjective feeling of presence: people confronted with a character displaying high behavioral realism reported a higher degree of mutual awareness.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
This study investigates whether humans perceive a higher degree of social presence when interacting with an animated character that displays natural as opposed to no listening behaviors and whether this interacts with people's believe that they are interacting with an agent or an avatar. In a 2x2 between subjects experimental design 83 participants were either made believe that they encounter an agent, or that they communicate with another participant mediated by an avatar. In fact, in both conditions the communication partner was an autonomous agent that either exhibited high or low behavioral realism. We found that participants experienced equal amounts of presence, regardless of interacting with an agent or an avatar. Behavioral realism, however, had an impact on the subjective feeling of presence: people confronted with a character displaying high behavioral realism reported a higher degree of mutual awareness.
Rosenbloom, Paul S.
Towards Uniform Implementation of Architectural Diversity Proceedings Article
In: Proceedings of the AAAI Fall Symposium on Multi-Representational Architectures for Human-Level Intelligence, 2009.
Abstract | Links | BibTeX | Tags: CogArch, Cognitive Architecture, Virtual Humans
@inproceedings{rosenbloom_towards_2009-1,
title = {Towards Uniform Implementation of Architectural Diversity},
author = {Paul S. Rosenbloom},
url = {http://ict.usc.edu/pubs/Towards%20Uniform%20Implementation%20of%20Architectural%20Diversity.pdf},
year = {2009},
date = {2009-11-01},
booktitle = {Proceedings of the AAAI Fall Symposium on Multi-Representational Architectures for Human-Level Intelligence},
abstract = {Multi-representational architectures exploit diversity to yield the breadth of capabilities required for intelligent behavior in the world, but in so doing can sacrifice too much of the complementary benefits of architectural uniformity. The proposal here is to couple the benefits of diversity and uniformity through establishment of a uniform graph-based implementation level for diverse architectures.},
keywords = {CogArch, Cognitive Architecture, Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
Multi-representational architectures exploit diversity to yield the breadth of capabilities required for intelligent behavior in the world, but in so doing can sacrifice too much of the complementary benefits of architectural uniformity. The proposal here is to couple the benefits of diversity and uniformity through establishment of a uniform graph-based implementation level for diverse architectures.
Sagae, Kenji; Gordon, Andrew S.
Clustering Words by Syntactic Similarity Improves Dependency Parsing of Predicate-Argument Structures Proceedings Article
In: International Conference on Parsing Technologies (IWPT-09), Paris, France, 2009.
Abstract | Links | BibTeX | Tags: The Narrative Group, Virtual Humans
@inproceedings{sagae_clustering_2009,
title = {Clustering Words by Syntactic Similarity Improves Dependency Parsing of Predicate-Argument Structures},
author = {Kenji Sagae and Andrew S. Gordon},
url = {http://ict.usc.edu/pubs/Clustering%20Words%20by%20Syntactic%20Similarity%20Improves%20Dependency%20Parsing%20of%20Predicate-Argument%20Structures.pdf},
year = {2009},
date = {2009-10-01},
booktitle = {International Conference on Parsing Technologies (IWPT-09)},
address = {Paris, France},
abstract = {We present an approach for deriving syntactic word clusters from parsed text, grouping words according to their unlexicalized syntactic contexts. We then explore the use of these syntactic clusters in leveraging a large corpus of trees generated by a high-accuracy parser to improve the accuracy of another parser based on a different formalism for representing a different level of sentence structure. In our experiments, we use phrase-structure trees to produce syntactic word clusters that are used by a predicate-argument dependency parser, significantly improving its accuracy.},
keywords = {The Narrative Group, Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
We present an approach for deriving syntactic word clusters from parsed text, grouping words according to their unlexicalized syntactic contexts. We then explore the use of these syntactic clusters in leveraging a large corpus of trees generated by a high-accuracy parser to improve the accuracy of another parser based on a different formalism for representing a different level of sentence structure. In our experiments, we use phrase-structure trees to produce syntactic word clusters that are used by a predicate-argument dependency parser, significantly improving its accuracy.
DeVault, David; Sagae, Kenji; Traum, David
Can I finish? Learning when to respond to incremental interpretation results in interactive dialogue Proceedings Article
In: 10th SIGdial Workshop on Discourse and Dialogue, London, UK, 2009.
Abstract | Links | BibTeX | Tags: Virtual Humans
@inproceedings{devault_can_2009,
title = {Can I finish? Learning when to respond to incremental interpretation results in interactive dialogue},
author = {David DeVault and Kenji Sagae and David Traum},
url = {http://ict.usc.edu/pubs/Can%20I%20%EF%AC%81nish%20Learning%20when%20to%20respond%20to%20incremental%20interpretation%20results%20in%20interactive%20dialogue.pdf},
year = {2009},
date = {2009-09-01},
booktitle = {10th SIGdial Workshop on Discourse and Dialogue},
address = {London, UK},
abstract = {We investigate novel approaches to responsive overlap behaviors in dialogue systems, opening possibilities for systems to interrupt, acknowledge or complete a user’s utterance while it is still in progress. Our specific contributions are a method for determining when a system has reached a point of maximal understanding of an ongoing user utterance, and a prototype implementation that shows how systems can use this ability to strategically initiate system completions of user utterances. More broadly, this framework facilitates the implementation of a range of overlap behaviors that are common in human dialogue, but have been largely absent in dialogue systems.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
We investigate novel approaches to responsive overlap behaviors in dialogue systems, opening possibilities for systems to interrupt, acknowledge or complete a user’s utterance while it is still in progress. Our specific contributions are a method for determining when a system has reached a point of maximal understanding of an ongoing user utterance, and a prototype implementation that shows how systems can use this ability to strategically initiate system completions of user utterances. More broadly, this framework facilitates the implementation of a range of overlap behaviors that are common in human dialogue, but have been largely absent in dialogue systems.
Melo, Celso M.; Zheng, Liang; Gratch, Jonathan
Expression of Moral Emotions in Cooperating Agents Proceedings Article
In: Proceedings of the 9th International Conference on Intelligent Virtual Agents (IVA), Amsterdam, The Netherlands, 2009.
Abstract | Links | BibTeX | Tags: Virtual Humans
@inproceedings{de_melo_expression_2009-1,
title = {Expression of Moral Emotions in Cooperating Agents},
author = {Celso M. Melo and Liang Zheng and Jonathan Gratch},
url = {http://ict.usc.edu/pubs/Expression%20of%20Moral%20Emotions%20in%20Cooperating%20Agents.pdf},
year = {2009},
date = {2009-09-01},
booktitle = {Proceedings of the 9th International Conference on Intelligent Virtual Agents (IVA)},
address = {Amsterdam, The Netherlands},
abstract = {Moral emotions have been argued to play a central role in the emergence of cooperation in human-human interactions. This work describes an experiment which tests whether this insight carries to virtual human-human interactions. In particular, the paper describes a repeated-measures experiment where subjects play the iterated prisoner's dilemma with two versions of the virtual human: (a) neutral, which is the control condition; (b) moral, which is identical to the control condition except that the virtual human expresses gratitude, distress, remorse, reproach and anger through the face according to the action history of the game. Our results indicate that subjects cooperate more with the virtual human in the moral condition and that they perceive it to be more human-like. We discuss the relevance these results have for building agents which are successful in cooperating with humans.},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
Moral emotions have been argued to play a central role in the emergence of cooperation in human-human interactions. This work describes an experiment which tests whether this insight carries to virtual human-human interactions. In particular, the paper describes a repeated-measures experiment where subjects play the iterated prisoner's dilemma with two versions of the virtual human: (a) neutral, which is the control condition; (b) moral, which is identical to the control condition except that the virtual human expresses gratitude, distress, remorse, reproach and anger through the face according to the action history of the game. Our results indicate that subjects cooperate more with the virtual human in the moral condition and that they perceive it to be more human-like. We discuss the relevance these results have for building agents which are successful in cooperating with humans.