Training and Simulation Journal Covers ICT Innovations for Clinical Care

ICT’s MedVR Lab was featured in an article in the latest issue of Training and Simulation Journal that discussed recent innovations in simulation and virtual reality that impact medical and psychological care. The article quoted Skip Rizzo and Belinda Lange, both of ICT’s Medical Virtual Reality group.

Read the full story.

Wired’s Danger Room Covers Multi-Center PTSD Study including ICT’s VR Exposure Therapy

A story on Wired’s Danger Room blog covers the multi-center DoD-funded trial that ICT is involved in. The story notes that part of the study will evaluate virtual reality exposure therapy and that ICT’s Skip Rizzo has pioneered this treatment for use treating soldiers and veterans from Iraq and Afghanistan.

Read the full story.

Louis-Philippe Morency, “Towards Multimodal Sentiment Analysis: Integrating Linguistic, Auditory and Visual Cues”

With more than 10,000 new videos posted online every day on social websites such as YouTube and Facebook, the internet is becoming an almost innite source of information. One crucial challenge for the coming decade is to be able to harvest relevant information from this constant ow of multimodal data.Subjectivity and sentiment analysis focuses on the automatic identication of private states, such as opinions, emotions, sentiments, evaluations, beliefs, and speculations in natural language. While subjectivity classication labels data as either subjective or objective, sentiment classification adds an additional level of granularity, by further classifying subjective data as either positive, negative or neutral. Much of the work to date on subjectivity and sentiment analysis has focused on textual data, and a number of resources have been created including lexicons or large annotated datasets. Given the accelerated growth of other media on the Web and elsewhere, which includes massive collections of videos (e.g., YouTube, Vimeo, VideoLectures), images (e.g., Flickr, Picasa, Facebook), audio (e.g., podcasts), the ability to address the identification of opinions and sentiment for diverse modalities is becoming increasingly important.

New York Daily News Covers DoD-Funded Study of ICT’s Virtual Reality Therapy

story in the New York Daily News featured the $11 million multi-center trial and collaboration between the USC Institute for Creative Technologies, New York-Presbyterian Hospital/Weill Cornell Medical Center and Emory University to continue research in using virtual reality therapy to heal those with combat-related PTSD.  The study uses the Virtual Iraq/Virtual Afghanistan virtual reality exposure therapy developed by ICT’s Skip Rizzo, who is one of the co-investigators of the study.

Read the NY Daily News story here.

Abhijeet Ghosh, Graham Fyffe, Borom Tunwattanapong, Jay Busch, XueMing Yu, Paul Debevec: “Multiview Face Capture using Polarized Spherical Gradient Illumination”

We present a novel process for acquiring detailed facial geometry with high resolution diffuse and specular photometric information from multiple viewpoints using polarized spherical gradient illumination. Key to our method is a new pair of linearly polarized lighting patterns which enables multiview diffuse-specular separation under a given spherical illumination condition from just two photographs. The patterns – one following lines of latitude and one following lines of longitude – allow the use of fixed linear polarizers in front of the cameras, enabling more efficient acquisition of diffuse and specular albedo and normal maps from multiple viewpoints. In a second step, we employ these albedo and normal maps as input to a novel multi-resolution adaptive domain message passing stereo reconstruction algorithm to create high resolution facial geometry. To do this, we formulate the stereo reconstruction from multiple cameras in a commonly parameterized domain for multiview reconstruction. We show competitive results consisting of high-resolution facial geometry with relightable reflectance maps using five DSLR cameras. Our technique scales well for multiview acquisition without requiring specialized camera systems for sensing multiple polarization states.

Local Vets Needed for $11 Million DoD-Funded Study of ICT-Developed Virtual Reality Therapy for PTSD

$11 Million DOD Grant Funds Study of Virtual Reality and Other Therapies for PTSD

Multi-Center Clinical Trial to Recruit Military and Civilian Personnel
With PTSD Related to the Iraq and Afghanistan Wars

Researchers at the University of Southern California Institute for Creative Technologies, New York- Presbyterian Hospital/Weill Cornell Medical Center, and the Emory University School of Medicine have been awarded an $11 million, four-year grant from the U.S. Department of Defense to test different ways to treat post-traumatic stress disorder (PTSD), including the use of virtual reality exposure therapy.

The study will involve 300 military and civilian personnel who have been diagnosed with PTSD as a consequence of their service in the Iraq and Afghanistan wars. The goals are to decrease the time needed for effective treatment of PTSD, give the right treatment to the right person, and identify factors involved in its development and response to treatment.

The researchers will also examine personal and genetic factors that may impact an individual’s chances of developing PTSD, as well as future response to therapy.

The grant is a culmination of years of collaborative and novel research by investigators who are known as leaders in the field. It is led by JoAnn Difede, director of the Program for Anxiety and Traumatic Stress Studies at New York-Presbyterian/Weill Cornell and Professor of Psychology in Psychiatry at Weill Cornell Medical College Co-investigators are Albert “Skip” Rizzo, associate director of medical virtual reality at the University of Southern California Institute for Creative Technologies (ICT) and a research professor at USC’s Davis School of Gerontology and the USC Department of Psychiatry & Behavioral Sciences; and Barbara Rothbaum, director of the Trauma and Anxiety Recovery Program and professor in the Department of Psychiatry and Behavioral Sciences at Emory University School of Medicine.

Each group of study participants will receive two educational sessions followed by seven weekly sessions of exposure therapy — either virtual reality exposure therapy or prolonged imaginal exposure therapy.

Each group will also be randomized to receive a pill containing either the drug D-cycloserine or placebo prior to exposure therapy sessions. D-cycloserine, or DCS, is an antibiotic approved by the FDA 20 years ago to treat tuberculosis, but has also been found by researchers at Emory to speed up the kind of learning required to reconcile fearful memories.

Exposure therapy is an evidence-based treatment that has been demonstrated to be effective in the treatment of PTSD. The first addition of virtual reality to exposure therapy for PTSD was known as Virtual Vietnam, created by Barbara Rothbaum, which was a simple VR simulation designed to treat soldiers who served in the Vietnam War.

JoAnn Difede began using VR with burn unit patients in the 1990s, and then in 2001 she worked with Hunter Hoffman, from the University of Washington to create a Virtual World Trade Center system to treat survivors of the 9/11 terrorist attack.

In 2003, Skip Rizzo developed Virtual Iraq to treat soldiers from the Iraq War. He initially used a modification of the ICT-developed video game Full Spectrum Warrior to create the sights, sounds, smells, even sensations, of the Iraq War — which has been continually updated to create the most sophisticated VR PTSD exposure therapy system to date.

Virtual Iraq and now Virtual Afghanistan are currently being used at 55 military, VA and university-based clinical sites.

The study will be conducted at three clinical sites: the Walter Reed Army Medical Center and National Intrepid Center of Excellence in the Washington, D.C. area; and the Veteran’s Administration in Long Beach, Calif., and New York-Presbyterian/Weill Cornell in New York city.

For more information about participation in the Southern California location for this study at the Long Beach VA Medical Center:
Email: VRatLongBeach@gmail.com
Phone 1 562.826.5784
Facebook: http://www.facebook.com/#!/PTSDResearch?sk=wall
Website: http://www.scire-lb.org/ptsd.html

For more information about the other study sites, interested parties may contact Brittany Mello by calling (212) 821-0783 or by email at brm2016@med.cornell.edu

Media Contacts:
University of Southern California
Institute for Creative Technologies
Orli Belman
belman@ict.usc.edu
(310) 709-4156

NewYork-Presbyterian/Weill Cornell
Takla Boujaoude
tab2016@med.cornell.edu
(212) 821-0560

Emory University
Kathi Baker,
kobaker@emory.edu
(404) 727-9371

William Yang Wang, Kallirroi Georgila: “Automatic Detection of Unnatural Word-Level Segments in Unit-Selection Speech Synthesis”

We investigate the problem of automatically detecting unnatural word-level segments in unit selection speech synthesis. We use a large set of features, namely, target and join costs, language models, prosodic cues, energy and spectrum, and Delta Term Frequency Inverse Document Frequency (TF-IDF), and we report comparative results between different feature types and their combinations. We also compare three modeling methods based on Support Vector Machines (SVMs), Random Forests, and Conditional Random Fields (CRFs). We then discuss our results and present a comprehensive error analysis.

Paul Debevec: “From Spider-Man to Avatar: Achieving Photoreal Digital Actors”

Somewhere between “Final Fantasy” in 2001 and “The Curious Case of Benjamin Button” in 2008, digital actors crossed the “Uncanny Valley” from looking strangely synthetic to believably real. This talk describes some of the technological advances that have enabled this achievement. For an in-depth example, the talk describes how high-resolution face scanning, advanced character rigging, and performance-driven facial animation were combined to create “Digital Emily”, a collaboration between our laboratory and Image Metrics. Actress Emily O’Brien was scanned in Light Stage 5 in 33 facial poses at the resolution of skin pores and fine wrinkles. These scans were assembled into a rigged face model driven by Image Metrics’ video-based animation software, and the resulting photoreal facial animation premiered at SIGGRAPH 2008. The talk also presents a 3D teleconferencing system that uses live facial scanning and an autostereoscopic display to transmit a person’s face in 3D and make eye contact with remote collaborators, and a new headmounted facial performance-capture system based on photometric stereo.

Angela Nazarian: “Duplicate: An Annotation Scheme”

Angela Nazarian: “Muri Overview: An Annotation Scheme for Cross Cultural Argumentation and Persuasion Dialogues”

Jacquelyn Ford Morie Speaks about Avatars at UCSD Neural Engineering Seminar

Virtual worlds are bigger than you might think.  Active users of these world number almost 2 billion worldwide. This talk will explore the once and future possibilities for connecting with our avatars, and why neuroscientists should care!

3D Display from the ICT Graphics Lab Featured in Science News

An article in Science News highlighted the 3D display and 3D teleconferencing systems developed by ICT’s Paul Debevec and colleagues.  “At the University of Southern California in Los Angeles, Paul Debevec has figured out a way to pack hundreds of different viewing angles together to eliminate eyestrain,” states the article. “He’s ditching flat screens in favor of a rapidly flickering projector. It bounces images off a pair of aluminum plates jointed together like an A-frame tent, a double-sided mirror of sorts that spins 900 times per minute.”

Read the full story here.

Matthew Jensen Hays, Teri Silva, Todd Richmond: “Assessing learning from a mixed-media, mobile counter-IED trainer”

Improvised explosive devices (IEDs) cause more than half of Coalition casualties. In 2009, the Institute for Creative Technologies (ICT) developed a mobile counter-IED training system. This system—the Experiential Counter-IED Immersive Training Environment (ExCITE)—was described at I/ITSEC 2010. Since that time, researchers at the ICT have developed several ways of assessing learning from ExCITE. One measure is a paper-based pretest/posttest. Another is a system that tracks trainees’ responses to interactive training tools as well as their performance during a simulated vehicle patrol. This paper describes how these measures were created and refined. This paper also presents findings from our empirical studies on the effectiveness of this counter-IED training.

Julia Campbell, Matthew Jensen Hays, Mark Core, Mike Birch, Matt Bosack, Richard E. Clark: “Interpersonal and Leadership Skills: Using Virtual Humans to Teach New Officers”

Successful leaders have strong interpersonal skills (Army FM 6-22, ALDS). With their subordinates, they must develop relationships founded on mutual trust and respect. However, the majority of new officers’ interpersonal skills development takes place on the job. Further, the formal training they do receive consists primarily of role-play sessions with their peers. This situation is not conducive to consistent practice, effective feedback, or accurate skill assessment and improvement. To address these issues, we have created a virtual-human based role-playing system. The Virtual Officer Leadership Trainer (VOLT) offers a controlled practice environment while maintaining instructor control and facilitating throughput. An officer in training interacts with a virtual human subordinate via branching, scripted dialog and behavior that allow the trainee to apply specific strategies and skills. A classroom of other trainees observes this interaction in real time. At each decision point in the interaction with the virtual subordinate, the trainees use personal response systems (“clickers”) to indicate what they believe to be the correct course of action. These data are available to an instructor, who monitors the class’s performance and conducts an after action review. VOLT thus provides a consistent training experience without sacrificing trainee throughput or instructor control. It allows an entire class to participate in a single role-play, stimulating discussion, and facilitating peer and instructor evaluation in real time. VOLT’s instructional design is based on cognitive task analysis (CTA) interviews with expert leaders. The CTA identified a set of learning objectives, which are organized into short, memorable key phrases such as “initiate, check, ask, respond, and evaluate.” These learning objectives were integrated into the training system through the virtual human’s speech and actions. This paper discusses VOLT’s educational goals, instructional design, and technological approach.

ICT’s Kevin Feeley and Eric Forbell Present SimCoach to Executive Education Consortium

Eric Forbell and Kevin Feeley presented SimCoach and the SimCoach Authoring Toolset as part of an event during the University Consortium For Executive Education Team Development Conference held at the USC Marshall School of Business.

John Galen Buckwalter, Skip Rizzo, Bruce Sheffield John, Lisa Finlay, Andrew Wong, Ester Chin, Stephanie Smolinski: ” Analyzing the Impact of Stress: A Comparison Between a Factor Analytic and a Composite Measurement of Allostatic Load”

Stress is possibly the hallmark characteristic of the current conflicts confronting the United States. Extended and repeated deployments require the ability on the part of war-fighters to effectively process stress in ways never before routinely encountered. Stress is well defined as a series of psychological and physiological processes that occur in response to a stressor, or the perception of stress. The physiological response to stress follows an identified path, a robust neuroendocrine response leads to responses in the cardiovascular, metabolic, renal, inflammatory and immune systems. After a stress response, the body’s natural tendency is to return to a steady state, a process called allostasis. If the body is not effective in returning to homeostasis, or if the environment is such that stress is repeated, markers of dysfunction may be apparent in the physiological systems that respond to stress. A method of measuring multiple biomarkers of stress responsive systems and determining who shows consistent evidence of dysfunction was developed by Bruce McEwen and labeled allostatic load (AL). AL is most frequently measured by developing a level of risk for each biomarker and obtaining an AL score for the number of biomarkers the criterion for risk is met. This provides a single, equal-weighted measure of AL and does not allow for the identification of multi-systems. We employed a principal component factor analysis on a set of biomarkers and scored each factor using unit weighting. We compared the predictive power of 7 obliquely rotated factors to that of a composite AL marker. The set of factors predicted more of the variance in measures of depression, anxiety, and medical outcomes, it also provided evidence of the systems most involved in the development of pathology. The results confirm that AL is best analyzed as a multi-system construct. Not only does this predict more variance, it also provides suggestions as to the mechanisms underlying stress related disorders.

Louis-Philippe Morency Presents at ACM Multimedia

Face-to-face communication is a highly dynamic process where participants mutually exchange and interpret linguistic and gestural signals. Even when only one person speaks at the time, other participants exchange information continuously amongst themselves and with the speaker through gesture, gaze, posture and facial expressions. To correctly interpret the high-level communicative signals, an observer needs to jointly integrate all spoken words, subtle prosodic changes and simultaneous gestures from all participants. In this paper, we present our ongoing research effort at USC MultiComp Lab to create models of human communication dynamic that explicitly take into consideration the multimodal and interpersonal aspects of human face-to-face interactions. The computational framework presented in this paper has wide applicability, including the recognition of human social behaviors, the synthesis of natural animations for robots and virtual humans, improved multimedia content analysis, and the diagnosis of social and behavioral disorders (e.g., autism spectrum disorder).

Stacy Marsella, Teresa Day: “Socially Optimized Learning in Virtual Environments (SOLVE)”

Although young men who have sex with men (MSM) are at high risk for contracting HIV, few interventions address the affective/automatic factors (e.g., sexual arousal, shame/stigma) that may precipitate young MSM’s risk-taking. A National Institutes of Health (NIH)-funded DVD interactive video intervention that simulated a “virtual date” with guides/mentors reduced sexual risk over 3-months for Black, Latino and Caucasian young MSM. In the current work, limitations of the DVD format (e.g., number of different risk challenges MSM encounter; DVD quickly becomes dated) were addressed with 3-D animated intelligent agents/interactive digital storytelling using a Unity Game platform. The development (e.g., design, art, social science formative research, etc.) of this NIH funded game for changing risky behavior is described as well as the ongoing national randomized “on-line” evaluation over 6 months.

Immersive Journalism Collaboration with ICT’s MxR Lab Heading to Sundance Film Festival

Hunger in Los Angeles, a virtual reality project with real-world impact is scheduled to appear at the 2012 Sundance Film Festival, January 19-29, in Park City, Utah and January 20 – May 19 at the Salt Lake Art Center in Salt Lake City, Utah.

Part of the New Frontiers exhibit highlighting work that pushes the boundaries of storytelling and the moving image, Hunger in Los Angeles recreates an eyewitness account of a crisis incident on a food bank line at the First Unitarian Church in Los Angeles. Former journalist and USC Annenberg Fellow Nonny De la Peña combined the game development tool Unity 3D, a head mounted display with motion tracking, and live audio, to construct a simulated world in which audiences can suit up, walk around, and interact with other characters on the scene. The project was developed at the ICT MxR Lab, with additional support from the USC Annenberg School of Communications and Journalism.  Several ICT team members contributed to the project. Mark Bolas served as the creative producer, Bradley Newman was the lead artist and programmer and John Brennan performed art direction and motion capture.

Read the story on the USC Annenberg School’s website.

Visit the Sundance New Frontier Website.

Visit Nonny de la Pena’s website.

Paul Debevec Speaks on Computer Animiation in NYC

Paul Debevec, a computer science professor at the University of Southern California (USC), as well as the associate director for graphics research at the USC Institute for Creative Technologies in Los Angeles, will delve into the world of digital animation in a special presentation at Manhattan’s Times Center on November 22, 2011 from 6:00 to 8:00 p.m.

Professor Debevec is an Academy-Award-winning visual effects director whose work has been featured in major films including Avatar and The Curious Case of Benjamin Button.  Joining Debevec will be the dean of the USC Viterbi School of Engineering, Yannis C. Yortsos.

CNN Features ICT’s SimCoach Project

CNN highlighted SimCoach, a web-based interactive simulation technology project co-led by Skip Rizzo and others at ICT.  SimCoach is designed to help veterans and their families find resources to deal with post-traumatic stress disorder. The story noted that the program will eventually attempt to detect the emotion behind spoken words, which is especially hard for computers to analyze. While humans may be able to tell when something is wrong with a close friend or family member through speech pattern, a computer can have a hard time picking up these signals, Rizzo said.

Andrew Jones, Graham Fyffe, XueMing Yu, Wan-Chun Ma, Jay Busch, Ryosuke Ichikari, Mark Bolas, Paul Debevec: “Head-mounted Photometric Stereo for Performance Capture”

Head-mounted cameras are an increasingly important tool for capturing facial performances to drive virtual characters. They provide a fixed, unoccluded view of the face, useful for observing motion capture dots or as input to video analysis. However, the 2D imagery captured with these systems is typically affected by ambient light and generally fails to record subtle 3D shape changes as the face performs. We have developed a system that augments a head-mounted camera with LED-based photometric stereo. The system allows observation of the face independent of the ambient light and generates per-pixel surface normals so that the performance is recorded dynamically in 3D. The resulting data can be used for facial relighting or as better input to machine learning algorithms for driving an animated face.

Louis-Philippe Morency, Rada Mihalcea, Paval Doshi: “Towards Multimodal Sentiment Analysis: Harvesting Opinions from The Web”

With more than 10,000 new videos posted online every day on social websites such as YouTube and Facebook, the in- ternet is becoming an almost in nite source of information. One crucial challenge for the coming decade is to be able to harvest relevant information from this constant ow of mul- timodal data. This paper addresses the task of multimodal sentiment analysis, and conducts proof-of-concept experi- ments that demonstrate that a joint model that integrates visual, audio, and textual features can be e ectively used to identify sentiment in Web videos. This paper makes three important contributions. First, it addresses for the rst time the task of tri-modal sentiment analysis, and shows that it is a feasible task that can bene t from the joint exploitation of visual, audio and textual modalities. Second, it identi es a subset of audio-visual features relevant to sentiment analy- sis and present guidelines on how to integrate these features. Finally, it introduces a new dataset consisting of real online data, which will be useful for future research in this area.

Borom Tunwattanapong, Abhijeet Ghosh, Paul Debevec: “Practical Image-Based Relighting and Editing with Spherical-Harmonics and Local Lights”

We present a practical technique for image-based relighting under environmental illumination which greatly reduces the number of required photographs compared to traditional techniques, while still achieving high quality editable relighting results. The proposed method employs an optimization procedure to combine spherical harmonics, a global lighting basis, with a set of local lights. Our choice of lighting basis captures both low and high frequency components of typical surface reflectance functions while generating close approximations to the ground truth with an order of magnitude less data. This technique benefits the acquisition process by reducing the number of required photographs, while simplifying the modification of reflectance data and enabling artistic lighting edits for post-production effects. Here, we demonstrate two desirable lighting edits, modifying light intensity and angular width, employing the proposed lighting basis.

Ari Shapiro: “Building a Character Animation System”

Abstract of Presentation: We synthesize natural-looking locomotion, reaching and grasping for a virtual character in order to accomplish a wide range of movement and manipulation tasks in real time. Our virtual characters can move while avoiding obstacles, as well as manipulate arbitrarily shaped objects, regardless of height, location or placement in a virtual environment. Our characters can touch, reach and grasp objects while maintaining a high quality appearance. We demonstrate a system that combines these skills in an interactive setting suitable for interactive games and simulations.

Jacquelyn Ford Morie Speaks on Virtual Worlds at 2011 Saban Symposium 2011

The Saban Symposium focused on the process of supporting the needs of youth and families as they make the transition from pediatric to adult-oriented health systems. Topics included health care reform, communication and use of social media, health economics, bench to bedside translation and measuring quality and efficiency in the health care system. Morie was part of a panel discussion and addressed ways online virtual worlds could support children with chronic diseases as they transition to adulthood.

The event honored Roberta G. Williams, professor of pediatrics at the Keck School of Medicine of the University of Southern California.

David DeVault, Anton Leuski, Kenji Sagae: “An Evaluation of Alternative Strategies for Implementing Dialogue Policies Using Statistical Classification and Rules”

We present and evaluate a set of architectures for conversational dialogue systems, exploring rule-based and statistical classification approaches. In a case study, we show that while a rule-based dialogue policy is theoretically capable of high accuracy if perfect natural language understanding (NLU) is assumed, a classification approach that combines the dialogue policy with NLU has practical advantages.

Former U.N.Commander Romeo Dallaire Visits USC and ICT

On Friday, November 4, Romeo Dallaire, a Canadian Senator and retired Lieutenant-General, toured the USC Institute for Creative Technologies as part of a visit to USC where he also spoke to students and recorded his testimonial of the Rwandan Genocide for the USC Shoah Foundation Institute for Visual History and Education. Dallaire is credited with saving more than 30,000 lives during the 1994 Rwandan Genocide and has spoken openly of his struggles with post-traumatic stress. At ICT, he saw demonstrations of the virtual reality work the institute is doing in the area of mental health.

Paul S. Rosenbloom: “Mental imagery in a graphical cognitive architecture”

Can mental imagery be incorporated uniformly into a cognitive architecture by leveraging the mixed (relational and probabilistic) hybrid (discrete and continuous) processing supported by factor graphs? This article takes an initial step towards answering this question via an implementation in a graphical cognitive architecture of a version of the Eight Puzzle based on a hybrid function representation and a factor node optimized for object translation.

Morteza Dehghani: “Analyzing the debate over the Construction of the ‘Ground Zero Mosque’”

There is evidence that striking differences in the very definitions of morality are at the root of many social-ideological differences within a country. Haidt and Graham (2007) propose that liberals and conservatives in the US have very different ways of seeing the social environment around them, and rely on distinct moral structures and ideologies. Consequently, several important differences have been noted in the political rhetoric employed by these groups (Lakoff 2002, 2008; Marietta 2008, 2009). Lakoff (2008) argues that the type of language used in political discussions is of utmost importance because it “is far more than a means of expression and communication… It organizes and provides access to the system of concepts used in thinking” (p. 231). In other words, the language in these discussions often conveys the value systems adhered to by liberal and conservative groups. In this paper, we analyze conservative and liberal blogs posts, and their corresponding comments, related to the construction of a Muslim community center close to Ground Zero. Most of the controversy and debates surrounding the issue took place online and thus this methodology seemed quite apropos. Using two different statistical text analysis techniques, we show that there are significant differences in the use of various linguistic features, and in choice of words, between liberals and conservatives. The aim of the first experiment was to see if the differences in language use and choice of words are great enough that blog posts can be automatically classified as conservative or liberal using a machine learning technique. If we are able to classify these blog posts, then we will be able to determine the indicative features of each group using feature analysis and gain insight into what makes the blogs conservative or liberal. Our results indicate that choice of words used by these two ideological groups were distinct enough that our system was able to classify their blog posts as conservative or liberal with an accuracy of 85.6%. Feature analysis revealed that the most distinctive aspect of either liberal or conservative blogs is not the description, or the ideology, of the in-group, but rather the use of words related to the negative portrayal of the out-group. Similar to Haidt and Graham (2007), Lakoff (2002) argues that the ideologies of conservatives and liberals embody their value systems and personal conceptions of morality. Instead, our results show that at least in political debates, the ideas that make these groups liberal or conservative, are stereotypes of the out-group. In the second experiment, by examining posts in different time blocks, we show that there was an increase in words related to affective processes and anger over time, especially for conservatives. We argued that this increase is potentially related to the use of sacred rhetoric, as there was a significant correlation between anger and the use of religious words. The use of sacred rhetoric has been linked to the emergence of sacred values (Marietta, 2008; Dehghani et al., 2010), as values that get tied to religion more easily achieve a sacred status (Marietta, 2009). In conclusion, by analyzing over 3000 conservative and liberal blog posts related to the constructions of Park51, our results confirm significant differences in the use of language, and its resultant emotions, between the two groups. Language use in these blogs reflects ideological differences between liberals and conservatives. We believe the ability to perform this type of mass text analysis and to track changes of different psychological processes over different periods of time, as they naturally unfold among diverse cultural groups, can provide new insights which arguably cannot be achieved in an experimental setting inside the lab.

ICT Virtual Humans Change the Face of Training

At a Naval training facility in Newport, R.I., gunner’s mate Jacob Cabrillo, a new junior sailor, sits hunched in his chair, fully outfitted in his Navy uniform.

The humming of the ventilation and the creaking of the ship buzz in the background. After getting into a fight with a fellow shipmate, Cabrillo nervously awaits a meeting with his superior officer.

The superior officer is actually a student at the U.S. Navy’s Officer Training Command, about to take part in a lesson in how to informally counsel a subordinate. However, Cabrillo is not really a subordinate, a sailor or even a real person. He’s a virtual human in a mock navy ship, created to look and act like a real-life sailor, as part of the Immersive Naval Officer Training System (INOTS).

INOTS is a groundbreaking leadership development tool created by the USC Institute for Creative Technologies (ICT), with collaboration from the Office of Naval Research and the Naval Service Training Command. This recently installed prototype is believed to mark the first time the military is using a life-sized virtual human for classroom training.

ICT also is developing an Army version scheduled to be delivered to Ft. Benning, Ga., later this year.

“Leadership training is about more than physical strength and giving orders,” said Kim LeMasters, creative director at ICT. “Officers are often called upon to handle personnel problems but may lack the necessary skills and experience to be effective managers or counselors. Our system, which combines believable characters and powerful storytelling, provides a compelling way to improve how these trainees deal with interpersonal issues.”

The issues can include financial problems, domestic disputes and on-the-job quarrels. While textbooks may provide procedures on how to handle such conflicts, they can’t substitute for real relationships.

INOTS is like a case study that has come to life. Virtual human Cabrillo has the ability to respond, raise his voice or shake his head in frustration. Students can learn from speaking with him or a diverse cast of characters created at ICT. These agents operate in virtual worlds, have the ability to express a multitude of emotional responses and can perform actions in their environment, all without human direction.

Using the INOTS program, trainees are able to see demonstrations of appropriate communication techniques and then can practice with a virtual human. While one student is interacting with the simulated sailor, up to 50 other students are in a classroom also participating by using clickers to select which action they would take. An instructor then leads the class through an evaluation and discussion of the conversation.

Traditionally, students might have practiced with an actor hired to play the part of a sailor or with fellow students role-playing. But employing live role-players can be very costly, and they may lack the ability to provide effective feedback. Virtual humans offer consistent performances, can work nonstop in any location and are equipped with software that allows for comparison and analysis of every exchange.

Most importantly, students seem to be paying attention to them.

“In the Navy, if a sailor feels like he or she is about to fall asleep, they’re told to stand up,” said Julia Campbell, a research associate at ICT and the instructional design lead for INOTS, who recently tested the system in Rhode Island. “No one stood up in the back of the class at the Newport base. They were completely engaged in the program.”

Read the story at USC News.

Paul S. Rosenbloom: “Bridging dichotomies in cognitive architectures for virtual humans”

Desiderata for cognitive architectures that are to support the extent of human-level intelligence required in virtual humans imply the need to bridge a range of dichotomies faced by such architectures. The focus here is first on two general approaches to building such bridges – addition and reduction – and then on a pair of general tools – graphical models and piecewise continuous functions – that exploit the second approach towards developing such an architecture. Evaluation is in terms of the architecture’s demonstrated ability and future potential for bridging the dichotomies.

Elnaz Nouri, Ron Artstein, Anton Leuski, David Traum: “Augmenting Conversational Characters with Generated Question-Answer Pairs”

We take a conversational character trained on a set of linked question-answer pairs authored by hand, and augment its training data by adding sets of question-answer pairs which are generated automatically from texts on different topics. The augmented characters can answer questions about the new topics, at the cost of some performance loss on questions about the topics that the original character was trained to answer.

Paul S. Rosenbloom: “Fusing Symbolic and Decision-Theoretic Problem Solving and Perception in a Graphical Cognitive Architecture”

A step is taken towards fusing symbolic and decision-theoretic problem solving in a cognitive architecture by implementing the latter in an architecture within which the former has already been demonstrated. The graphical models upon which the architecture is based enable a uniform implementation of both varieties of problem solving. They also enable a uniform combination with forms of decision-relevant perception, highlighting a potential path towards a tight coupling between central cognition and peripheral perception.

ABC’s Extreme Makeover: Home Edition Installs ICT Technology in a New Home for Veteran with PTSD

On Friday, November 4, ABC’s Extreme Makeover: Home Edition will feature the family of a veteran who suffers from post-traumatic stress disorder and brain injury. Jewel Mine, a customized ICT physical therapy video-game developed by Belinda Lange of ICT’s Medical Virtual Reality lab, was installed in home so that veteran Brian Hill could continue his motor rehabilitation work at home. We are proud to be a part of this episode which shines a light on PTSD, traumatic brain injuries and the soldiers and families living with these conditions. Check your local listings for times.

USC Daily Trojan Features ICT’s Virtual Patient Collaboration with the USC School of Social Work

The USC Daily Trojan covered ICT’s virtual patient and how it will be used to train future clinicians through a collaboration with the USC School of Social Work Center for Innovation and Research on Veterans and Military Families. USC Social Work students are expected to begin using the system, which has been undergoing several rounds of testing and evaluation, in early 2012. The article explains that virtual patients are artificially intelligent, interactive agents that serve as a realistic clinical representation of a service member, veteran or military family member. The technology provides students with a chance to advance practical skills with realistic client interactions. The virtual patients can speak, express body language, show emotion and offer immediate feedback. The article also notes that ICT’s Natural Language Dialogue group is contributing to the system.

Mark Bolas and ICT’s MxR Lab In Fast Company

Fast Company article about the power of narrative to influence behavior change, covered the recent USC Body Computing Conference and quoted Mark Bolas, associate director of mixed reality at ICT, who was a panelist at the conference.

“Don’t assume the iPhone is the interface you’ll be stuck with forever,” he said, challenging attendees to imagine new settings where the quest for health could play out. Bolas believes that environments that fully employ one’s perception and cognition can create a useful, visceral memory of the experience that leaves a lasting impression, stated the article.

Matthew Jensen Hays, Adriel Boals: “Is Measuring Short-Term Memory as Easy as Reducing Test-Expectancy?”

Learners intuitively understand that they must rely on long-term memory processes in order to recall information after an intervening distractor (e.g., Watkins & Watkins, 1974). Several researchers (e.g., Muter, 1980) have supposed that reducing the expectation of a test after a distractor reduces long-term memory involvement, permitting a “pure” measure of forgetting from short-term memory. We report an experiment that contradicts this notion by revealing substantial proactive interference in a low-test-expectancy Brown-Peterson paradigm. Because proactive interference indicates that long-term memory is involved (e.g., Craik & Birtwistle, 1971), these results suggest that reducing test-expectancy alone is not sufficient to reduce long-term memory involvement.

Thomas Parsons, James Reinebold: “Neuroscience and Simulation Interface for Adaptive Assessment in Serious Games”

While advances in military relevant simulations provide potential for increasing assessment of Soldier readiness to Return-to-Duty (e.g., following a blast injury), little has been done to develop these simulations into adaptive virtual environments (AVE) for improved neurocognitive and psychophysiological assessment. Adaptive assessments offer the potential for dynamically adapting the difficulty level specific to the patient’s knowledge or ability. We present an iteration of the Virtual Reality Cognitive Performance Test (VRCPAT) that proffers a framework for adapting scenarios in the Virtual Battlespace 2 (VBS2) game engine based upon the user’s neurocognitive and psychophysiological (e.g., heart rate, skin response, heart rate, and pupil diameter) states.

Louis-Philippe Morency: “Computational Study of Human Communication Dynamics”

Face-to-face communication is a highly dynamic process where participants mutually exchange and interpret linguistic and gestural signals. Even when only one person speaks at the time, other participants exchange information continuously amongst themselves and with the speaker through gesture, gaze, posture and facial expressions. To correctly interpret the high-level communicative signals, an observer needs to jointly integrate all spoken words, subtle prosodic changes and simultaneous gestures from all participants. In this paper, we present our ongoing research effort at USC MultiComp Lab to create models of human communication dynamic that explicitly take into consideration the multimodal and interpersonal aspects of human face-to-face interactions. The computational framework presented in this paper has wide applicability, including the recognition of human social behaviors, the synthesis of natural animations for robots and virtual humans, improved multimedia content analysis, and the diagnosis of social and behavioral disorders (e.g., autism spectrum disorder).

Mark Bolas Gives Keynote Speech at International Symposium on Mixed and Augmented Reality (ISMAR)

Mixed and augmented reality is crossing the chasm from research to widespread adoption. This requires us to think beyond registered planes of pixels in space, and confront the unregistered mess of users, culture, and application.

Skip Rizzo Honored with The Help Group’s Annual Professional Achievement Award

Skip Rizzo, ICT’s associate director for medical virtual reality, received The Help Group’s Professional Achievement Award for his contributions to the field of virtual reality technology and its clinical applications. Rizzo was also the featured speaker at the organization’s Summit 2011, Advances and Best Practices in Austim, Learning Disabilities and ADHD. The Help Group is recognized internationally as one of the leading center’s in the treatment of childhood disorders and for driving public/political policy on access to care for children with special needs.

ICT in Army Magazine

The November issue of Army Magazine features two stories about ICT research and prototypes. The first story showcases ICT’s immersive technologies, with a focus on ICT’s two new virtual human training demonstrations for interpersonal skills practice that were recently installed at Army and Navy bases, as well as on ICT’s work using virtual characters in the mental health arena. The second story describes recent updates to the Joint Fires and Effects Trainer that ICT delivered as a research prototype in 2004.

“The U.S. Army and ICT partnership is creating immersive technologies that are changing the way we train, rehabilitate, educate and promote resilience,” explained Randall W. Hill Jr., Ph.D., executive director of ICT in the magazine. “But we’re not just trying to create cool immersive experiences. We’re creating experiences that matter in the virtual arena.”

Read the Virtual Humans article here.

Read the JFETS story here.

ICT MedVR Collaboration Wins Best Poster Award

A collaborative virtual reality motor rehabilitation project between ICT and the UCLA Wireless Health Institute that integrates Belinda Lange’s Jewel Mine game with a SMARTglove for upper limb rehabilitation won the best poster/demo award at the Wireless Health Conference in San Diego.  Lange heads up the motor rehab group in the ICT Medical Virtual Reality Lab.

The paper is titled: Gaming for Upper Extremities Rehabilitation and the authors are Ming-Chun Huang, Ethen Chen, Chien-Yen Chang, Belinda Lange, Majid Sarrafzadeh.

Congratulations to all.

Emmett Tomai, Laxman Thapa, Andrew Gordon, Sin-hwa Kang: “Causality in Hundreds of Narratives of the Same Events”

Empirical research supporting computational models of narrative is often constrained by the lack of large-scale corpora with deep annotation. In this paper, we report on our annotation and analysis of a dataset of 283 individual narrations of the events in two short video clips. The utterances in the narrative transcripts were annotated to align with known events in the source videos, offering a unique opportunity to study the regularities and variations in the way that different people describe the exact same set of events. We identified the causal relationships between events in the two video clips, and investigated the role that causality plays in determining whether subjects will mention a particular story event and the likelihood that these events will be told in the order that they occurred in the original videos.

Celso de Melo, Peter Carnevale, Dimitrios Antos, Jonathan Gratch: “A Computer Model of the Interpersonal Effect of Emotion Displayed in a Social Dilemma”

The paper presents a computational model for decision-making in a social dilemma that takes into account the other party’s emotion displays. The model is based on data collected in a series of recent studies where participants play the iterated prisoner’s dilemma with agents that, even though following the same action strategy, show different emotion displays according to how the game unfolds. We collapse data from all these studies and fit, using maximum likelihood estimation, probabilistic models that predict likelihood of cooperation in the next round given different features. Model 1 predicts based on round outcome alone. Model 2 predicts based on outcome and emotion displays. Model 3 also predicts based on outcome and emotion but, considers contrast effects found in the empirical studies regarding the order with which participants play cooperators and non-cooperators. To evaluate the models, we replicate the original studies but, substitute the humans for the models. The results reveal that Model 3 best replicates human behavior in the original studies and Model 1 does the worst. The results, first, emphasize recent research about the importance of nonverbal cues in social dilemmas and, second, reinforce that people attend to contrast effects in their decision-making. Theoretically, the model provides further insight into how people behave in social dilemmas. Pragmatically, the model could be used to drive an agent that is engaged in a social dilemma with a human (or another agent).

Wu Dongrui, Thomas Parsons: “Inductive Transfer Learning for Handling Individual Differences in Affective Computing”

Although psychophysiological and affective computing approaches may increase facility for development of the next generation of human-computer systems, the data resulting from research studies in affective computing include large individual differences. As a result, it is important that the data gleaned from an affective computing system be tailored for each individual user by re-tuning it using user-specific training examples. Given the often time-consuming and/or expensive nature of efforts to obtain such training examples, there is a need to either 1) minimize the number of user-specific training examples required; or 2) to maximize the learning performance through the incorporation of auxiliary training examples from other subjects. In [11] we have demonstrated an active class selection approach for the first purpose. Herein we use transfer learning to improve the learning performance by combining user-specific training examples with auxiliary training examples from other subjects, which are similar but not exactly the same as the user-specific training examples. We report results from an arousal classification application to demonstrate the effectiveness of transfer learning in a Virtual Reality Stroop Task designed to elicit varying levels of arousal.

Geovany Ramirez, Tadas Baltrusaitis, Louis-Philippe Morency: “Modeling Latent Discriminative Dynamic of Multi-Dimensional Affective Signals”

During face-to-face communication, people continuously exchange para-linguistic information such as their emotional state through facial expressions, posture shifts, gaze patterns and prosody. These affective signals are subtle and complex. In this paper, we propose to explicitly model the interaction between the high level perceptual features using Latent-Dynamic Conditional Random Fields. This approach has the advantage of explicitly learning the sub-structure of the a ective signals as well as the extrinsic dynamic between emotional labels. We evaluate our approach on the Audio-Visual Emotion Challenge (AVEC 2011) dataset. By using visual features easily computable using o -theshelf sensing software (vertical and horizontal eye gaze, head tilt and smile intensity), we show that our approach based on LDCRF model outperforms previously published baselines for all four a ective dimensions. By integrating audio features, our approach also outperforms the audio-visual baseline.

Mohammed (Ehsan) Hoque, Louis-Philippe Morency, Rosalind Picard: “Are you friendly or just polite? – Analysis of smiles in spontaneous face-to-face interactions”

This work is part of a research effort to understand and characterize the morphological and dynamic features of polite and amused smiles. We analyzed a dataset consisting of young adults (n=61), interested in learning about banking services, who met with a professional banker face-to-face in a conference room while both participants’ faces were unobtrusively recorded. We analyzed 258 instances of amused and polite smiles from this dataset, noting also if they were shared, which we defined as if the rise of one starts before the decay of another. Our analysis confirms previous findings showing longer durations of amused smiles while also suggesting new findings about symmetry of the smile dynamics. We found more symmetry in the velocities of the rise and decay of the amused smiles, and less symmetry in the polite smiles. We also found fastest decay velocity for polite but shared smiles.

Paul Debevec: “From Spider-Man to Avatar: Achieving Photoreal Digital Actors”

Somewhere between “Final Fantasy” in 2001 and “The Curious Case of Benjamin Button” in 2008, digital actors crossed the “Uncanny Valley” from looking strangely synthetic to believably real. This talk describes some of the technological advances that have enabled this achievement. For an in-depth example, the talk describes how high-resolution face scanning, advanced character rigging, and performance-driven facial animation were combined to create “Digital Emily”, a collaboration between our laboratory and Image Metrics. Actress Emily O’Brien was scanned in Light Stage 5 in 33 facial poses at the resolution of skin pores and fine wrinkles. These scans were assembled into a rigged face model driven by Image Metrics’ video-based animation software, and the resulting photoreal facial animation premiered at SIGGRAPH 2008. The talk also presents a 3D teleconferencing system that uses live facial scanning and an autostereoscopic display to transmit a person’s face in 3D and make eye contact with remote collaborators, and a new headmounted facial performance-capture system based on photometric stereo.

“Paul Debevec: A Pioneer in Virtual Cinematography”

Paul Debevec (Spider-Man 2, Avatar) is a Research Professor at the University of Southern California’s Institute for Creative Technologies in Playa Vista, CA. A pioneer in Virtual Cinematography, he invented High Dynamic Range Image-Based Lighting and received a Scientific Academy Award for the Light Stage facial capture systems used in films such as Superman Returns, The Curious Case of Benjamin Button, and TRON: Legacy. He is a member of AMPAS, VES, and serves as Vice-President of ACM SIGGRAPH. He will speak about his research and professional work.

A Foreign Policy Magazine Writer Tries out ICT’s UrbanSim Application

Foreign Policy‘s Michael Peck frequently writes about the U.S. military and video games and jumped at the chance to try out the ICT-developed UrbanSim game. After testing his hand at counter-insurgency tactics, Peck had some great insights about the complexities of officier’s decisions and the consequences of military action.

Read the full article.

Mark Bolas Serves as Expert Games and Social Media Panelist at USC’s 2011 Body Computing Conference

Mark Bolas served on the Games, Social Media, Entertainment—New Roles in Medicine panel at the USC Body Computing Conference. The conference provides a forum for the world’s foremost scientific thinkers, inventors, entrepreneurs, authors and others working in the field of body computing to discuss the steps necessary to give body computing a staunch and compelling foundation in the future of medical science.

Belinda Lange: “Virtual Reality: General overview and perspectives for neurorehabilitation”

This presentation will provide a general overview of Virtual Reality using examples from applications that have been developed for neurorehabilitation. The use of video games for rehabilitation will be also discussed. The use of these technologies will be discussed within the perspective of neurorehabilitation. New tools and future directions in the field will be presented to the group and open for discussion at the end of the presentation.

Congkai Sun, Louis-Philippe Morency: “Towards Speaker Adaptation for Dialogue Act Recognition”

Dialogue act labels are being used to represent a higher level intention of utterances during human conversation (Stolcke et al., 2000). Automatic dialogue act recognition is still an active research topic. The conventional approach is to train one generic classifier using a large corpus of annotated utterances (Stolcke et al., 2000). One aspect that makes it so challenging is that people can express the same intentions using a very different set of spoken words. Imagine how different the vocabulary used by a native English speaker or a foreigner can be. Even more, people can have different intentions when using the exact same spoken words. These idiosyncratic differences in dialogue acts make the learning of generic classifiers extremely challenging. Luckily, in many applications such as face-to-face meetings or tele-immersion, we have access to archives of previous interactions with the same participants. From these archives, a small subset of spoken utterances can be efficiently annotated. As we will later show in our experiments, even a small number of annotated utterances can make a significant differences in the dialogue act recognition performance.

Jacki Morie: “The Application of Simulation Technology for Training in Military Medicine”

Morie was an invited speaker on her work in bringing Mindfulness Based Stress Reduction (MBSR) to the Virtual World to help veterans reduce stress.

THE APPLICATION OF SIMULATION TECHNOLOGY FOR TRAINING IN MILITARY MEDICINE
Discussion Panel
Wednesday, September 21, 1:30–3:00 p.m.

Human Performance Modeling
Chair: Valerie Rice, U.S. Army Research Lab;Cochair:
Petra Alfred, U.S. Army Research Laboratory
Panelists: Raymond Bateman and Gary Boykin, U.S.
Army Research Lab; Jacki Morie, U. of Southern
California; Jack Norfleet, Simulation and Training
Technology Ctr.; Mark Scerbo, Old Dominion U.

Louis-Philippe Morency: “Towards Speaker Adaptation for Dialogue Act Recognition”

In this work we study the effectiveness of speaker adaptation for Dialogue Act recognition in multiparty meetings. First, we analyze idiosyncracy in dialogue verbal acts by qualitatively studying the differences and conflicts among speakers and by quantitively comparing speaker-specific models. Based on these observations, we propose a new approach for dialogue act recognition based on reweighted domain adaptation which effectively balance the influence of speaker specific and other speakers’ data. Our experiments on a real-world meeting dataset show that with even only $200$ speaker-specific annotated dialogue acts, the performances on dialogue act recognition are significantly improved when compared to several baseline algorithms. To our knowledge, this work is the first to tackle this promising research direction of speaker adaptation for dialogue act recogntion.

USC Trojan Family Magazine Features ICT

An article in the Autumn 2011 issue of USC Trojan Family Magazine features ICT research and prototypes. Under the headline, Brave New World, the story highlights the futuristic tools developed at ICT that are used today to train soldiers, treat patients, teach students and more.

“It is not enough to use technologies to create a cool experience,” says Randall W. Hill Jr., ICT’s executive director. “We are creating a whole new way for people to engage with computers so that they can practice, learn and perform better.”

A photo slideshow accompanies the online version of the story.

View the full story and slideshow here.

ICT and Viterbi Researchers Win Best Paper Award at IVA 2011

A paper written by Jason Tsai, a Ph.D. candidate in the Department of Computer Science of the USC Viterbi School of Engineering received the Best Paper Award at the 11th International Conference on Intelligent Virtual Agents in Reykjavik, Iceland. The paper, Empirical Evaluation of Computational Emotional Contagion Models, was also authored by Stacy Marsella of ICT, Milind Tambe of the Viterbi School, and Emma Bowring of the University of the Pacific. The award-winning research compares the ability of two computer models to reproduce two real crowd panic scenes.  In the paper the authors explain, “the underlying mechanism of the spread of emotions has seen little quantification and application to computational agents despite extensive evidence of its impacts in everyday life.”

ICT Workshop on Overlap in Human-Computer Dialogue

Organizing Committee: David Traum, Jens Edlund, Nigel Ward

Background
Natural Human dialogue (especially multiparty dialogue) contains many instances of more than one person talking at the same time. Types of overlap include

  • backchannels – in which one person is agreeing or indicating understanding of what another person is saying
  • completions – in which one person trys to help the other complete their utterance (while the first one is still speaking)
  • reactions – in which one party reacts to or follows up from the previous utterance before it is completed or when the first speaker is trying to say something more.
  • interruptions – in which one party tries to say something new or unrelated
  • floor battle – in which two speakers compete for the floor.
  • side-talk – talk simulaneous with the main speaker that is meant for someone other than the main speaker (Goffman’s byplay, crossplay, and sideplay)

These kinds of phenomena are pervasive in informal human interaction, and are highly correlated with real-time, engaged behavior. However current dialogue systems have only a very limited ability to engage in this kind of behavior. Workshop Goals
The goal of this workshop is to explore details of overlap in human dialogue and discuss how best to extend current dialogue systems and architectures to engage in more human-like overlap behavior. This workshop will immediately preceed the SEMDIAL 2011 Los Angelogue workshop at the Institute for Creative Technologies of the University of Southern California in Los Angeles.

Workshop Format
The format of the workshop will include talks on overlap in human dialogue, demos, talks and proposals about overlap in human-computer dialogue, plenary discussion sessions, and time permitting, small break-out groups analysizing overlap data and discussing policies for overlap in computer dialogue systems.

Submission
The workshop will not include archival publication of new papers. Details on the application procedure are on the submission page.

We invite several kinds of contributions to the workshop, including

  • presentation of data sets involving overlap in human-human or human-computer dialogue
  • Analysis of overlap in human dialogue, especially studies of the correlation of frequency and type of overlap to other factors such as
    • the language, culture, number and/or social roles of the participants
    • the task they are engaged in
    • other aspects of the context in which the dialogue takes place
    • the specific content and functions of the overlapping dialogue
    • cross-modality overlap
  • models of dialogue processing (understanding and generation), including overlap
  • demos of computer systems that can engage in meaningful, human-like overlap
  • discussion or break-out group topics
  • links to or other bibliographic info on publications concerning overlap

Contact traum@ict.usc.edu

Jacki Morie, Nonny de la Pena: “Sexual Harassment, Gender & Gender-swapping in SL”

Second Life, as a popular online virtual world, has emerged over the last several years as a fertile ground for researching various aspects of digitally-mediated interpersonal interaction. For example, griefing, a term used to broadly describe how players use functionality within the game to harass or virtually harm others, has been frequently studied. However, few attempts have been made to directly examine if and how online griefing of a sexual nature in virtual worlds overlaps with the concepts and effects of offline sexual harassment. Similarly, gender swapping with avatars has been considered, but how that might affect perceptions on sexual harassment has not been well studied, particularly in virtual spaces. In fact, there has been a relative lack of research on whether females find unwanted sexual encounters more harassing than males in massively multiplayer online role-playing games (MMORPGs) and specifically in Second Life. Additionally, given that virtual worlds offer a unique opportunity for gender-swapping, the obvious question arises: How do males playing with female avatars feel when faced with similar unwanted sexual encounters?

Thomas Parsons Presents at Arizona Psychology Training Consortium in Pheonix

At the University of Southern California’s Institute for Creative Technologies, we have developed an adaptive virtual environment for assessment and rehabilitation of neurocognitive and affective functioning. This project brings together a team of researchers to incorporate cutting edge neuropsychological and psychophysiological assessment into state of the art interactive/adaptive virtual Iraqi/Afghani scenarios (City, Checkpoint, HMMWV). Two primary goals define these virtual and adaptive environments: 1) a Virtual Reality Cognitive Performance Assessment Test (VRCPAT 1.0) that includes a normative database drawn from a battery of neuropsychological and psychophysiological measures for diagnostic assessment and treatment of Soldiers with affective disorders, brain injury, or neurocognitive deficits; and 2) a Virtual Reality for Cognitive Performance and Adaptive Treatment (VRCPAT 2.0) that develops an adaptive environment, in which data gleaned from the assessment module (VRCPAT 1.0) will be used for refined analysis, management, and rehabilitation of Soldiers who have suffered blast injuries (varying levels of traumatic brain injury).

Lixing Huang, Louis-Philippe Morency, Jonathan Gratch: “Virtual Rapport 2.0”

Rapport, the feeling of being “in sync” with your conversational partners, is argued to underlie many desirable social effects. By generating proper verbal and nonverbal behaviors, virtual humans have been seen to create rapport during interactions with human users. In this paper, we introduce our approach to creating rapport following Tickle-Degnen and Rosenberg’s three-factor (positivity, mutual attention and coordination) theory of rapport. By comparing with a previously published virtual agent, the Rapport Agent, we show that our virtual human predicts the timing of backchannel feedback and end-of-turn more precisely, performs more natural behaviors and, thereby creates much stronger feelings of rapport between users and virtual agents.

Jina Lee, Stacy Marsella: “Modeling Side Participants and Bystanders: the Importance of Being a Laugh Track”

Research in virtual agents has largely ignored the role and behavior of side participants and especially bystanders. Our view is that the behavior of these other participants is critical in multi-party interactions, especially in interactive drama. In this paper, we provide an analysis of nonverbal behaviors associated with these roles. We first review studies of interpersonal relationships and nonverbal behavior. From this review, we construct an analysis framework based on characters’ interpersonal relationships, conversational roles, and communicative acts. We then assess this framework by analyzing improv sessions of an old west scenario involving 4 characters. Informed by this analysis, we implemented a general model for participant and bystander behavior.

Jonathan Gratch, Philipp Kulms, Nicole Krämer, Sin-hwa Kang: “It’s in their eyes: A study on female and male virtual humans’ gaze”

Social psychological research demonstrates that the same behavior might lead to different evaluations depending on whether it is shown by a man or a woman. With a view to design decisions with regard to virtual humans it is relevant to test whether this pattern also applies to gendered virtual humans. In a 2×2 between subjects experiment we manipulated the Rapport Agent’s gaze behavior and its gender in order to test whether especially female agents are evaluated more negatively when they do not show gender specific immediacy behavior and avoid gazing at the interaction partner. Instead of this interaction effect we found two main effects: gaze avoidance was evaluated negatively and female agents were rated more positively than male agents.

Priti Aggarwal, Kevin Feeley, Fabrizio Morbini, Ron Artstein, Anton Leuski, David Traum, Julia Kim: “Interactive characters for cultural training of small military units”

CHAOS, the Combat Hunter Action and Observation Simulation, is an immersive simulation training environment which gives small military units the experience of interacting with local Afghan villagers during a patrol. It is a physical build-out of a housing compound in a mock Afghan village, with several life-size reactive and interactive animated Pashto-speaking virtual characters. The exercise requires an infantry squad to locate and interview a character named Omar, communicating through a live human interpreter and attending to proper protocol regarding Omar’s family. Character animation and behavior is based on extensive interviews with Afghan experts to provide a realistic setting of the intended locale. The system combines virtual human technology, story engineering, and physical set building to provide a compelling training environment that can handle a full squad, requiring trainees to integrate tasks such as working with an interpreter, dealing with non-English speakers from another culture, and assessing information and disposition to make decisions in a mission context.

Dusan Jan, Eric Chance, Dinesh Rajpurohit, David DeVault, Anton Leuski, Jacki Morie, David Traum: “Checkpoint Exercise: Training with Virtual Actors in Virtual Worlds”

We have implemented a checkpoint exercise in Second Life where the user interacts with several computer avatars in a team based activity. We describe the experience and the implementation of our solution and show some evaluation results.

Zhiyang Wang, Jina Lee, Stacy Marsella: “Towards More Comprehensive Listening Behavior: Beyond the Bobble Head”

Realizing effective listening behavior in virtual humans has become a key area of research, especially as research has sought to realize more complex social scenarios involving multiple participants and bystanders. A human listener’s nonverbal behavior is conditioned by a variety of factors, from current speaker’s behavior to the listener’s role and desire to participate in the conversation and unfolding comprehension of the speaker. Similarly, we seek to create virtual humans able to provide feedback based on their participatory goals and their partial understanding of, and reaction to, the relevance of what the speaker is saying as the speaker speaks. Based on a survey of existing psychological literature as well as recent technological advances in recognition and partial understanding of natural language, we describe a model of how to integrate these factors into a virtual human that behaves consistently with these goals.We then discuss how the model is implemented into a virtual human architecture and present an evaluation of behaviors used in the model.

Antonio Roque, Mark Core, Dusan Jan, David Traum: “Using virtual tour behavior to build dialogue models for training review”

We develop an intelligent agent that builds a user model of a learner during a tour of a virtual world. The user model is based on the learner’s answers to questions during the tour. A dialogue model for a simulated instructor is tailored to the individual learner based upon this user model. We describe an evaluation to track system accuracy and user perceptions.

Sin-hwa Kang, Candy Sidner, Jonathan Gratch, Ron Artstein, Lixing Huang, Louis-Philippe Morency: “Modeling Nonverbal Behavior of a Virtual Counselor during Intimate Self-Disclosure”

Humans often share personal information with others in order to create social connec-tions. Sharing personal information is especially important in counseling interactions [2]. Research studying the relationship between intimate self-disclosure and human behavior critically informs the development of virtual agents that create rapport with human interaction partners. One significant example of this application is using virtu-al agents as counselors in psychotherapeutic situations. The capability of expressing different intimacy levels is key to a successful virtual counselor to reciprocally induce disclosure in clients. Nonverbal behavior is considered critical for indicating intimacy and is important when designing a social virtual agent such as a counselor. One key research question is how to properly express intimate self-disclosure. In this study, our main goal is to find what types of interviewees’ nonverbal behavior is associated with different intimacy levels of verbal self-disclosure. Thus, we investigated hu-mans’ nonverbal behavior associated to self-disclosure during interview setting (with intimate topics).

Chung-Cheng Chiu, Stacy Marsella: “How to train your avatar”

The ability to gesture is key to realizing virtual characters that can engage in face-to-face interaction with people. Many applications take an approach of predefining possible utterances of a virtual character and building all the gesture animations needed for those utterances. We can save effort on building a virtual human if we can construct a general gesture controller that will generate behavior for novel utterances. Because the dynamics of human gestures are related to the prosody of speech, in this work we propose a model to generate gestures based on prosody. We then assess the naturalness of the animations by comparing them against human gestures. The evaluation results were promising, human judgments show no significant difference between our generated gestures and human gestures and the generated gestures were judged as significantly better than real human gestures from a different utterance.

Morteza Dehghani: “Cultural Frame-Switching using Accented Spoken Language by a Virtual Character”

In this paper, we examine whether embodied conversational agents can be used to implement socio-cultural markers. We investigate whether the accent of a virtual character, as a marker for culture, can cause cultural frame-shifts in individuals. We report an experiment, per- formed among bicultural and monocultural individuals, in which we test the above hypothesis. Our results show that a virtual agent can have a socio-cultural effect on people’s cognition. This work makes unique contributions to the design and evaluation of intelligent virtual agents as well as the theoretical psychological literature

Peter Khooshabeh, Sudeep Gandhe, Cade McCall, Jonathan Gratch, James Blascovich, David Traum: “The effects of virtual agent humor and gaze behavior on human-virtual agent proxemics”

We study whether a virtual agent that delivers humor through verbal behavior can affect an individual’s proxemic behavior towards the agent. Participants interacted with a virtual agent through natural language and, in a separate task, performed an embodied interpersonal interaction task in a virtual environment. The study used minimum distance as the dependent measure. Humor generated by the virtual agent through a text chat did not have any significant effects on the proxemic task. This is likely due to the experimental constraint of only allowing participants to interact with a disembodied agent through a textual chat dialogue.

Jennifer Klatt, Stacy Marsella, NIcole Kramer: “Negotiations in the Context of AIDS Prevention: An Agent-Based Model Using Theory of Mind”

For the purpose of an AIDS prevention game, a model was developed that focuses on training safe sex negotiations. Non-player characters in the game are socially intelligent agents that are equipped with a Theory of Mind that allows them to reason about the mental processes and behavior of others. The underlying model for the negotiation about safe sex between player and agent was implemented in multi-agent simulation software. It consists of two agents who have different goals of either safe or unsafe sex, actions to achieve these goals, and the wish to come to an agreement. The model was evaluated for the agent-agent conversation to test the basic functioning.

Herwin van Welbergen, Yuyu Xu, Marcus Thiebaux, Andrew W. Feng, Jingqiao Fu, Dennis Reidsma, Ari Shapiro: “Demonstrating and Testing the Compliance of BML Realizers”

BML realizers are complex software modules that implement a standardized interface, the BML speci cation language, to steer the behavior of a virtual human. We aim to promote and test the compliance of realizers that implement this interface. To this end we contribute a corpus of example BML scripts and a tool called RealizerTester that can be used to formally test and maintain adherence of realizers to the BML standard. The standardized interface of realizers allowed us to implement RealizerTester as an automatic testing framework that can test any realizer. RealizerTester can 1) help in maintaining the stability and extensibility that is crucial for realizers and 2) contribute to the formalization of the emerging BML standard, both by providing test scripts and a formal description of their constraints and by identifying and resolving execution inconsistencies between realizers. We illustrate the testing practices used in the development of two realizers and demonstrate how RealizerTester is integrated with these practices. The scripts in the example corpus were executed on both realizers. This resulted in a video corpus that demonstrates the semantic equivalences and differeces in execution of BMLscripts by the two realizers.

USC Institute for Creative Technologies Receives $135 Million Contract Extension From U.S. Army

Award allows for continued university-based research and development of interactive media for military training, health therapies and more.

Sept.1, 2011 – The Department of Defense announced this week that the University of Southern California Institute for Creative Technologies (ICT) has been given a contract extension through 2014. The extension allows the Army to fund up to an additional $135 million dollars of research and prototype development over the next three years.

The Army looks to the institute to provide a vision for the future of training. The contract extension is an indication that ICT helps today’s military meet its goal of staying on the cutting-edge. The institutes’ research and development of effective training technologies reach larger groups of soldiers, reducing the need for expensive live simulations and human role-players.

“ICT brings USC’s computer scientists together with artists, writers and cinematographers, creating compelling and immersive training systems. It exemplifies how USC is able to bring disciplines together in highly creative research,” said Randolph Hall, USC’s vice president of research.

“We are thankful for the opportunity to continue to serve the men and women in the U.S. Army and all the military services,” said Randall W. Hill Jr., ICT’s executive director. “This extension is a strong endorsement of the institute’s success in developing immersive technologies that have led to effective prototypes for training, leader development and physical rehabilitation. And the impact has gone beyond the military to society at large.”

ICT was founded with an initial five-year contract in 1999, when USC was selected as the site for a new research institute with a mandate to bring film and game industry artists together with computer and social scientists to develop engaging simulations that would transform training and improve performance across a wide variety of subject areas related to critical thinking and decision-making.

“ICT will be a joint effort of the Army, the entertainment industry and academe – an innovative team to advance dazzling new media and ultimately benefit training and education for everyone in America,” said then-Secretary of the United States Army, Louis Caldera, at the time.

A second five-year contract was awarded in 2004 and extended again for five years in 2009. The latter extension was implemented in two phases, one through 2011 and the new one just signed by the Pentagon.

ICT estimates that over 75,000 soldiers have trained with technologies from the institute, which specializes in using virtual humans and storytelling to enhance learning. ICT has developed virtual reality systems that offer accessible care for troops suffering from post-traumatic stress and brain injuries that are being used and evaluated at close to 60 sites across the country, including Walter Reed Army Medical Center. Interactive virtual human role-players now provide practice in how to lead, negotiate and diagnose and have been incorporated into portable video games, mixed reality-training installations at U.S. bases and the curriculum at the USC School of Social Work as part of their specialized training in military social work.

Additional projects include a virtual human museum exhibit for science education, 3-D displays for video teleconferencing and game-based based systems for physical therapy.

The institute is also a recognized leader in several areas of academic research. ICT scientists serve of the faculty of the computer science department of the USC Viterbi School of Engineering, the Interactive Media Division of USC School for Cinematic Arts and the USC Davis School of Gerontology. They have been recognized as the top in their fields in graphics and virtual humans research, cognitive architectures, contributions to the AI community, emotion modeling and social simulation. One even received an Academy Award. In addition to their research at ICT, they teach courses on the USC campus and publish regularly in leading international journals and present at major conferences related to their subject areas.

“This combination of scientists and storytellers is what makes ICT so unique,” said John Hart, program manager at the U.S. Army’s Simulation and Training Technology Center, which oversees ICT’s Army contract. “We look forward to continuing our successful collaboration with USC.”

Belinda Lange, Chien-Yen Chang, Evan Suma, Bradley Newman, Skip Rizzo, Mark Bolas: “Development and Evaluation of Low Cost Game-Based Balance Rehabilitation Tool Using the Microsoft Kinect Sensor”

The use of the commercial video games as rehabilitation tools, such as the Nintendo® WiiFit™, has recently gained much interest in the physical therapy arena. Motion tracking controllers such as the Nintendo® Wiimote are not sensitive enough to accurately measure performance in all components of balance. Additionally, users can figure out how to “cheat” inaccurate trackers by performing minimal movement (e.g. wrist twisting a Wiimote instead of a full arm swing). Physical rehabilitation requires accurate and appropriate tracking and feedback of performance. To this end, we are developing applications that leverage recent advances in commercial video game technology to provide full-body control of animated virtual characters. A key component of our approach is the use of newly available low cost depth sensing camera technology that provides markerless full-body tracking on a conventional PC. This allows the user to puppet a virtual character on screen that directly represents their movements and pose in the real world. Using low cost depth sensing cameras (PrimeSense and Microsoft Kinect) and a flexible software framework, individuals can interact with game-based rehabilitation tools that are tailored to their individual therapy goals. Not only does our approach sense full-body motion with six degrees-of-freedom, but it does so without encumbering the user with tracking devices or markers. This allows more natural and intuitive interaction, without having to alter natural motor movements to accommodate the tracking hardware. The aim of this research was to develop and assess an interactive game-based rehabilitation tool for balance training of adults with neurological injury.

Belinda Lange Presents at the 33rd Annual International IEEE EMBS Conference

The use of the commercial video games as rehabilitation tools, such as the Nintendo WiiFitT, has recently gained much interest in the physical therapy arena. Motion tracking controllers such as the Nintendo Wiimote are not sensitive enough to accurately measure performance in all components of balance. Additionally, users can figure out how to “cheat” inaccurate trackers by performing minimal movement (e.g. wrist twisting a Wiimote instead of a full arm swing). Physical rehabilitation requires accurate and appropriate tracking and feedback of performance. To this end, we are developing applications that leverage recent advances in commercial video game technology to provide full-body control of animated virtual characters. A key component of our approach is the use of newly available low cost depth sensing camera technology that provides markerless full-body tracking on a conventional PC. This allows the user to puppet a virtual character on screen that directly represents their movements and pose in the real world. Using low cost depth sensing cameras (PrimeSense and Microsoft Kinect) and a flexible software framework, individuals can interact with game-based rehabilitation tools that are tailored to their individual therapy goals. Not only does our approach sense full-body motion with six degrees-of-freedom, but it does so without encumbering the user with tracking devices or markers. This allows more natural and intuitive interaction, without having to alter natural motor movements to accommodate the tracking hardware. The aim of this research was to develop and assess an interactive game-based rehabilitation tool for balance training of adults with neurological injury.

David DeVault, Kenji Sagae, David Traum: “Detecting the Status of a Predictive Incremental Speech Understanding Model for Real-Time Decision-Making in a Spoken Dialogue System”

We explore the potential for a responsive spoken dialogue system to use the real-time status of an incremental speech understanding model to guide its incremental decision-making about how to respond to a user utterance that is still in progress. Spoken dialogue systems have a range of potentially useful real-time response options as a user is speaking, such as providing acknowledgments or backchannels, interrupting the user to ask a clarification question or to initiate the system’s response, or even completing the user’s utterance at appropriate moments. However, implementing such incremental response capabilities seems to require that a system be able to assess its own level of understanding incrementally, so that an appropriate response can be selected at each moment. In this paper, we use a data-driven classification approach to explore the trade-offs that a virtual human dialogue system faces in reliably identifying how its understanding is progressing during a user utterance.

Kallirroi Georgila: “Reinforcement Learning of Argumentation Dialogue Policies in Negotiation”

We build dialogue system policies for negotiation, and in particular for argumentation. These dialogue policies are designed for negotiation against users of different cultural norms (individualists, collectivists, and altruists). In order to learn these policies we build simulated users (SUs), i.e. models that simulate the behavior of real users, and use reinforcement learning (RL). The SUs are trained on a spoken dialogue corpus in a negotiation domain, and then tweaked towards a particular cultural norm using hand-crafted rules. We evaluate the learned policies in a simulation setting. Our results are consistent with our SUs, in other words, the policies learn what they are designed to learn, which shows that RL is a promising technique for learning policies in domains, such as argumentation, that are more complex than standard slot-filling applications.

Morteza Dehghani: “Cultural Frame-Switching using Accented Spoken Language by a Virtual Character”

In this paper, we examine whether embodied conversational agents can be used to implement socio-cultural markers. We investigate whether the accent of a virtual character, as a marker for culture, can cause cultural frame-shifts in individuals. We report an experiment, per- formed among bicultural and monocultural individuals, in which we test the above hypothesis. Our results show that a virtual agent can have a socio-cultural e ect on people’s cognition. This work makes unique con- tributions to the design and evaluation of intelligent virtual agents as well as the theoretical psychological literature.

Paul Rosenbloom Wins Best Idea Award at 2011 Conference on Artificial General Intelligence

Paul Rosenbloom, professor of computer science at the USC Viterbi School of Engineering and project leader at the USC Institute for Creative Technologies (ICT), received the Kurzweil Prize for Best Artificial General Intelligence Idea at the fourth conference on artificial general intelligence held in August.

The prize recognized Rosenbloom’s pioneering work toward building computer systems that can behave like people in how they make decisions and solve problems.

Artificial general intelligence, or AGI, refers to the design of systems that can emulate full-range human intelligence as opposed to artificial intelligence systems that focus on modeling narrow or specific functions like generating speech, acquiring language or planning actions.

“Significant progress has been made in many individual areas since the founding of AI, but such progress by itself doesn’t yield human level intelligence,” Rosenbloom said. “AGI represents a return to this original vision of AI.”

Rosenbloom leads such an effort at ICT, where he is building the next generation of virtual human architecture – sort of a brain for computer-driven characters – that should enable them to behave appropriately when combined with the proper knowledge and skills. ICT is a leader in research and development of virtual humans, and it is hoped that Rosenbloom’s new architecture – when combined with other developments at ICT and elsewhere – will lead to more human-like systems.

Potential applications could include more intelligent virtual humans, robots and agents – virtual negotiation partners, for example, that learn from their mistakes, react to current situations and alter their behaviors depending on past and present interactions.

“The goal is to integrate all the mechanisms for thought, language, speech and motor control into a single system that can learn from experience,” he said. “Usually when you add functionality in architectures, you add complexity. My work is trying to simplify the process with an architecture that combines elegance with generality.”

Rosenbloom’s winning idea is explained in his paper “From Memory to Problem Solving: Mechanism Reuse in a Graphical Cognitive Architecture,” which he presented at the conference on general artificial intelligence.

The award is sponsored by the group KurzweilAI, which is devoted to promoting work of and inspired by noted American futurist Raymond Kurzweil. In awarding the $1,000 prize to Rosenbloom, the committee cited the work as representing “a very interesting and valuable research direction” and “a wonderful example” of the type of work that explores “the extent any single mechanism can be used to achieve all the capabilities needed for human level AGI.”

Rosenbloom began his work toward virtual human architectures in the late 1970s and early 1980s as a Ph.D. student at Carnegie Mellon University. He continued that work as a professor at Stanford University. He joined USC in 1987 and wrote the initial proposal for bringing ICT to USC in 1999.

During his tenure at USC, he served in various capacities at the university’s Information Sciences Institute. He currently is writing On Computing: A Relational Approach to the Fourth Great Domain of Science, a book to be published by MIT Press.

Read the story on USC News.

Wan-Chun Ma, Graham Fyffe, Paul Debevec: “Optimized Local Blendshape Mapping for Facial Motion Retargeting”

One of the popular methods for facial motion retargeting is local blendshape mapping [Pighin and Lewis 2006], where each local facial region is controlled by a tracked feature (for example, a vertex in motion capture data). To map a target motion input onto blendshapes, a pose set is chosen for each facial region with minimal retargeting error. However, since the best pose set for each region is chosen independently, the solution likely has unorganized pose sets across the face regions, as shown in Figure 1(b). Therefore, even though every pose set matches the local features, the retargeting result is not guaranteed to be spatially smooth. In addition, previous methods ignored temporal coherence which is key for jitter-free results.

Jonathan Ito, Stacy Marsella: “Contextually-Based Utility: An Appraisal-Based Approach at Modeling Framing and Decisions”

Creating accurate computational models of human decision making is a vital step towards the realization of socially intelligent systems capable of both predicting and simulating human behavior. In modeling human decision making, a key factor is the psychological phenomenon known as “framing”, in which the preferences of a decision maker change in response to contextual changes in decision problems. Existing approaches treat framing as a one-dimensional contextual influence based on the perception of outcomes as either gains or losses. However, empirical studies have shown that framing effects are much more multifaceted than one-dimensional views of framing suggest. To address this limitation, we propose an integrative approach to modeling framing which combines the psychological principles of cognitive appraisal theories and decision-theoretic notions of utility and probability. We show that this approach allows for both the identification and computation of the salient contextual factors in a decision as well as modeling how they ultimately affect the decision process. Furthermore, we show that our multidimensional, appraisal-based approach can account for framing effects identified in the empirical literature which cannot be addressed by one-dimensional theories, thereby promising more accurate models of human behavior.

Soldiers Magazine Features ICT

Soldiers Magazine featured ICT’s efforts combining storytelling and technology to improve Army training. The result has been multiple breakthroughs in virtual reality and emotionally engaging training simulations, movies and games focused on helping servicemembers, said ICT Executive Director Dr. Randall Hill Jr., in the story.

From treating post-traumatic stress disorder, instilling battlefield ethics and teaching improvised explosive device recognition, to addressing myriad other issues for the military, the ICT is making its mark in advancing the Army’s capabilities, the story noted.

Among the projects mentioned in the story, is ELITE (formerly known as VOLT), a new leadership development tool that uses virtual humans for interpersonal skills training.

The program introduces the most advanced virtual human graphics and artificial intelligence technologies to junior officers to give them evidence-based educational techniques. The virtual human to real human interaction gives young leaders practice resolving authentic, complex problems, states the article.

“A lot of leadership is having interpersonal communication skills,” said Kim LeMasters, ICT’s creative director who is working with scientists and Army experts to make sure the program’s scenarios are relevant. “We asked, ‘How do you make it where people can sit down and practice them?’”

Executive Director Randall Hill adds that it’s simple to use the institute’s abilities and he hopes more leaders will.

“We’re sitting in an interesting place in history,” he said. “We can give Soldiers the ability to prepare in ways that weren’t even possible before.”

Read the full story here.

Andrew Gordon, Cosmin Bejan, Kenji Sagae: “Commonsense Causal Reasoning Using Millions of Personal Stories”

The personal stories that people write in their Internet weblogs include a substantial amount of information about the causal relationships between everyday events. In this paper we describe our efforts to use millions of these stories for automated commonsense causal reasoning. Casting the commonsense causal reasoning problem as a Choice of Plausible Alternatives, we describe four experiments that compare various statistical and information retrieval approaches to exploit causal information in story corpora. The top performing system in these experiments uses a simple co-occurrence statistic between words in the causal antecedent and consequent, calculated as the Pointwise Mutual Information between words in a corpus of millions of personal stories.

Predictive Models of Human Communication Dynamics

In the study of human face-to-face communication, the patterning of interlocutor actions and interactions, moment-by-moment, is a matter of great scientific interest; and predictive models of such behavior are needed in order to build systems that can understand and interact with people in more natural ways.

We organized this workshop in order to bring together researchers with diverse backgrounds and perspectives to share knowledge, advance the field, and roadmap the future.

Cyrus Wilson, Oleg Alexander, Borom Tunwattanapong, Pieter Peers, Abhijeet Ghosh, Jay Busch, Arno Hartholt, Paul Debevec: “Facial Cartography: Interactive Scan Correspondence”

We present a semi-automatic technique for computing surface correspondences between 3D facial scans in different expressions, such that scan data can be mapped into a common domain for facial animation. The technique can accurately correspond high-resolution scans of widely differing expressions – without requiring intermediate pose sequences – such that they can be used, together with reflectance maps, to create high-quality blendshape-based facial animation. We optimize correspondences through a combination of Image, Shape, and Internal forces, as well as Directable forces to allow a user to interactively guide and refine the solution. Key to our method is a novel representation, called an Active Visage, that balances the advantages of both deformable templates and correspondence computation in a 2D canonical domain. We show that our semi-automatic technique achieves more robust results than automated correspondence alone, and is more precise than is practical with unaided manual input.

Paul S. Rosenbloom: “From Memory to Problem Solving: Mechanism Reuse in a Graphical Cognitive Architecture”

This article describes the extension of a memory architecture that is implemented via graphical models to include core aspects of problem solving. By extensive reuse of the general graphical mechanisms originally developed to support memory, this demonstrates how a theoretically elegant implementation level can enable increasingly broad architectures without compromising overall simplicity and uniformity. In the process, it bolsters the potential of such an approach for developing the more complete architectures that will ultimately be necessary to support autonomous general intelligence.

Jonathan Gratch: “Rise of the Machines”

Advances in computational methods for recognizing, modeling, and shaping human emotions Abstract: The last decade has seen the emergence of the field of Affective Computing: a new partnership between the computational and affective sciences. New tools often transform science, opening up new approaches and new questions. By developing computational techniques that recognize, model and simulate human emotion, affective computing is opening up new possibilities for measurement and theory. But further, technology itself is being transformed by theoretical and empirical findings from affective science. In this symposium, we will present a broad sampling of research that illustrates some of this potential for synergy. We begin with a discussion of methods for recognizing human emotional cues, including studies using these methods to explore psychological phenomena. Next we discuss computational models of the cognitive antecedents and consequences of emotion Work in computational models of emotion impacts research in human emotion by transforming how theories are formulated and evaluated, for example by concretizing concepts in the theory. Next we present techniques for digitally synthesizing emotional behaviors, an important methodological tool for exploring the impact of such cues on human behavior. Finally, we will present empirical evidence that that people respond emotionally to computational artifacts, and that emotion theory can give insight into explaining these effects. Together, these four talks illustrate how new technology is both transforming and being transformed by affective science.

Peter Khooshabeh, Jonathan Gratch, Lixing Huang, J. Tao: “Does culture affect the perception of emotion in virtual faces?”

Previous research, which has used images of real human faces and mostly from the same facial expression database [Matsumoto and Ekman 1988], has shown that individuals perceive emotions universally across cultures. We conducted an experiment to determine whether culture affects the perception of emotions rendered on virtual faces. Specifically, we test the holistic perception hypothesis that individuals from collectivist cultures, such as East Asians, visually sample information from central regions of the face (near the top of the nose by the eyes), as opposed to sampling from specific features of the face. If the holistic perception hypothesis is true, then individuals will confuse emotional facial expressions that are different in terms of the shape of the mouth facial feature. Our stimuli were computer generated using a face graphical rendering tool, which affords a high level of experimental control for perception researchers.

Celso de Melo, Peter Carnevale, Jonathan Gratch: “Reverse Appraisal: Inferring from Emotion Displays who is the Cooperator and the Competitor in a Social Dilemma”

This paper explores whether and how facial displays of emotion can impact emergence of cooperation in a social dilemma. Three experiments are described where participants play the iterated prisoner’s dilemma with (computer) players that display emotion. Experiment 1 compares a cooperative player, whose displays reflect a goal of mutual cooperation, with a control player that shows no emotion. Experiment 2 compares a competitive player, whose displays reflect a goal of getting more points than the participant, and the control player. Experiment 3 compares the cooperative and competitive players. Results show that people: cooperate more with the cooperative than the control player (Experiment 1); do not cooperate differently with the competitive and control players (Experiment 2); and, cooperate more with the cooperative than the competitive player, when they play the latter first (Experiment 3). In line with a social functions view of emotion, we argue people infer, from emotion displays, the other player’s propensity to cooperate by reversing the emotion appraisal process. Post-game surveys show that people interpret the emotion displays according to appraisal variables (desirability, responsibility and controllability) in ways that are consistent with predictions from appraisal theories of emotion.

Morteza Dehghani, Jonathan Gratch, Sonya Sachdeva, Kenji Sagae: “Analyzing Conservative and Liberal Blogs Related to the Construction of the Ground Zero Mosque”

The issue of the „Ground Zero Mosque‟ has been one of the most controversial political issues in US politics in the last several years. Using two different statistical text-analysis techniques, we analyze conservative and liberal blog posts, related to the construction of this Muslim community center and the debates surrounding the issue. In the first experiment, we use a machine learning technique to automatically classify the blogs according to which group wrote them. We also examine the distinctive features that make these blogs liberal or conservative. In the second experiment, by examining posts in consecutive time blocks, we show that there was a significant increase over time in affective processing, and in anger, especially for conservatives. Overall, our results show that there are significant differences in the use of various linguistic features between liberals and conservatives, highlighting the differences between the ideologies and the moral frameworks of the two groups.

Ron Artstein, Michael Rushforth, Sudeep Gandhe, David Traum, Aram Donigian: “Limits of simple dialogue acts for tactical questioning dialogues”

A set of dialogue acts, generated automatically by applying a dialogue act scheme to a domain representation designed for easy scenario authoring, covers approximately 72%–76% of user utterances spoken in live interaction with a tactical questioning simulation trainer. The domain is represented as facts of the form and conversational actions of the form . User utterances from the corpus that fall outside the scope of the scheme include questions about temporal relations, relations between facts and relations between objects, questions about reason and evidence, assertions by the user, conditional offers, attempts to set the topic of conversation, and compound utterances. These utterance types constitute the limits of the simple dialogue act scheme.

ICT Research Earns Best Paper Award at the Human-Computer Interaction International Conference

A paper, Emotions and Telerehabilitation: Pilot Clinical Trials for Virtual Telerehabilitation Application Using Haptic Device and its Impact on Post Stroke Patients’ Mood and Motivation, by former ICT-er Luke Yeh, along with Belinda Lange, Skip Rizzo and others was awarded the best paper in the Virtual and Mixed Reality Conference, a theme-area conference affiliated with the 2011 Human-Computer Interaction International Conference in Orlando, Florida. Congrats to the ICT authors and their collaborators.

Visit the conference award website.

NPR Kinect Story Features Mark Bolas and Belinda Lange’s Jewel Mine Game

story on NPR’s All Things Considered covered the many ways researchers and others are modifying the Microsoft Kinect. For the story reporter Alex Schmidt spoke to Mark Bolas, ICT’s associate director for mixed reality and an associate professor in the Interactive Media Division at the USC School of Cinematic Arts, about the trend.

“I remember getting this wrench with my father when I was 13 or 14 years old,” said Bolas. “And then with it, I could start working on my bicycle. And I got into motorcycles and all these things that I could build. The wrenches of today aren’t physical. They’re the software wrenches.”

Schmidt also visited a USC rehabilitation research clinic with Belinda Lange who leads the motor rehab group in ICT’s Medical Virtual Reality Lab. In the story Lange takes a patient through Jewel Mine, the Kinect-based motor rehab tool she developed, which gets a postive review from patient Stacey Holmes.

“It causes you to try things at a pace and a precision that you wouldn’t otherwise try to do,” he said.

Listen to the story here.

Read the transcript here.

Jacki Morie, Eric Chance, John Galen Buckwalter: “Report on a Preliminary Study Using Breath Control and a Virtual Jogging Scenario as Biofeedback for Resilience Training”

Alternative methods of treating psychological stress are needed to treat some veterans of recent military conflicts. The use of virtual world technologies is one possible platform for treatment that is being explored by the “Coming Home” project at the University of Southern California’s Institute for Creative Technologies (ICT). One of the novel ways ICT is attempting to mitigate stress via virtual worlds is with a virtual jogging scenario, where the movement of an avatar is controlled via rhythmic breathing into a standard microphone. We present results from a preliminary study of 27 participants that measured the mood and arousal effects produced by engaging in this scenario.

Kallirroi Georgila, David Traum: “Learning Culture-Specific Dialogue Models from Non Culture-Specific Data”

We build culture-specific dialogue policies of virtual humans for negotiation and in particular for argumentation and persuasion. In order to do that we use a corpus of non-culture specific dialogues and we build simulated users (SUs), i.e. models that simulate the behavior of real users. Then using these SUs and Reinforcement Learning (RL) we learn negotiation dialogue policies. Furthermore, we use research findings about specific cultures in order to tweak both the SUs and the reward functions used in RL towards a particular culture. We evaluate the learned policies in a simulation setting. Our results are consistent with our SU manipulations and RL reward functions.

Jacki Morie: “Body Buddies: Social Signaling through Puppeteering

While virtual worlds have evolved to provide a good medium for social communication, they are very primitive in their social and affective communication design. The social communication methods within these worlds have progressed from early text-based social worlds, e.g. MUDS (multi-user dungeons) to 3D graphical interfaces with avatar control, such as Second Life. Current communication methods include triggering gestures by typed commands, and/or selecting a gesture by name through the user interface. There are no agreed-upon standards for organizing such gestures or interfaces. In this paper, we address this problem by discussing a Unity-based avatar pupeteering prototype we developed called Body Buddies. Body Buddies sits on top of the communication program Skype, and provides additional modalities for social signaling through avatar pupeteering. Additionally, we discuss results from an exploratory study we conducted to investigate how people use the interface. We also outline steps to continuously develop and evolve Body Buddies.

Belinda Lange and Skip Rizzo: “Virtual and Mixed Reality”

ICT’s Belinda Lange and Skip Rizzo with present their paper, Virtual and Mixed Reality, at the 14th International Conference on Human-Computer Interaction in Orlando, Florida.

Kotaku calls Mother Nature, A Kinect Game Co-Developed at ICT, Possibly the Best of Them All

The video game blog Kotaku recently featured Diane Tucker’s Kincect-based game, Mother Nature, under the headline, The Best Kinect Game You Never Heard of (And Maybe the Best of them All).  The post includes a video demonstration of the game and calls the controls, “smartly elegant”.

Mother Nature was deliberately designed to exploit the positive emotions linked to certain gestures to enhance players’ experience. Project Lead Diane Tucker uncovered the research revealing the interrelationships among movement, emotion and story and developed early levels of the game using OpenNI working at Mark Bolas’s MxR Lab in the Institute for Creative Technologies at USC. The ideas for the game were first generated and pitched as part of the Project Natal development course Microsoft and USC offered in Spring 2010. The prototype developed for that course helped Mother Nature win a spot in USC’s competitive Advanced Game Project, administered collaboratively by the USC School of Cinematic Arts and the USC Viterbi School of Engineering, through which a student team of designers, artists and engineers – headed by lead engineer Thai Phan—designed and built the game.

Read the post and watch the video

Visit the Mother Nature site

Morteza Dehghani: “Investigating and Modeling the Role of Cultural Narratives in Moral Decision-Making”

Sacred values are different from secular values in that they are often associated with violations of the cost-benefit logic of rational choice models. Understanding and modeling the impacts of sacred values on decision making is especially important in resolving intergroup conflicts and negotiations. While a secular value can easily be substituted with another value, tradeoffs involving a sacred value result in strong negative emotions and moral outrage. In this talk, I first examine Iran’s stance on its national nuclear program, using it as an indicator of how sacred values can emerge from sacred rhetoric. Next, I examine whether the principles of analogical retrieval and mapping govern the processes by which cultural and sacred narratives are applied. In particular, I examine how analogical accessibility and alignability influence the use of canonical moral narratives. To understand and model this process computationally, I have developed MoralDM as a model of recognition-based moral decision-making. This model relies on a combination of first-principles reasoning and analogical reasoning to model the recognition- based mode of decision making. Overall, I argue that understanding sacred values and the processes by which they emerge are vital for understanding and modeling decision-making in cultural contexts.

H. Chad Lane: “An Introduction to the Learning Sciences”

In this talk I will review the basic principles of human learning and education through the lens of American popular media. This is an invited address to be delivered at the evening banquet address for the AIED society.

H. Chad Lane, Dan Noren, Daniel Auerbach, Mike Birch, William Swartout: “Intelligent Tutoring Goes to the Museum in the Big City: A Pedagogical Agent for Informal Science Education”

In this paper, we describe Coach Mike, a virtual staff member at the Boston Museum of Science that seeks to help visitors at Robot Park, an interactive exhibit for computer programming. By tracking visitor interactions and through the use of animation, gestures, and synthesized speech, Coach Mike provides several forms of support that seek to improve the experiences of museum visitors. These include orientation tactics, exploration support, and problem solving guidance. Additional tactics use encouragement and humor to entice visitors to stay more deeply engaged. Preliminary analysis of interaction logs suggest that visitors can follow Coach Mike’s guidance and may be less prone to immediate disengagement, but further study is needed.

ICT Researchers Win Best Paper Award at IEEE Procams 2011

Congratulations to Joel Jurik and co-authors Andrew Jones, Mark Bolas and Paul Debevec on receiving the best paper award for their paper, “Prototyping a Light Field Display Involving Direct Observation of a Video Projector Array” at the IEEE Procams 2011 Conference in Colorado Springs, Colorado.

Read the paper.

ICT’s Sin-hwa Kang Receives New Investigator Award at CyberTherapy & CyberPsychology Conference

ICT research associate Sin-hwa Kang was given the New Investigator Award at the 16th Annual CyberTherapy & CyberPsychology Conference in Gatineau, Canada. The prize honors a researcher new to the field of cybertherapy who makes a presentation of outstanding research quality at the Cybertherapy conference. Kang’s winning presentation, People Like Virtual Counselors that Highly Disclose about Themselves, described findings from research designed to explore the effect of self-disclosure between virtual human counselors (interviewers) and human users (interviewees) on users’ social responses in counseling sessions. The results demonstrated that users reported more co-presence and social attraction to virtual humans who disclosed highly intimate information about themselves than when compared to other virtual humans who disclosed less intimate or no information about themselves. Jon Gratch, ICT’s associate director for virtual humans research co-authored the paper.

Read the paper.

Bill Swartout: “Let Me Tell You a Story”

Stories are a fundamental means through which we learn and share experiences. At the USC Institute for Creative Technologies, stories are the backbone behind a wide variety of simulated experiences that we create to educate and train skills in leadership, negotiation, cultural awareness and negotiation. These experiences range from filmed dramatizations of case studies to fully interactive experiences with virtual humans set in virtual worlds. But authoring these stories and constructing the knowledge structures, such as language and dialogue models, needed to support their use in our simulations is a highly labor intensive task. Additionally, it is important to ensure that the stories are authentic and reflect current realities, which may require access to outside subject matter experts if the stories are developed in-house. In this talk I will present work we have been pursuing at the ICT to use data-driven methods to gather stories from the web and other sources and incorporate them into simulations. This work consists of several efforts which taken together, hold the promise of providing more authentic experiences while at the same time reducing the modeling and knowledge base construction effort required to support them.

Belinda Lange, Skip Rizzo, Tamar Weiss: “Microsoft Kinect/Primesense Sensing Systems for Virtual Rehabilitation”

One of the exciting new developments in the field of Virtual Rehabilitation involves the release of the new Xbox Kinect system by Microsoft. This revolutionary game platform uses an infrared “depth-sensing” camera (produced by an Israeli company, Primesense) to capture users’ full body movement in 3D space for interaction within game activities. This system does not require the user to hold an interface device or move on a pad as the source of interaction within the game. Instead, the user’s body is the game controller operating in 3D space and multiple users can be tracked in this fashion for both cooperative and competitive interactive activities. This technology is a significant advance over previously available 2D video capture systems. Such low cost sensing systems for tracking human movement could revolutionize how virtual rehabilitation will be done in the future. Following a stroke, brain injury or other form of neurological disorder, a patient using this system can naturally interact with game content as part of their physical, occupational and cognitive therapy and they may be more motivated to do therapy when it is embedded in a game context. An attractive feature is the fact that while the Primesense camera provides the tracking functionality for the Kinect, it will soon be available as a low-cost stand-alone USB depth-sensing camera. This option will allow homegrown developers and researchers to produce game software and content that is specifically designed to promote rehabilitation, and perhaps “exergaming” activities beyond what the Xbox console games may offer. Researchers have thus far integrated the MS Kinect/Primesense movement tracking system with custom-built rehab games and with associated software that allows it to drive any PC-based computer game by emulating standard mouse and keyboard commands, all based on the designated physical activity of the user. This will provide a new dimension for interactive rehabilitation and exergaming in many ways by opening up a multitude of existing game content for full body interaction. These advances could stand to promote healthcare research and application development that could be widely disseminated at a low cost in user’s homes. The objective of this workshop is to provide participants with an introduction to the technology and illustrate how it has thus far been applied in application development and evaluation. Participants will have an opportunity to try out the system and take part in a discussion regarding future research and clinical developments.

Rutu Mulkar-Mehta, Andrew Gordon, Jerry R. Hobbs, Eduard Hovy: “Causal Markers across Domains and Genres of Discourse”

This paper is a study of causation as it occurs in different domains and genres of discourse. There have been various initiatives to extract causality from discourse using causal markers. However, to our knowledge, none of these approaches have displayed similar results when applied to other styles of discourse. In this study we evaluate the nature of causal markers – specifically causatives, between corpora in different domains and genres of discourse and measure the overlap of causal markers using two metrics – Term Similarity and Causal Precision. We find that causal markers, specially causatives (causal verbs) are extremely domain dependent, and moderately genre dependent.

Joel Jurik, Andrew Jones, Mark Bolas, Paul Debevec: “Prototyping a Light Field Display Involving Direct Observation of a Video Projector Array”

We present a concept for a full-parallax light field display achieved by having users look directly into an array of video projectors. Each projector acts as one angularly varying pixel, so the display’s spatial resolution depends on the number of video projectors and the angular resolution depends on the pixel resolution of any one video projector. We prototype a horizontal-parallax-only arrangement by mechanically moving a single pico-projector to an array of positions, and use long-exposure photography to simulate video of a horizontal array. With this setup, we determine the minimal projector density required to produce a continuous image, and describe practical ways to achieve such density and to realize the resulting system. We finally show that if today’s pico-projectors become sufficiently inexpensive, immersive full-parallax displays with arbitrarily high spatial and angular resolution will become possible.

Sin-hwa Kang, Jonathan Gratch: “People Like Virtual Counselors That Highly-Disclose About Themselves”

In this paper, we describe our findings from research designed to explore the effect of self-disclosure between virtual human counselors (interviewers) and human users (interviewees) on users’ social responses in counseling sessions. To investigate this subject, we designed an experiment involving three conditions of self-disclosure: high-disclosure, low-disclosure, and non-disclosure. We measured users’ sense of co-presence and social attraction to virtual counselors. The results demonstrated that users reported more co-presence and social attraction to virtual humans who disclosed highly intimate information about themselves than when compared to other virtual humans who disclosed less intimate or no information about themselves. In addition, a further analysis of users’ verbal self-disclosure showed that users revealed a medium level of personal information more often when interacting with virtual humans that highly-disclosed about themselves, than when interacting with virtual humans disclosing less intimate or no information about themselves.

CBS Evening News Features Interview with Soldier Treated with ICT’s VR Therapy

segment on the national CBS Evening News told the personal story of 27-year-old former Army Staff Sergeant Jeff Matthews, who is one of 90 soldiers taking part in a study of virtual reality exposure therapy for treating post-traumatic stress. The study, being conducted at Emory University, uses the VR exposure therapy developed by Skip Rizzo, ICT’s associate director for medical virtual reality. In the segment, Matthews credits the therapy for helping him come to peace with his combat memories and his wife credits it with saving their marriage.

Without therapy, she says, “It would’ve ended. It would have ended completely.”

Watch the CBS News story here.

Derya Ozkan: “Modeling Wisdom of Crowds Using Latent Mixture of Discriminative Experts”

In many computational linguistic scenarios, training labels are subjectives making it necessary to acquire the opinions of multiple annotators/experts, which is referred to as ”wisdom of crowds”. In this paper, we propose a new approach for modeling wisdom of crowds based on the Latent Mixture of Discriminative Experts (LMDE) model that can automatically learn the prototypical patterns and hidden dynamic among different experts. Experiments show improvement over state-of-the-art approaches on the task of listener backchannel prediction in dyadic conversations.

Fabrizio Morbini Presents at SIGdial

Individual utterances often serve multiple communicative purposes in dialogue.  We present a data-driven approach for identification of multiple dialogue acts in single utterances in the context of dialogue systems with limited training data.  Our approach results in significantly increased understanding of user intent, compared to two strong baselines.

Ron Artstein: “Error Return Plots”

Error-return plots show the rate of error (misunderstanding) against the rate of non-return (non-understanding) for Natural Language Processing systems. They are a useful visual tool for judging system performance when other measures such as recall/precision and detection-error tradeoff are less informative, specifically when a system is judged on the correctness of its responses, but may elect to not return a response.

Sudeep Gandhe, Alysa Taylor, Jillian Gerten, David Traum: “Rapid Development of Advanced Question-Answering Characters by Non-experts”

We demonstrate a dialogue system and the accompanying authoring tools that are designed to allow authors with little or no experience in building dialogue systems to rapidly build advanced question-answering characters. To date seven such virtual characters have been built by non-experts using this architecture and tools. Here we demonstrate one such character, PFC Sean Avery, which was developed by a non-expert in 3 months.

Eliza Margaretha, David DeVault: ” An Approach to the Automated Evaluation of Pipeline Architectures in Natural Language Dialogue Systems”

We present an approach to performing automated evaluations of pipeline architectures in natural language dialogue systems. Our approach addresses some of the difficulties that arise in such automated evaluations, including the lack of consensus among human annotators about the correct outputs within the processing pipeline, the availability of multiple acceptable system responses to some user utterances, and the complex relationship between system responses and internal processing results. Our approach includes the development of a corpus of richly annotated target dialogues, simulations of the pipeline processing that could occur in these dialogues, and an analysis of how system responses vary based on internal processing results within the pipeline. We illustrate our approach, and the kinds of insights it can provide into system performance, in two implemented virtual human dialogue systems.

David DeVault: “Toward Learning and Evaluation of Dialogue Policies with Text Examples”

We present a dialogue collection and enrichment framework that is designed to explore the learning and evaluation of dialogue policies for simple conversational characters using textual training data. To facilitate learning and evaluation, our framework enriches a collection of role-play dialogues with additional training data, including paraphrases of user utterances, and multiple independent judgments by external referees about the best policy response for the character at each point. As a case study, we use this framework to train a policy for a limited domain tactical questioning character, reaching promising performance. We also introduce an automatic policy evaluation metric that recognizes the validity of multiple conversational responses at each point in a dialogue. We use this metric to explore the variability in human opinion about optimal policy decisions, and to automatically evaluate several learned policies in our example domain.

Kallirroi Georgila, Ron Artstein, Angela Nazarian, Michael Rushforth, David Traum, Katia Sycara: “An annotation scheme for cross-cultural argumentation and persuasion dialogues”

We present a novel annotation scheme for cross-cultural argumentation and persuasion dialogues. This scheme is an adaptation of existing coding schemes on negotiation, following a review of literature on cross-cultural differences in negotiation styles. The scheme has been refined through application to coding both two-party and multi-party negotiation dialogues in three different domains, and is general enough to be applicable to different domains with few if any extensions. Dialogues annotated with the scheme have been used to successfully learn culture-specific dialogue policies for argumentation and persuasion.

Canada’s CBC News Features ICT’s Virtual Reality Treatment for PTSD

An in-depth story on returning troops suffering from post-traumatic stress featured Skip Rizzo and the virtual reality exposure therapy he developed at ICT. The segment illustrated how the treatment works and highlighted the personal story of one soldier who credits the therapy for helping him overcome his PTSD.

Watch the story here.

Italian Newspaper Features ICT’s Paul Debevec

The Italian publication L’Espresso quoted Paul Debevec, ICT’s associate director for graphics research, and discussed advances in creating digital characters for movies in a story headlined: There’s an Actor in your PC.

“The truth is we’re facing an epic change,” said Debevec, a pioneer of these technologies, who has worked with stars such as Will Smith and Charlize Theron. “We’ve never been so close to capturing the essence of performance; only actors truly skilled in their craft will be successful, at the expense of those who are merely attractive or well-connected.”

The revolution has just begun, or, la rivoluzione e appena cominciata, states the story.

Read the story in Italian.

INCOPD ALC2015 Immersive Training Workshop

The purpose of the immersive training program is to introduce Army trainers to engaging and relevant instructional technologies such as highly realistic, interactive games and video-based scenarios, and to consider how such games and scenarios can be incorporated into distance learning (DL) and other technology-based courseware. All workshops, seminars, and classes will remain grounded in evidence-based instructional design principles that require the use of best practices to drive the development of training technologies. There is no doubt that such strategies can strengthen DL by helping Army trainers transition from traditional lecture/slide presentations to more powerful, effective, scenario-based instruction. For this reason, the immersive training program will be an ongoing initiative for Army trainers involved in designing and developing multimedia courseware.

Signal Magazine Covers ICT

A training and education story in Signal Magazine, featured ICT’s advances in virtual humans research and the ways these characters are being used for military training, movie special effects and more.

“Our scientists have been recognized as [being at] the top of their fields in graphics and virtual humans research, contributions to the artificial intelligence community, trauma psychology, emotion modeling and social simulation. And in a sign that we’ve really made it in Los Angeles, we’ve even received an Academy Award, which was given to one of our researchers, Paul Debevec, for his advances in creating believable digital faces. His techniques and technologies have been used in Avatar, The Curious Case of Benjamin Button, Spider-Man 2 and many other blockbuster films,” said Randall W. Hill Jr., ICT’s executive director.

Signal is the magazine of the Armed Forces Communications and Electronics Association.

Read the full story here.

Kip Haynes, Eric Chance: “CaveSL: A Large Format Scalable Multi-display System for Social and Scientific Visualization in Second Life”

As virtual worlds have become more popular for education and socialization, many researchers have begun to utilize virtual worlds like Second Life as a novel method for viewing scientific data [Bor08]. However, the typical means of accessing SL is through a single computer screen, which lessens the immersion that is inherent in such a rich 3D world. Because of this, the SL virtual world is a good candidate for adaptation to large scale immersive displays such as a CAVE and other multi projector systems. CaveSL is a freely available modified SL Viewer we developed that allows researchers to utilize large format multi-display systems for social and scientific visualization.

Peter Carnevale, Yoo Kyung Kim, Celso de Melo, Morteza Dehghani, Jonathan Gratch: “These Are Ours: The Effects of Ownership and Groups on Property Negotiation”

Huffington Post Names ICT’s Gmail Motion Response Among Best Kinect Hacks

An online slideshow in the Huffington Post features what they consider the top nine Kinect Hacks to date. Coming in at number two is ICT’s April Fools Day response to Google’s Gmail motion prank.

“They’re fun, surprising, and some even offer potential solutions to real-world problems.,” says the Huffington Post of their winning selections.

Congrats to the team at ICT’s Mixed Reality Lab. You’re number one on our list.

View the slideshow.

View the ICT Mixed Reality Lab’s GMail Motion Spoof.

View the original Gmail Motion video.

Edward Haynes and Eric Chance: “CaveSL : A Large Format Scalable Multi-display System for Social and Scientific Visualization in Second Life”

As virtual worlds have become more popular for education and socialization, many researchers have begun to utilize virtual worlds like Second Life as a novel method for viewing scientific data [Bor08]. However, the typical means of accessing SL is through a single computer screen, which lessens the immersion that is inherent in such a rich 3D world. Because of this, the SL virtual world is a good candidate for adaptation to large scale immersive displays such as a CAVE and other multi projector sys- tems. CaveSL is a freely available modified SL Viewer we developed that allows researchers to utilize large format multi-display systems for social and scientific visualiza- tion.

Alex Wan-chun Ma, Ko-Yun Lui, Chun-Fa Chang, Chuan-Chang Wang, Paul Debevec: “A Framework for Locally Retargeting and Rendering Facial Performance”

We present a facial motion retargeting method that enables the control of a blendshape rig according to marker-based motion capture data. The main purpose of the proposed technique is to allow a blendshape rig to create facial expressions, which conforms best to the current motion capture input, regardless the underlying blendshape poses. In other words, even though all of the blendshape poses may comprise symmetrical facial expressions only, our method is still able to create asymmetrical expressions without physically splitting any of them into more local blendshape poses. An automatic segmentation technique based on the analysis of facial motion is introduced to create facial regions for local retargeting. We also show that it is possible to blend normal maps for rendering in the same framework. Rendering with the blended normal map significantly improves surface appearance and details.

Boston Globe Covers ICT’s Virtual Human Museum Guides

An article in the Boston Globe features ICT’s Ada and Grace, virtual human museum guides in the Boston Museum of Science that were based upon real-life model Bianca Rodriguez. The story describes scanning Rodriguez in the ICT Light Stage and follows her as she visits the museum to meet her digital doubles.

“The voices are different, but the movements are my movements,’’ she said as her voice trailed off and she shook her head at her cyberchildren. “The facial expressions, the way they hold their hands . . .’’

The online version of the article also shows ICT video footage of the process of turning Rodriguez into a virtual human.

Read the full story and watch the video here.

Visit the project webpage.

USC Viterbi Magazine Features Story on ICT

The Spring 2011 issue of Viterbi Magazine covers ICT’s move to Playa Vista and the ribbon cutting ceremony held here in October.

Read the magazine here.

ICT Researchers Win Best Poster Award at Florida Artificial Intelligence Research Society Conference

Work by ICT researchers Grace Chen, Emma Tosch, Ron Artstein, Anton Leuski, and David Traum received the Best Paper Award at the 24th Florida Artificial Intelligence Research Society Conference (FLAIRS-24) held in Palm Beach, Florida, May 18 – 20, 2011. The poster, Evaluating Conversational Characters Created through Question Generation describes research the team conducted using question generation tools to turn encyclopedia articles into question-answer knowledge bases for conversational characters.  Characters were tested by collecting questions and answers from naive participants, running the questions through the character, and comparing the system responses to the participant answers. Characters gave a good or decent answer to 56% of the user questions which had an answer available in the source text, and 43% of all questions asked. Answer quality varied by question type: the best results were answers to who questions, while answers to yes/no questions were among the poorer performers. The results show that question generation is a promising method for creating a question answering conversational character from an existing text.

Congrats to all!

Ron Artstein: “Improving Spoken Dialogue Understanding using Phonetic Mixture Models”

Augmenting word tokens with a phonetic representation, derived from a dictionary, improves the performance of a Natural Language Understanding component that interprets speech recognizer output: we observed a 5% to 7% reduction in errors across a wide range of response return rates. The best performance comes from mixture models incorporating both word and phone features. Since the phonetic representation is derived from a dictionary, the method can be applied easily without the need for integration with a specific speech recognizer. The method has similarities with autonomous (or bottom-up) psychological models of lexical access, where contextual information is not integrated at the stage of auditory perception but rather later.

Belinda Lange: “Harness the Tech Behind the Kinect: Achievements and Challenges in the Implementation of the Microsoft Kinect Technology into Game-based Rehabilitation Applications”

This presentation will describe initial user testing of game-based rehabilitation applications that use the Kinect sensor. Within the Medical VR Group Motor Rehabilitation Lab at the Institute for Creative Technologies (ICT), we have been addressing the challenge for creating low cost home-based video game systems for motor assessment and rehabilitation. The most recent application we have been developing leverages the technology of the Microsoft Kinect 3D depth-sensing camera. We have developed a flexible framework that allows for integration of the 3D depth sensing technology of the PrimeSense camera or the Microsoft Kinect with associated software to drive any PC-based computer game through tracking of the full body skeleton without any markers. This allows the user to control a virtual character on the screen that directly represents their movements in the real world. The application requires only the 3D depth sensing camera and USB connection to a PC. The current prototypes are focused on static balance training and upper limb exercises. The game can be tailored to the player�s level of reach in order to provide the appropriate level of challenge. Game scores and performance data are saved to the computer and can be analyzed following the completion of the session. The technology and prototypes will be presented and the achievements and challenges associated with the implementation of these game-based prototypes within the clinical setting will be presented and discussed.

Ron Artstein: “Evaluating Conversational Characters Created through Question Generation”

Question generation tools can be used to extract a question-answer database from text articles. We investigate how suitable this technique is for giving domain-specific knowledge to conversational characters. We tested these characters by collecting questions and answers from naive participants, running the questions through the character, and comparing the system responses to the participant answers. Characters gave a full or partial answer to 53% of the user questions which had an answer available in the source text, and 43% of all questions asked. Performance was better for questions asked after the user had read the source text, and also varied by question type: the best results were answers to who questions, while answers to yes/no questions were among the poorer performers. The results show that question generation is a promising method for creating a question answering conversational character from an existing text.

Lixing Huang, Louis-Philippe Morency, Jonathan Gratch: “A Multimodal End-of-Turn Prediction Model: Learning from Parasocial Consensus Sampling”

Virtual humans, with realistic behaviors and increasingly human-like social skills, evoke in users a range of social behaviors normally only seen in human face-to-face interactions. One of the key challenges in creating such virtual humans is giving them human-like conversational skills. Traditional conversational virtual humans usually make turn-taking decisions depending on explicit cues, such as “press-to-talk buttons”, from the human users. In contrast, people decide when to take turns by observing their conversational partner’s behavior. In this paper, we present a multimodal end-of-turn prediction model. Instead of recording face-to-face conversations, we collect the turn-taking data using Parasocial Consensus Sampling (PCS) framework, where participants are guided to interact with media representation of people parasocially. Then, we analyze the relationship between verbal and nonverbal features and turn-taking behavior using the consensus data and show how these features influence the time people use to take turns. Finally, we present a probabilistic multimodal end-of-turn prediction model learned from the consensus data, which enables virtual humans to make real-time turn-taking predictions. The evaluation results show that our model achieves a high accuracy and takes human-like pauses, in terms of length, before taking its turns. Our work demonstrates the validity of Parasocial Consensus Sampling and generalizes this framework to model turn-taking behavior.

Celso de Melo, Peter Carnevale, Jonathan Gratch: “The Effect of Expression of Anger and Happiness in Computer Agents on Negotiations with Humans”

There is now considerable evidence in social psychology, economics, and related disciplines that emotion plays an important role in negotiation. For example, humans make greater concessions in negotiation to an opposing human who expresses anger, and they make fewer concessions to an opponent who expresses happiness, compared to a no-emotion-expression control. However, in AI, despite the wide interest in negotiation as a means to resolve differences between agents and humans, emotion has been largely ignored. This paper explores whether expression of anger or happiness by computer agents, in a multiissue negotiation task, can produce effects that resemble effects seen in human-human negotiation. The paper presents an experiment where participants play with agents that express emotions (anger vs. happiness vs. control) through different modalities (text vs. facial displays). An important distinction in our experiment is that participants are aware that they negotiate with computer agents. The data indicate that the emotion effects observed in past work with humans also occur in agent-human negotiation, and occur independently of modality of expression. The implications of these results are discussed for the fields of automated negotiation, intelligent virtual agents and artificial intelligence.

Chung-Cheng Chiu, Stacy Marsella: “A style controller for generating virtual human behaviors”

Creating a virtual character that exhibits realistic physical behaviors requires a rich set of animations. To mimic the variety as well as the subtlety of human behavior, we may need to animate not only a wide range of behaviors but also variations of the same type of behavior influenced by the environment and the state of the character, including the emotional and physiological state. A general approach to this challenge is to gather a set of animations produced by artists or motion capture. However, this approach can be extremely costly in time and effort. In this work, we propose a model that can learn styled motion generation and an algorithm that produce new styles of motions via style interpolation. The model takes a set of styled motions as training samples and creates new motions that are the generalization among the given styles. Our style interpolation algorithm can blend together motions with distinct styles, and improves on the performance of previous work. We verify our algorithm using walking motions of different styles, and the experimental results show that our method is significantly better than previous work.

Jason Tsai, Natalie Fridman, Emma Bowring, Matthew Brown, Shira Epstein, Gal Kaminka, Stacy Marsella, Andrew Ogden, Rika Inbal, Ankur Sheel, Matthew Taylor, Xuezhi Wang, Avishay Zilka, Milind Tambe: “ESCAPES – Evacuation Simulation with Children, Authorities, Parents, Emotions, and Social comparison”

In creating an evacuation simulation for training and planning, realistic agents that reproduce known phenomenon are required. Evacuation simulation in the airport domain requires additional features beyond most simulations, including the unique behaviors of first time visitors who have incomplete knowledge of the area and families that do not necessarily adhere to often-assumed pedestrian behaviors. Evacuation simulations not customized for the airport domain do not incorporate the factors important to it, leading to inaccuracies when applied to it. In this paper, we describe ESCAPES, a multiagent evacuation simulation tool that incorporates four key features: (i) different agent types; (ii) emotional interactions; (iii) informational interactions; (iv) behavioral interactions. Our simulator reproduces phenomena observed in existing studies on evacuation scenarios and the features we incorporate substantially impact escape time. We use ESCAPES to model the International Terminal at Los Angeles International Airport (LAX) and receive high praise from security officials.

Celso De Melo: “The Effect of Expression of Anger and Happiness in Computer Agents on Negotiations with Humans”

There is now considerable evidence in social psychology, economics, and related disciplines that emotion plays an important role in negotiation. For example, humans make greater concessions in negotiation to an opposing human who expresses anger, and they make fewer concessions to an opponent who expresses happiness, compared to a no-emotion-expression control. However, in AI, despite the wide interest in negotiation as a means to resolve differences between agents and humans, emotion has been largely ignored. This paper explores whether expression of anger or happiness by computer agents, in a multiissue negotiation task, can produce effects that resemble effects seen in human-human negotiation. The paper presents an experiment where participants play with agents that express emotions (anger vs. happiness vs. control) through different modalities (text vs. facial displays). An important distinction in our experiment is that participants are aware that they negotiate with computer agents. The data indicate that the emotion effects observed in past work with humans also occur in agent-human negotiation, and occur independently of modality of expression. The implications of these results are discussed for the fields of automated negotiation, intelligent virtual agents and artificial intelligence.

Peter Khooshabeh, Cade McCall, Jonathan Gratch, James Blascovich: “Does it matter if a computer jokes?”

The goal here was to determine whether computer interfaces are capable of social influence via humor. Users interacted with a natural language capable virtual agent that told persuasive information, and they were given the option to use information from the dialogue in order to complete a problem-solving task. Individuals interacting with an ostensibly humorous virtual agent were influenced by it such that those who judged the agent unfunny were less likely to be persuaded and departed from the agent’s suggestions. We discuss the implications of these results for HCI involving natural language systems and virtual agents.

Bill Swartout: “What Have We Learned From Virtual Humans”

For a little over a decade, we have been building virtual humans � computer-generated characters � at the USC Institute for Creative Technologies. In this talk I will outline some of the lessons we have learned from building these characters. Ultimately, our vision is to create virtual humans that look and behave just like real people. They will think on their own, model and exhibit emotions, and interact using natural language along with the full repertoire of verbal and non-verbal communication techniques that people use. Although the realization of that goal is still in the future, making steps toward it has required us to weave together different threads of AI research such as computer vision, natural language understanding and emotion modeling that are often treated as independent areas of investigation. Interestingly, this is not just an exercise in systems integration, but instead has revealed synergies across areas that have allowed us to address problems that are difficult to solve if addressed from one perspective alone. I will illustrate some of these synergies in this talk. I will also discuss the role of story in our work and show how embedding virtual humans in a compelling story or scenario can both make them more feasible to implement and suggest new areas of research. Finally, I will suggest future areas of research in virtual humans and suggest what might be possible in the not too distant future.

Graham Fyffe, Tim Hawkins, Chris Watts, Wan-Chun Ma, Paul Debevec: “Comprehensive Facial Performance Capture”

We present a system for recording a live dynamic facial performance, capturing highly detailed geometry and spatially varying diffuse and specular reflectance information for each frame of the performance. The result is a reproduction of the performance that can be rendered from novel viewpoints and novel lighting conditions, achieving photorealistic integration into any virtual environment. Dynamic performances are captured directly, without the need for any template geometry or static geometry scans, and processing is completely automatic, requiring no human input or guidance. Our key contributions are a heuristic for estimating facial reflectance information from gradient illumination photographs, and a geometry optimization framework that maximizes a principled likelihood function combining multi-view stereo correspondence and photometric stereo, using multi-resolution belief propagation. The output of our system is a sequence of geometries and reflectance maps, suitable for rendering in off-the-shelf software. We show results from our system rendered under novel viewpoints and lighting conditions, and validate our results by demonstrating a close match to ground truth photographs.

Yvonne Jung, Arjan Kuijper, Dieter Fellner, Michael Kipp, Jan Miksatko, Jonathan Gratch, Daniel Thalmann: “Believable Virtual Characters in Human-Computer Dialogs

For many application areas, where a task is most naturally represented by talking or where standard input devices are difficult to use or not available at all, virtual characters can be well suited as an intuitive man-machineinterface due to their inherent ability to simulate verbal as well as nonverbal communicative behavior. This type of interface is made possible with the help of multimodal dialog systems, which extend common speech dialog systems with additional modalities just like in human-human interaction. Multimodal dialog systems consist at least of an auditive and graphical component, and communication is based on speech and nonverbal communication alike. However, employing virtual characters as personal and believable dialog partners in multimodal dialogs entails several challenges, because this requires not only a reliable and consistent motion and dialog behavior but also regarding nonverbal communication and affective components. Besides modeling the �mind� and creating intelligent communication behavior on the encoding side, which is an active field of research in artificial intelligence, the visual representation of a character including its perceivable behavior, from a decoding perspective, such as facial expressions and gestures, belongs to the domain of computer graphics and likewise implicates many open issues concerning natural communication. Therefore, in this report we give a comprehensive overview how to go from communication models to actual animation and rendering.

Graham Fyffe, XueMing Yu, Paul Debevec: “Single-Shot Photometric Stereo by Spectral Multiplexing”

We propose a novel method for single-shot photometric stereo by spectral multiplexing. The output of our method is a simultaneous per-pixel estimate of the surface normal and full-color reflectance. Our method is well suited to materials with varying color and texture, requires no time-varying illumination, and no high-speed cameras. Being a single-shot method, it may be applied to dynamic scenes without any need for optical flow. Our key contributions are a generalization of three-color photometric stereo to more than three color channels, and the design of a practical six-color-channel system using off-the-shelf parts.

Wall Street Journal Features ICT’s Virtual Reality Therapy for PTSD

The Wall Street Journal highlighted technology developed by Skip Rizzo, ICT director of medical virtual reality, which uses virtual reality simulations to help treat Iraq veterans suffering from post-traumatic stress disorder (PTSD). Many clinical psychologists are using the tool to treat PTSD, with marked improvement demonstrated within six weeks, the story reported.

The story was part of a slide show featuring examples of virtual reality from the new book, Infinite Realities: Avatars, Eternal Life, New Worlds and the Dawn of the Virtual Revolution, by virtual human researchers Jim Blascovich and Jeremy Bailenson

ICT’s Skip Rizzo in NYC at the UN and in New Orleans

Dr. Skip Rizzo will present in NYC at the UN to the World Health Organization on Trauma Intervention about aid worker preparation using virtual humans and about expanding access to care via SimCoach. The next day in New Orleans, Skip will present on similar topics (PTSD, Barriers to Care, Clinical Training) at the New Orleans VA.

New York Times, Washington Post, NPR, CNN and More Feature ICT MxR Lab’s Gmail Motion Video

Continuing coverage of the April 1 video out of Mark Bolas’ MxR Lab here at ICT that turns Google’s Gmail Motion spoof into a working prototype…

The New York TimesWashington PostCNNNPRMashable.com and LA Weekly covered the project, which leverages MxR’s FAAST middleware created by post-doc Evan Suma. With the open-source FAAST, which is based on the OpenNI library, users can replace keyboard strokes with body movements, as they demonstrate with their Gmail Motion video. Other more serious applications and research initiatives include using FAAST to develop applications for physical therapy and rehabilitation purposes.

Watch ICT MxR Researchers Make Gmail Motion Work, Read Press Coverage

Evan Suma and the folks in Mark Bolas’ MxR Lab put together this demo illustrating how to make Gmail Motion, Google’s April Fools Day prank, really work. Their video on the ICT YouTube Channel has around five hundred thousand views and got the attention of the folks at TimeEngadetPCWorld and PC Mag, among others, including the folks at Gmail.

They call their program SLOOW, for the Software Library Optimizing Obligatory Waving. We actually think it is pretty quick and we’re gesturing our hands like crazy to give them a round of applause and a high-five.

You may recoginze Suma from his earlier video using his FAAST toolkit, to play World of Warcraft. Both projects are based on the OpenNI library.

Read Engadget’s Coverage.

Read Time’s Coverage.

Read PC World’s Coverage.

Paul Debevec and Andrew Gordon Recieve Promotions at the USC Viterbi School of Engineering

Paul Debevec, associate director for graphics research at ICT, has been named a full research professor in the Computer Science department at the USC Viterbi School of Engineering.
The department also promoted ICT research scientist Andrew Gordon to the title of research associate professor.

Los Angeles Times Features ICT Virtual Patients Deployed to USC School of Social Work for Training

A story in the Los Angeles Times explains how ICT-developed virtual patients are being used to train clinicians at the USC School of Social Work, which began the nation’s first specialization in military social work. The program is also offered online through the school’s new Virtual Academic Center. ICT-developed technologies like the Virtual PatientSimCoach and Virtual Iraq/Afghanistan projects are being incorporated in the specialized curriculum to provide training and support in addressing the needs of military members and their families, including identifying and treating post-traumatic stress.

Read the story here.

ICT Second Life Project Wins Award at 2011 Federal Virtual Worlds Challenge

Congrats to ICT’s VIGOR team on their 2nd place win in the Patterns of Life category of the Federal Virtual World Challenge.

ICT’s Jacki Morie, Dusan Jan and Anton Leuski accepted the award today, which includes $3000, at a luncheon awards ceremony at the Defense Game Tech Conference in Orlando, Florida. TheVIGOR project, sponsored by the Army’s Training and Doctrine Command, is focused on creating different types of virtual characters to inhabit in virtual worlds, like Second Life, and provide a richer experience for human visitors to such spaces.

ICT’s winning entry was a virtual character-driven checkpoint training exercise, which uses over a dozen intelligent and semi-autonomous bots. It was created as a proof of concept to see if a complicated training scenario could be deployed in Second Life.

And apparently it can.

The Federal Virtual World Challenge is an annual, global challenge, led by the U.S. Army Research Laboratory – Simulation & Training Technology Center, to explore innovative and interactive solutions in virtual environments.

Click here for the Federal Virtual World Challenge Site.

Click here for the GameTech Conference site.

ICT Researchers Win Best Tech-Note Award at IEEE Symposium on 3D User Interface

A paper from Evan Suma, David Krum and Mark Bolas of ICT’s MxR lab received the Best Tech-Note Award at the IEEE Symposium on 3D User Interfaces in Singapore. The paper, Effects of Redirection on Spatial Orientation in Real and Virtual Environments, describes an experiment probing the relationship between the real world and virtual world models for spatial orientation using a pointing task during which participants were redirected. In an effort to create the illusion of being in a virtual space that is larger than the real world space it is housed in, this research aims to discover how redirecting users in a virtual environment impacts their spatial orientation in the real world. Samantha Finkelstein of UNC Charlotte was a co-author on the tech-note, which contained unpublished preliminary results of research or design work.

Visit the conference website.

Jon Gratch Gives the Keynote Speech at The 9th Conference on Automatic Face and Gesture Recognition

Face and gesture research has made enormous progress in recognizing human nonverbal signals, but still faces important challenges in understanding the social meaning and significance of such cues. In this talk I will discuss a number of successes and some failures in using expression and gesture recognition techniques in a variety of human-human and human-computer social contexts. On the one hand, I will describe research on virtual humans (interactive digital characters) that can engage users in rich face-to-face interactions. I will describe evidence that endowing these artifacts with the ability to recognize and respond to human nonverbal cues has important social effects such as enhanced mutual understanding and persuasiveness. On the other hand, by facilitating the annotation of human nonverbal behavior, face and gesture research is revolutionizing the study of human social processes and providing new insights into theories of human social behavior. These two topics work hand-in-hand, as theories of human social processes have important implications, not only for the design of interactive virtual humans, but for face and gesture research as well. I will end by describing how social psychological theory, especially theories of emotion, has implications for research in the automatic recognition and understanding of human social signals.

Giota Stratou: “Effect of Illumination on Automatic Expression Recognition: A Novel 3D Relightable Facial Database”

One of the main challenges in facial expression recognition is illumination invariance. Our long-term goal is to develop a system for automatic facial expression recognition that is robust to light variations. In this paper, we introduce a novel 3D Relightable Facial Expression (ICT-3DRFE) database that enables experimentation in the fields of both computer graphics and computer vision. The database contains 3D models for 23 subjects and 15 expressions, as well as photometric information that allow for photorealistic rendering. It is also facial action units annotated, using FACS standards. Using the ICT-3DRFE database we create an image set of different expressions/illuminations to study the effect of illumination on automatic expression recognition. We compared the output scores from automatic recognition with expert FACS annotations and found that they agree when the illumination is uniform. Our results show that the output distribution of the automatic recognition can change significantly with light variations and sometimes causes the discrimination of two different expressions to be diminished. We propose a ratio-based light transfer method, to factor out unwanted illuminations from given images and show that it reduces the effect of illumination on expression recognition.

Konstantinos Bousmalis, Louis-Philippe Morency, Maja Pantic: “Modeling Hidden Dynamics of Multimodal Cues for Spontaneous Agreement and Disagreement Recognition”

This paper attempts to recognize spontaneous agreement and disagreement based only on nonverbal multimodalcues. Related work has mainly used verbal and prosodic cues. We demonstrate that it is possible to correctly recognize agreement and disagreement without the use of verbal context(i.e. words, syntax). We propose to explicitly model the complex hidden dynamics of the multimodal cues using a sequential discriminative model, the Hidden Conditional Random Field (HCRF). In this paper, we show that the HCRF model is able to capture what makes each of these social attitudes unique. We present an efficient technique to analyze the concepts learned by the HCRF model and show that these coincide with the findings from social psychology regarding which cues are most prevalent in agreement and disagreement. Our experiments are performed on a spontaneous dataset of real televised debates. The HCRF model outperforms conventional approaches such as Hidden Markov Models and Support Vector Machines.

David Krum: “A Design for a Smartphone-Based Head Mounted Display”

Thin computing clients, such as smartphones and tablets, have experienced recent growth in display resolutions and graphics processing power. In this poster, we show how to leverage these trends to create an experimental wide field of view, 3D stereoscopic head mounted display (HMD), based on two high resolution smartphones. This HMD prototype is unique in that the graphics system is entirely onboard, allowing it to be lightweight, wireless, and convenient to use.

Melissa Roemmelle, Cosmin Bejan, Andrew Gordon: “Choice of Plausible Alternatives: An Evaluation of Commonsense Causal Reasoning”

Research in open-domain commonsense reasoning has been hindered by the lack of evaluation metrics for judging progress and comparing alternative approaches. Taking inspiration from large-scale question sets used in natural language processing research, we authored one thousand English-language questions that directly assess commonsense causal reasoning, called the Choice Of Plausible Alternatives (COPA) evaluation. Using a forced- choice format, each question gives a premise and two plausible causes or effects, where the correct choice is the alternative that is more plausible than the other. This paper describes the authoring methodology that we used to develop a validated question set with sufficient breadth to advance open-domain commonsense reasoning research. We discuss the design decisions made during the authoring process, and explain how these decisions will affect the design of high-scoring systems. We also present the performance of multiple baseline approaches that use statistical natural language processing techniques, establishing initial benchmarks for future systems.

Andrew Gordon, Jerry R. Hobbs: “A Commonsense Theory of Mind-Body Interaction”

We propose a logical formalization of a commonsense theory of mind-body interaction as a step toward a deep lexical semantics for words and phrases related to this topic.

Logan Olson, David Krum, Evan Suma and Mark Bolas: “A Design for a Smartphone-Based Head Mounted Display”

Thin computing clients, such as smartphones and tablets, have experienced recent growth in display resolutions and graphics processing power. In this poster, we show how to leverage these trends to create an experimental wide field of view, 3D stereoscopic head mounted display (HMD), based on two high resolution smartphones. This HMD prototype is unique in that the graphics system is entirely onboard, allowing it to be lightweight, wireless, and convenient to use.

Evan Suma, Belinda Lange, Skip Rizzo, David Krum, Mark Bolas: “FAAST: The Flexible Action and Articulated Skeleton Toolkit”

The Flexible Action and Articulated Skeleton Toolkit (FAAST) is middleware to facilitate integration of full-body control with virtual reality applications and video games using OpenNI-compliant depth sensors (currently the PrimeSensor and the Microsoft Kinect). FAAST incorporates a VRPN server for streaming the user’s skeleton joints over a network, which provides a convenient interface for custom virtual reality applications and games. This body pose information can be used for goals such as realistically puppeting a virtual avatar or controlling an on-screen mouse cursor. Additionally, the toolkit also provides a configurable input emulator that detects human actions and binds them to virtual mouse and keyboard commands, which are sent to the actively selected window. Thus, FAAST can enable natural interaction for existing off-the-shelf video games that were not explicitly developed to support input from motion sensors. The actions and input bindings are configurable at run-time, allowing the user to customize the controls and sensitivity to adjust for individual body types and preferences. In the future, we plan to substantially expand FAAST’s action lexicon, provide support for recording and training custom gestures, and incorporate real-time head tracking using computer vision techniques.

Evan Suma, David Krum, Mark Bolas: “Sharing Space in Mixed and Virtual Reality Environments Using a Low-Cost Depth Sensor”

We describe an approach for enabling people to share virtual space with a user that is fully immersed in a head-mounted display. By mounting a recently developed low-cost depth sensor to the user�s head, depth maps can be generated in real-time based on the user�s gaze direction, allowing us to create mixed reality experiences by merging real people and objects into the virtual environment. This enables verbal and nonverbal communication between users that would normally be isolated from one another. We present the implementation of the technique, then discuss the advantages and limitations of using commercially available depth sensing technology in immersive virtual reality applications.

Evan Suma, Mark Bolas, David Krum, Samantha Finkelstein: “Effects of Redirection on Spatial Orientation in Real and Virtual Environments”

We report a user study that investigated the effect of redirection in an immersive virtual environment on spatial orientation relative to both real world and virtual stimuli. Participants performed a series of spatial pointing tasks with real and virtual targets, during which they experienced three within-subjects conditions: rotation-based redirection, change blindness redirection, and no redirection. Our results indicate that when using the rotation technique, participants spatially updated both their virtual and real world orientations during redirection, resulting in pointing accuracy to the targets� recomputed positions that was strikingly similar to the control condition. While our data also suggest that a similar spatial updating may have occurred when using a change blindness technique, the realignment of targets appeared to be more complicated than a simple rotation, and was thus difficult to measure quantitatively.

Evan Suma, Seth Clark, Samantha Finkelstein, Zachary Wartell, David Krum, Mark Bolas: “Leveraging Change Blindness for Redirection in Virtual Environments”

We present change blindness redirection, a novel technique for allowing the user to walk through an immersive virtual environment that is considerably larger than the available physical workspace. This approach, based on a dynamic environment model, improves on previous redirection techniques, as it does not introduce any visual-vestibular conflicts from manipulating the mapping between physical and virtual motions, nor does it require breaking presence to stop and explicitly reorient the user. We conducted two user studies to evaluate the effectiveness of the change blindness illusion when exploring a virtual environment that was an order of magnitude larger than the physical walking space. Despite the dynamically changing environment, participants were able to draw coherent sketch maps of the environment structure, and pointing task results indicated that they were able to maintain their spatial orientation within the virtual world. Only one out of 77 participants across both both studies definitively noticed that a scene change had occurred, suggesting that change blindness redirection provides a remarkably compelling illusion. Secondary findings revealed that a wide field-of-view increases pointing accuracy and that experienced gamers reported greater sense of presence than those with little or no experience with 3D video games.

ICT’s Morteza Dehghani and David DeVault Named USC Viterbi Research Assistant Professors

The Computer Science Department of the USC Viterbi School of Engineering appointed Morteza Dehghani and David DeVault to research assistant professor positions. Dehghani works as a member of the Virtual Humans team at ICT where his research focus is on how culture and emotion interact to influence social interactions. He earned his Ph.D. at Northwestern University. DeVault is a member of ICT’s Natural Language Dialogue Group, where his research centers on enabling dialogue systems to respond to the inevitable uncertainties of communication in a way that is more flexible, more robust, and more human-like. He earned his Ph.D. at Rutgers University.

Jonathan Gratch: “So she’s smiling, now what?”

Face and gesture research has made enormous progress in recognizing human nonverbal signals, but still faces important challenges in understanding the social meaning and significance of such cues. In this talk I will discuss a number of successes and some failures in using expression and gesture recognition techniques in a variety of human-human and human-computer social contexts. On the one hand, I will describe research on virtual humans (interactive digital characters) that can engage users in rich face-to-face interactions. I will describe evidence that endowing these artifacts with the ability to recognize and respond to human nonverbal cues has important social effects such as enhanced mutual understanding and persuasiveness. On the other hand, by facilitating the annotation of human nonverbal behavior, face and gesture research is revolutionizing the study of human social processes and providing new insights into theories of human social behavior. These two topics work hand-in-hand, as theories of human social processes have important implications, not only for the design of interactive virtual humans, but for face and gesture research as well. I will end by describing how social psychological theory, especially theories of emotion, has implications for research in the automatic recognition and understanding of human social signals.

Julia Campbell: “Developing INOTS to Support Interpersonal Skills Practice”

The Immersive Naval Officer Training System (INOTS) is a blended learning environment that merges traditional classroom instruction with a mixed reality training setting. INOTS supports the instruction, practice and assessment of interpersonal communication skills. The goal of INOTS is to provide a consistent training experience to supplement interpersonal skills instruction for Naval officer candidates without sacrificing trainee throughput and instructor control. We developed an instructional design from cognitive task analysis interviews with experts to serve as a framework for system development. We also leveraged commercial student response technology and research technologies including natural language recognition, virtual humans, realistic graphics, intelligent tutoring and automated instructor support tools. In this paper, we describe our methodologies for developing a blended learning environment, and our challenges adding mixed reality and virtual human technologies to a traditional classroom to support interpersonal skills training.

Skip Rizzo: “Clinical Virtual Reality: A Brief Review of the Future”

This presentation will provide a brief overview of the many forms of Virtual Reality that have been applied across a diverse range of clinical disorders and research questions. I will then present a detailed overview of the use of VR for Exposure Therapy for anxiety disorders, addictive behaviors and with OIF/OEF military personnel with PTSD. This will be followed by overviews of research and clinical applications of VR for cognitive assessment/rehabilitation, motor rehabilitation, pain distraction and social interaction. The social interaction overview will conclude with the detailing of an emerging project area that involves the creation of artificially intelligent virtual human “patients” for clinical training. I will also bring a VR Headmounted display and demonstrate some virtual reality programs that can be run from an off-the-shelf laptop computer.

Workshop on Human-Agent Social Interaction in Open Online Virtual Worlds

A virtual world (VW) is a social virtual environment depicted by a rich graphical landscape, where visitors come together to interact. The inhabitants of a VW, are represented by graphical avatars, communicate using verbal and non-verbal means, and can navigate the world and modify the virtual environment. The world is persistent, so it exists and evolves even when no inhabitants are present. Some VWs are goal-oriented, e.g., World of Warcraft™, there the inhabitants are participating in well-defined activities, solving quests, following explicit rules. These worlds are often called Massive Multiplayer Online Games (MMOGs). Other worlds, e.g., SecondLife™ are goal-agnostic, where the visitors generally invent the goals for themselves, visit places, attend events, or simply gather to talk. VWs and MMOGs are relatively new but important media for information exchange and social interaction. Some estimates put the current number of VW users in excess of a half billion. As participation in these VWs broadens and deepens, the need arises for understanding the nature of interactions and behaviors in these worlds and reconstructing them in Virtual Humans (VHs) – computer-driven characters that can communicate and act with human inhabitants in the VW.

We are interested in addressing the following issues:

  • Studies and theories of how human group interaction in virtual worlds is similar or different from interaction in the real world?
  • What kinds of joint activities are well-suited to virtual worlds (e.g. distance-education, virtual collaborations, etc.)
  • What kinds of roles can virtual humans play in such activities in these worlds?
  • What are the best ways of constructing virtual humans for virtual worlds?
  • What are the best ways of observing, and analyzing interactions in virtual worlds?
  • How can we work together in the future to create the best possible virtual humans who can live in and support meaningful activities in online worlds?

The workshop will consist of talks, discussion and possibly activity breakout sessions.

ICT Attending and Presenting at the Association of Baccalaureate Social Work Program Directors

Josh Williams, Patrick Kenny and Tomer Mor-Barak will attend and present at the Association of Baccalaureate Social Work Program Directors’ “The Nuts and Bolts of BSW Education: From Basics to Competency Assessment” Conference, Cincinnati, OH.

Reasoning With Text

Intelligent robots of the future are going to need to know a lot of stuff. They are going to need to know how a power outage will affect my plans for a holiday party. They need to know what to do if the dishwasher floods my apartment while I am away on a business trip. They need to know to stay off the carpet if my son pours wet paint all over them. How are we ever going to be able to teach all of this stuff to these robots?

Maybe the answer is to give them a lot to read. There are millions of personal stories in Internet weblogs that have a lot of this knowledge in them. There are millions of how-to articles on the web that give step-by-step instructions for a lot of stuff. There are dedicated websites that collect and catalog millions of people’s plans and goals; maybe these can be put to good use. But how?

Natural language understanding is hard work. If we require deep understanding in order to put this text to use, then we’ve simply exchanged one impossible problem for another. Instead, we should be looking for just the right text corpus, and just the right mix of natural language processing necessary to support the reasoning task at hand. If we can figure out exactly what class of reasoning problems we are trying to solve, and how that relates to the structure of the text in a particular corpus, then deep understanding may not be required.

Tucked away deep in the bowels of research labs all across the planet, creative PhD students and junior researchers are working on this problem. They share a common desire to harness natural language processing technologies and the web in order to tackle long-standing artificial intelligence problems. However, they are largely unaware of each other, or even that there is a community of more-senior researchers that care deeply about what they are doing. Now would be a good time to gather all of these people together.

How about a workshop? It would be a great way to bring together junior and senior researchers, where former can share their new ideas and the latter can help foster a healthy research community. Los Angeles is nice, even in the middle of winter. The University of Southern California’s Institute for Creative Technologies (ICT) has a fancy new building that would work perfectly, so we’ve booked the facilities for February 18 and 19. ICT workshops are well organized and always a lot of fun. We’ve secured some funding to ensure that all be well fed and entertained, and that some of the airfare and hotel expenses can be reimbursed. Let’s do it!

Variety Features Use of High Dynamic Range Imagery in Films Up for Best Visual Effects at the Oscars

An article on the films nominated for best visual effects at this year’s Academy Awards focused on the visual effects supervisors’ use of high dynamic range imagery, or HDRI, as a key to achieving the needed level of realism.  The use of HDRI lighting in films is a technique that ICT’s Paul Debevec developed in his postdoctoral work at UC Berkeley in the late 1990’s. Debevec now serves as ICT’s associate director for graphics research.

In the article, Tim Burke, a VFX supervisor on “Harry Potter and the Deathly Hallows: Part 1” explained that taking reference photographs of what was being shot gave the team complete lighting conditions for any scene.

“The light falling on the effects the same way it falls on the things that are not effects is what makes the difference,” he said.

Read the story here

Learn more about Debevec’s Graphics Work at ICT

Read Paul Debevec’s SIGGRAPH 98 paper introducing HDRI lighting.

ICT is Accepting Applications for our 2011 Summer Intern Program

ICT offers select interdisciplinary paid internships for creative and technical students wishing to pursue careers in simulation, interactive media and virtual reality fields. ICT’s mission is to create compelling immersive systems for effective learning for military, entertainment and educational purposes. Applications are due February 18, 2011.

For more information about the program and application process click here.

ICT Researchers Earn Best Paper Award

ICT’s Patrick Kenny and Thomas Parsons received a best paper award at the I/ITSEC 2010 Conference in Orlando, Florida. Their paper, Emerging Concepts and Innovative Technologies: Virtual Patients for Virtual Sick Call Medical Training, received the top honor in the Emerging Concepts and Innovative Technologies category and was also nominated for best paper overall. Pat Garrity of the U.S. Army Simulation and Training Technology Center was a co-author on the paper.

Visit the conference site.

Skip Rizzo Talks about VR Treatment for PTSD on Federal News Radio

Federal News Radio featured ICT’s virtual reality simulations to help treat Iraq and Afghanistan veterans suffering from post-traumatic stress disorder (PTSD). “By confronting the traumatic memories you’re actually processing them in a way that makes them less painful and less anxiety provoking. If you avoid thinking about the things that went on in your deployment and try to put it away and not deal with it, they still come out,” said Albert Rizzo of the institute. “And that’s really at the core of PTSD, this avoidance factor.”

For more info click here.

Celso de Melo Presents Poster at 2011 Emotion Pre-Conference

The Impact of Emotion Displays in Embodied Agents on Emergence of Cooperation with People

Acknowledging the social functions of emotion in people, there has been growing interest in the interpersonal effect of emotion on cooperation in social dilemmas. This work explores whether and how facial displays of emotion in computer agents with virtual bodies impact cooperation with human users. We describe an experiment where participants play the iterated prisoner’s dilemma with two agents that play the same strategy (tit-for-tat), but communicate different goal orientations (cooperative vs. individualistic) through their facial displays. The patterns of facial display reflect the agents’ goal (cooperation or self-interest) in a way that is consistent with appraisal theories of emotion. Results show that people cooperate significantly more with the cooperative than the individualistic agent. Aside from showcasing the potential of using embodied computer agents for doing basic research on the impact of emotion display in human decision-making, the results suggest that the nature of emotion displays (i.e., type and timing) is critical for promoting cooperation. These results are in contrast to the evolutionary argument that cooperators can simply be identified by high emotional expressivity, irrespective of whether the emotion is positive or negative (Boone & Buck, 2003; Schug et al 2010). In our case, both agents are equally emotionally expressive and people are cooperating differently with them. In line with recent research that shows that appraisals mediate inferences of personality characteristics from facial displays (Hareli & Hess, 2009), we argue people are inferring the agent’s goals from the emotion displays by reversing the emotion appraisal mechanism and, then, deciding whether to cooperate with the agents.

Authors are:
Celso M. de Melo, Institute for Creative Technologies – USC
Peter Carnevale, University of Southern California Marshall School of Business
Jonathan Gratch, Institute for Creative Technologies – USC

ICT’s Skip Rizzo Discusses VR Therapies on the DoD’s Blogger’s Roundtable

The January 25, 2011 DoD Blogger’s Roundtable featured Skip Rizzo, ICT’s director of medical virtual reality, talking about how virtual reality applications are being designed and implemented across various points in the military deployment cycle to prevent, identify and treat combat-related post-traumatic stress disorder or PTSD.

Read more and download the podcast here.

AFP Story on Rizzo’s work

Using Kinect and OpenNI to Embody Avatars in Second Life

Continuing on their previous work using gestures to play World of Warcraft, ICT researchers have developed a new software, using the OpenNI toolkit as a foundation, which utilizes Kinect to read gestures and triggers corresponding server-side scripts within Second Life. Instead of having to think about pressing the right sequence of keys to make a ‘wave’ gesture, the user can simply raise their hand and wave. The software was developed by ICT engineer Thai Phan who is also a computer science grad student at the USC Viterbi School of Engineering. Phan expects a downloadable version to be ready in a few days and demonstrates how it works in this YouTube video.

The work comes out of the MxR Lab run by Mark Bolas, ICT’s associate director for mixed reality and an associate professor in the interactive media division at the USC School of Cinematic Arts.

“This work begins to uncover what can be expected as users begin to use their bodies to interact with computers,” said Bolas. The line between what is and is not “me” – where I begin and end – quickly becomes blurred.”

Mark Core: “Natural Language Understanding Considerations for Lifelong Learning Companions”

In this paper, we present considerations for natural language processing for a lifelong learning companion. In the context of these considerations, we review related work in automated assessment of learner writing and present an idea for augmenting keyword spotting with syntactic information. However, the extra information given by syntax is offset by parser errors and added burden on the author. The results suggest that while standard keyword spotting is a quick approach to adding NLU capabilities it has inherent limitations.

Voice of America Covers ICT

Voice of America reporter Mike O’Sullivan spent some time with our mixed reality and medical VR folks and put together a widely-distributed story and video on our efforts to use immersive systems to for therapy, rehab and support. He featured our Virtual Iraq, Sim Coach, Virtual Patient projects as well as our newer motor rehab systems.

Medical VR Group at the 2011 International Consumer Electronics Show

Dr. Albert “Skip” Rizzo and the ICT Medical Virtual Reality group will be attending and presenting SimCoach and Virtual Iraq/Afghanistan demonstrations at the 2011 International Consumer Electronics Show (CES), Las Vegas, NV.

World of Warcraft Kinect YouTube Video Gets Over One Million Views and Much Press Coverage

So much for talking the holiday week off. ICT’s Evan Suma has been busy fielding calls and emails about his new FAAST toolkit which facilitates integration of full-body control with games and VR applications. It might have something to do with the fact that Suma chose the hugely popular World of Warcraft to demonstrate how his program, which can be used to control off-the-shelf video games, works. His video has gotten over one million YouTube hits and led to countess online mentions, including WiredEngadgetThe Wall Street JournalThe Seattle PIUSA TodayMSNBCFast CompanyKotakuLalawag and Yahoo. Suma and his collaborators at ICT see this as opening up huge possibilities in rehab, exercise and play. Stay tuned, this one is quite literally a game changer.