ICT 3D Teleconferencing Mention in the Wall Street Journal

Sarah McBride of the Wall Street Journal talks about new media including ICT’s 3D teleconferencing technology that creates a hologram effect.

Read the article.

Andrew Jones, Matt Jen-Yuan Chiang, Pieter Peers, Paul Debevec, Alex Wan-chun Ma, Tim Hawkins, Marko Vukovic, Ming Ouhyoung: “Facial Performance Synthesis using Deformation-Driven Polynomial Displacement Maps”

We present a novel method for acquisition, modeling, compression, and synthesis of realistic facial deformations using polynomial displacement maps. Our method consists of an analysis phase where the relationship between motion capture markers and detailed facial geometry is inferred, and a synthesis phase where novel detailed animated facial geometry is driven solely by a sparse set of motion capture markers. For analysis, we record the actor wearing facial markers while performing a set of training expression clips. We capture real-time high-resolution facial deformations, including dynamic wrinkle and pore detail, using interleaved structured light 3D scanning and photometric stereo. Next, we compute displacements between a neutral mesh driven by the motion capture markers and the high-resolution captured expressions. These geometric displacements are stored in a polynomial displacement map which is parameterized according to the local deformations of the motion capture dots. For synthesis, we drive the polynomial displacement map with new motion capture data. This allows the recreation of large-scale muscle deformation, medium and fine wrinkles, and dynamic skin pore detail. Applications include the compression of existing performance data and the synthesis of new performances. Our technique is independent of the underlying geometry capture system and can be used to automatically generate high-frequency wrinkle and pore details on top of many existing facial animation systems.

Abhijeet Ghosh, Pieter Peers, Paul Debevec, Tim Hawkins, Sune Frederiksen: “Practical Modeling and Acquisition of Layered Facial Reflectance”

We present a practical method for modeling layered facial reflectance consisting of specular reflectance, single scattering, and shallow and deep subsurface scattering. We estimate parameters of appropriate reflectance models for each of these layers from just 20 photographs recorded in a few seconds from a single viewpoint. We extract spatially-varying specular reflectance and single-scattering parameters from polarization-difference images under spherical and point source illumination. Next, we employ direct-indirect separation to decompose the remaining multiple scattering observed under cross-polarization into shallow and deep scattering components to model the light transport through multiple layers of skin. Finally, we match appropriate diffusion models to the extracted shallow and deep scattering components for different regions on the face. We validate our technique by comparing renderings of subjects to reference photographs recorded from novel viewpoints and under novel illumination conditions.

Jacki Morie, Gustav Verhulsdonck: “Body/Persona/Action! Emerging non-anthropomorphic communication and interaction in virtual worlds”

Avatars are traditionally understood as representing their human counterpart in virtual contexts by closely mimicking their real world physical characteristics. A new approach to designing avatars around non-anthropomorphic (non-human) characteristics currently questions the use of anthropomorphic principles and expands the use of avatars for virtual world interaction and communication. This paper provides a brief history of non-anthropomorphic avatars, with a focus on exploring the current use of such avatars in virtual worlds. In order to explain the shift in degree of anthropomorphism, we discuss Goffman’s theory of symbolic interactionism, which holds that the self is constructed as a persona through social performance and relates identity to social behavior rather than appearance. Since non-anthropomorphic avatars are persistent characters engaged in a prolonged performance in virtual worlds, their use also may motivate emerging social mores, politics and ideologies. This paper argues that such avatar species create new social interactions and modes of communication that may signal interesting directions for future research.

Anton Leuski, David Traum: “A Statistical Approach for Text Processing in Virtual Humans”

We describe a text classification approach based on statistical language modeling. We show how this approach can be used for several natural language processing tasks in a virtual human system. Specifically, we show it can applied to language understanding, language generation, and character response selection tasks. We illustrate these applications with some experimental results.

David Traum, Anton Leuski, Antonio Roque, Sudeep Gandhe, David DeVault, Jillian Gerten, Susan Robinson, Bilyana Martinovski: “Natural Language Dialogue Architectures for Tactical Questioning Characters”

In this paper we contrast three architectures for natural language questioning characters. We contrast the relative costs and benefits of each approach in building characters for tactical questioning. The first architecture works purely at the textual level, using cross-language information retrieval techniques to learn the best output for any input from a training set of linked questions and answers. The second architecture adds a global emotional model and computes a compliance model, which can result in different outputs for different levels, given the same inputs. The third architecture works at a semantic level and allows authoring of different policies for response for different kinds of information. We describe these architectures and their strengths and weaknesses with respect to expressive capacity, performance, and authoring demands.

Ron Artstein, Jacob Cannon, Sudeep Gandhe, Jillian Gerten, Anton Leuski, David Traum, Joe Henderer: “Coherence of off-topic responses for a virtual character”

We demonstrate three classes of off-topic responses which allow a virtual question-answering character to handle cases where it does not understand the user’s input: ask for clarification, indicate misunderstanding, and move on with the conversation. While falling short of full dialogue management, a combination of such responses together with prompts to change the topic can improve overall dialogue coherence.

Patrick Kenny, Thomas Parsons, Skip Rizzo, Greg Reger, Caroly Pataki, Jeffery Sugar, Michele Pato, Cheryl St. George: “Virtual Patients for Future Leaders”

War is one of the most challenging environments that persons may experience. The cognitive, emotional and physical demands of combat environments place enormous stress on even the best-prepared military personnel. The OIF/OEF combat theatre, with its ubiquitous battlefronts, ambiguous enemy identification, and repeated extended deployments have resulted in a significant number of returning American SMs with PTSD and other mental disorders. As a result, military leaders and clinicians in training need to develop clinical skills for identifying potential stress related disorders. Although traditional approaches make use of standard clinic patients to teach, there is limited ability to evaluate skills in a systematic fashion. There is the concern related to the time and money needed to train those involved in the role play for standardized patients. Perhaps most difficult is the “standardization” of standardized patients—will they in fact consistently proffer psychometrically reliable and valid interactions with the training clinicians. Virtual Human technology has evolved to a point where researchers are developing mental health applications that make use of virtual standardized patients. These virtual patients are embodied characters that have the ability to recognize speech, respond to questions and generate verbal and non-verbal behavior. We have conducted several pilot studies with clinical residents at USC’s Keck School of Medicine and will describe the ongoing study and methodology of our virtual patient approach that allows novice mental health clinicians to conduct an interview with a character emulating PTSD. The paper will summarize the data from the studies and discuss the preliminary standardization of the interactions with the virtual patients. The underlying virtual patient technology will be described. Finally future work will be discussed and recommendations related to the ways in which these characters may enable future leaders to learn, train and win.

Louis-Philippe Morency: “Real-time Head Pose Estimation Using a Webcam”

Accurately estimating the person’s head position and orientation is an important task for a wide range of applications such as driver awareness and human-robot interaction. Over the past two decades, many approaches have been suggested to solve this problem, each with its own advantages and disadvantages. In this paper, we present a probabilistic framework called Monocular Adaptive View-based Appearance Model (MAVAM) which integrates the advantages from two of these approaches: (1) the relative precision and user-independence of differential registration, and (2) the robustness and bounded drift of keyframe tracking. In our experiments, we show how the MAVAM model can be used to estimate head position and orientation in real-time using a simple monocular camera. Our experiments on two previously published datasets show that the MAVAM framework can accurately track for a long period of time (>2 minutes) with an average accuracy of 3.9 degrees and 1.2in with an inertial sensor and a 3D magnetic sensor.

Ryan McAlinden, Matt Bosack, Adrian Macha, Esau Vargas, Tim Walker, John Mann, Julio de la Cruz: “Towards an Automated Pipeline for the Translation and Optimization of Geospatial Data for Virtual Environments”

The infusion of commercial game technology into U.S. Army training, simulation, and instructional domains has resulted in more immersive and engaging experiences for Soldiers to hone their skills. However, the influx of such technology comes at a significant cost, specifically in the creation of virtual environments in which these skills are simulated and practiced. Today’s typical commercial triple-A game title cost upwards of $40-$60M and four to six years to develop, much of which is spent on producing the digital assets used to populate the scene (models, animations, etc). Additionally, this content is often suited for a custom type of rendering technology, and often cannot be reused without significant manual modification. Unfortunately, the Army has neither the financial or personnel resources available to create such highly immersive, reusable virtual content, nor the time to invest when current operations call for training or simulation data in a matter of hours, not months or years. In this paper, we discuss a research initiative aimed at significantly reducing the time and cost for converting, optimizing, and enhancing existing geospatial data for today’s virtual environments. The goal is a completely automated process for ingesting existing military terrain data and outputting a technology-agnostic representation in less than 24 hours.

Ryan McAlinden, Andrew Gordon, H. Chad Lane, John Hart, Paula Durlach: “UrbanSim: A game-based instructional package for conducting counterinsurgency operations”

Operation Iraqi Freedom and Operation Enduring Freedom have identified the need for instructional and training solutions that develop the skills of Battalion and Brigade Commanders in formulating situational understanding in order to successfully lead operations in a counterinsurgency environment. In this paper we describe the UrbanSim Learning Package, a game-based instructional software suite for Commanders and their staffs for directing and coordinating full-spectrum operations where the stability component is predominant. We describe a formal instructional design approach to the development of this instructional software, which consists of a component that introduces key concepts in counterinsurgency operations and a component that allows students to develop their skills in a simulated counterinsurgency environment. We describe how intelligent automated tutoring is used to provide formative feedback to students in the practice environment, and discuss our approach to student performance assessment.

Training for Iraq, Virtually

A new game-based training application recently was adopted by the U.S. Army to enhance negotiation skills and cultural understanding of soldiers overseas with an instructional method that emphasizes experience over more conventional learning models.

USC Rossier School of Education professor Richard Clark, who also co-directs the Center for Cognitive Technology, teamed up with immersive technology experts at the Institute of Creative Technologies and gamers at the USC School of Cinematic Arts to build the tool, called ELECT BiLAT.

Read the USC News story.

Two ICT Projects Win Army Modeling and Simulation Awards for FY08

Under the theme “Pushing MS to the Tactical Edge,” James C. Cooke, Director, Army Modeling and Simulation, Office of the Deputy Chief of Staff, announced winners of the Army Modeling and Simulation Awards for FY08.  Two USC Institute for Creative Technologies projects were recipients of this award: Distribution Management Cognitive Trainer (DMCT) and Bi-lateral Negotiation (BiLAT) Simulation.

The DMCT team, including ICT, U.S. Army Research Development (RDECOM STTC) and Product Manager Battle Command Sustainment Support System (PdM BCS3), created a Personal Computer, game-based training tool, that is being used to train US Army logistical planners on exploiting the capabilities of the Battle Command Sustainment Support System, the capabilities of logistics information management systems, and training the key principals of Distribution Management to improve the analysis, mission planning, and decision making skills of tomorrow’s logistical leaders.

ELECT BiLAT, which also runs on a PC, provides students with a training environment to practice their skills in conducting meetings and negotiations in a specific cultural context. ELECT BiLAT, a simulation developed under the Learning with Adaptive Simulation and Training (LAST) Army Technology Objective (ATO), was a collaboration between the University of Southern California’s Institute for Creative Technologies (ICT), U.S. Army Research Institute for the Behavioral and Social Sciences (ARI), U.S. Army Research Laboratory Human Research and Engineering Directorate (ARL-HRED) and U.S. Army Research Development, and Engineering Command’s Simulation and Training Technology Center (RDECOM STTC). BiLAT was recently delivered to PEO STRI for broad use across the Army.

The Army Modeling and Simulation Awards Selection Board evaluated fifty-seven high quality nominations this year compared to seventeen last year. Cooke, in his announcement, commented that the nominations demonstrated that great strides are being made towards the goal of pushing M&S to the Tactical Edge, noting, “It is vital that we continue to provide the Army with the most effective M&S support now and in the future.”

The awards will be presented at the I/ITSEC conference in Orlando, FL on December 2nd, 2008.

Andrew Gordon, Reid Swanson: “Say Anything: A Massively Collaborative Open Domain Story Writing Companion”

Interactive storytelling is an interesting cross-disciplinary area that has importance in research as well as entertainment. In this paper we explore a new area of interactive storytelling that blurs the line between traditional interactive fiction and collaborative writing. We present a system where the user and computer take turns in writing sentences of a fictional narrative. Sentences contributed by the computer are selected from a collection of millions of stories extracted from Internet weblogs. By leveraging the large amounts of personal narrative content available on the web, we show that even with a simple approach our system can produce compelling stories with our users.

ICT Scientists Earn Appointments in USC’s Computer Science Department

Anton Leuski, an ICT research scientist working in the areas of interactive information retrieval, information organization and document visualization, and Pieter Peers, a senior computer scientist with the ICT Graphics Lab, were both named research assistant professors in the computer science department at USC.

Rich DiNinni, Millie Abell: “Educating Learners to be Adaptive, Agile Thinkers”

The military must produce leaders capable of operating within today’s rapidly changing environment, often with less time for schoolhouse instruction. For this reason, the Army is investigating strategies for efficiently educating adaptive, agile thinkers. This presentation offers three evidence-based strategies having the potential to make education more powerful: (1) rich context using immersive technology-based scenarios; (2) Guided Experiential Learning (GEL) design to ensure far transfer, and (3) learning self regulation skills for deeper understanding.

Patrick Kenny, Thomas Parsons, Skip Rizzo, Caroly Pataki, Jeffery Sugar, Michele Pato: “A Virtual Adolescent Patient with PTSD for Training Psychiatrists”

ICT Research Recognized at Two International Conferences

Two different ICT papers recently received awards for best paper. Work by ICT’s Louis-Philippe Morency was recognized at the International Conference for Multimodal Interfaces in Greece. His paper, “Context-based Recognition during Human Interactions: Automatic Feature Selection and Encoding Dictionary”, describes how contextual information from participants in a conversation can be used to improve gesture recognition. This marks the third time Morency has won the best paper award at this particular conference and also his third best paper award this year.  The paper was co-authored with former ICT intern Iwan de Kok and Jonathan Gratch, ICT’s associate director for virtual humans research.

ICT research scientist H. Chad Lane and colleagues were recognized for their work considering the use of virtual humans and an intelligent tutoring system for the teaching of cultural social conventions. Their paper, “Coaching Intercultural Communication in a Serious Game,” was named best paper at the 18th International Conference on Computers in Education in Taipei, Taiwan. Co-authors were Matthew Hays of UCLA and Mark Core, Dave Gomboc, Eric Forbell, Daniel Auerbach, and Milton Rosenberg of ICT.

Read Morency’s paper.

Read Lane’s paper.

H. Chad Lane, Mark Core, Dave Gomboc, Eric Forbell, Daniel Auerbach, Milton Rosenberg, Matthew Jensen Hays: “Coaching Intercultural Communication in a Serious Game”

We consider the use of virtual humans and an intelligent tutoring system (ITS) for the teaching of cultural social conventions. Learning occurs in a serious game that requires the learner to establish trust and reach agreements with virtual characters of a different culture. Our tutoring system provides culturally focused learning support during and after the meetings with these virtual characters. In a study intended to determine the effectiveness of the ITS, we found that guidance provided during meetings seemed to improve learners’ understandings of culturally-related “phases” in meetings (e.g., when to talk about business) as well as greater success in an unsupported posttest meeting, but with no overall increase in cultural understanding when compared with learning in passive and unguided conditions.

Dave Gomboc, H. Chad Lane, Daniel Auerbach, Milton Rosenberg, Mark Core, Ashish Karnavat: “An Intelligent Tutoring Framework for Simulation-based Training”

Truly generic, reusable intelligent tutoring software frameworks remain elusive. As part of our effort to develop ITSs for simulations, a software framework with minimal dependencies on domain specifics has emerged. Herein, we describe this framework, its functionality, components, configurability, and use of natural language generation.

The LA Times: ICT Virtual Patients to Train USC Social Work Students in New Military Specialization

ICT will be collaborating with the USC School of Social Work in a first-of-its-kind master’s specialization program to help prepare social workers for caring for the needs of soldiers, veterans and thier families. The Military Social Work Specialization will hold its first classes in fall 2009 and plans to use some of ICT’s immersive technologies to train social workers and treat patients.  ICT has created virtual patients to aid clinicians in their interviewing and diagnostic skills and also developed a virtual reality exposure therapy to treat soldiers and veterans suffering from PTSD.

Read the LA Times story.

Read the USC News story.

Steve Solomon, Jonathan Gratch, Vadim Bulitko, Michael Christopher van Lent: “Modeling Culturally and Emotionally Affected Behavior”

Culture and emotions have a profound impact on human behavior. Consequently, high-fidelity simulated interactive environments (e.g., trainers and computer games) that involve virtual humans must model socio-cultural and emotional effects on agent behavior. In this paper we discuss two recently fielded systems that do so independently: Culturally Affected Behavior (CAB) and EMotion and Adaptation (EMA). We then propose a simple language that combines the two systems in a natural way thereby enabling simultaneous simulation of culturally and emotionally affected behavior. The proposed language is based on matrix algebra and can be easily implemented on single- or multi-core hardware with an off-the-shelf matrix package (e.g., MATLAB or a C++ library). We then show how to extend the combined culture and emotion model with an explicit representation of religion and personality profiles.

Louis-Philippe Morency, Jonathan Gratch, Iwan de Kok: “Context-based Recognition during Human Interactions: Automatic Feature Selection and Encoding Dictionary”

During face-to-face conversation, people use visual feedback such as head nods to communicate relevant information and to synchronize rhythm between participants. In this paperwe describe how contextual information from other participants can be used to predict visual feedback and improve recognition of head gestures in human-human interactions. The main challenges addressed in this paper are optimal feature representation using an encoding dictionary and automatic selection of the optimal feature-encoding pairs. We evaluate our approach on a dataset involving 78 human participants. Using a discriminative approach to multi-modal integration, our context-based recognizer significantly improves head gesture recognition performance over a vision- only recognizer.

Jacki Morie: “The Performance of the Self and Its Effect on Presence in Virtual Worlds”

This paper discusses the many types of roles that people play within digital arenas such as online virtual worlds, and how those authored selves can enhance the sense of Self presence. Erving Goffman maintains that we play many roles in our everyday lives and that our identity is constantly being redefined by both aspects of a situation and the other people with whom we interact. With the explosion of online virtual worlds, the possibilities for such performances of self have multiplied. We now have more opportunities to explore aspects of our personalities including those that we might be reluctant to expose in real life situations. This is a new development for virtual reality: participants can create their appearance in online virtual worlds and become extremely connected to it. The potential for these personas to affect and enhance the sense of Presence should be addressed and both quantitative and qualitative methods developed to measure their effects.

Jacki Morie, Celia Pearce: “The uses of digital enchantment: Computer games as the new fairy tales”

In this paper we argue that digital games have come to fill the cultural niche traditionally occupied by fairytales, and that they are ideally suited to realize some of the unique characteristics of this genre of folklore and literature. Arguably one of the most influential authors on game narrative and genre, J.R.R. Tolkien wrote extensively about fairytales, authored fairytales and considered his great epic work of high fantasy, “The Trilogy of the Ring,” to be a fairy tale of sorts. He argued that fairytales were not about fairies per se but took place in the “realm of faerie,” the magical world that fairies inhabit. “The realm of fairy-story is wide and deep and high and filled with many things: all manner of beasts and birds are found there; shoreless seas and stars uncounted; beauty that is an enchantment, and ever-present peril; both joy and sorrow as sharp as swords.” [1] The “realm of faerie” provides a context for archetypal characters and narratives that express the inner life of the child and the process of transitioning to adulthood, a universal theme with has equal resonance with adults. In The Uses of Enchantment, controversial psychologist Bruno Betttelheim argues that “The motifs of fairy tales are experienced as wondrous because the child feels understood and appreciated deep down in his feelings, hopes, and anxieties, without these all having to be dragged up and investigated in the harsh light of a rationality that is still beyond him.” [2] “…the internal processes are externalized and become comprehensible as represented by the figures of the story and its events.” [3] These externalized processes can be seen in a wide range of digital games that put the player in the role of fairytale heroine, or more often, hero. Single-player adventure-style games such as the Zelda and Final Fantasy series, Ico, Shadow of the Collosus, Beyond Good and Evil, Okami and the Longest Journey series bring the unique affordances of the computer as a purveyor of magic to bear on this classic literary genre. Science fiction author Arthur C. Clark famously asserted that “Any sufficiently advanced technology is indistinguishable from magic.” [4] Frederick Brooks, in The Mythical Man-Month [5], brings another level of refinement to this by describing the alchemic conjuring qualities of the computer thusly: “One types the correct incantation on a keyboard and a display screen comes to life, showing things that never were nor could be.” Indeed even the nomenclature of MUDs, in which programmers are referred to as “wizards,” seems to confer this quality of magical enchantment to the very creators of games themselves. Given its propensity for magic, the computer is particularly well-suited as a means of expression for the fairytale genre, shifting the focus from empathy with a central character engaged in an epic journey, to endowing a player with the agency to fulfill his or her destiny. We see the trajectory of the “realm of faerie” in the tradition from Tolkien’s literary masterworks to the contemporary MMOG. Tolkien’s world formed the inspiration for the tabletop role-playing games of the seventies, particularly Dungeons and Dragons, which gave rise to the MUDs of the 1980s and finally the fully realized multiplayer 3D computer fantasy worlds of the 1990s to the present, and the recent release of Lord of the Rings Online. This instrumentalizaton of fantasy environments through mathematical constructs provided a vital transition for the fairytale genre from the world of words to the world of numbers, and hence the world of computers. Today, the fantasy worlds of Tolkien, as well as the new fairy tales of game developers, have been rendered in their full glory via the “correct incantation on a keyboard.” While it remains to be seen how or if these new digital fairytales will stand the tests of time as their literary counterparts have done, we argue that they fulfill a similar and vital role in providing today’s children a sense of ritual and power in their own hero’s journey from child to adulthood. References [1] Tolkien, J.R.R. (1966). The Tolkien Reader. New York: Ballantine. [2] Bettelheim, Bruno. (1975). The Uses of Enchantment: The Meaning and Importance of Fairy Tales. New York: Alfred K. Knopf. [3] Ibid [4] Clark, Arthur C. (1962). Profiles of the Future; an Inquiry into the Limits of the Possible. New York: Harper & Row. [5] Brooks, Frederick P. (1975). The mythical man month: Essays on software engineering. Reading, MA: Addison-Wesley.

ICT Researcher Wins Best Paper Award

ICT Research Scientist Louis-Philippe Morency received the best paper award at the 8th International Conference on Automatic Face and Gesture Recognition. The research, co-authored by UCSD’s Jacob Whitehill and Javier Movellan, advances the field of human-computer interaction by presenting a model for estimating head position and orientation in real-time.  Possible applications include novel computer input devices, head gesture recognition, driver fatigue recognition systems, attention awareness for intelligent tutoring systems and social interaction analysis.

Read the paper.

Visit the conference website.

PTSD Virtual Exposure Therapy Featured in German Science Magazine

ICT’s virtual reality exposure therapy was the subject of a lenghty article in the October issue of German science magazine “Bild der Wissenschaft”. Reporter Desiree Karge visited ICT and interviewed psychologist Skip Rizzo about the treatment. The story also includes general information about the history and focus of ICT.

Visit the magazine’s website.

Sudeep Gandhe, David DeVault, Antonio Roque, Ron Artstein, Anton Leuski, Jillian Gerten, David Traum, Bilyana Martinovski: “From Domain Specification to Virtual Humans”

We present a new approach for rapidly developing dialogue capabilities for virtual humans. Starting from domain specification, an integrated authoring interface automatically generates dialogue acts with all possible contents.These dialogue acts are linked to example utterances in order to provide training data for natural language understanding and generation. The virtual human dialogue system contains a dialogue manager following the information-state approach, using finite-state machines and SCXML to manage local coherence, as well as explicit modeling of emotions and compliance level and a grounding component based on evidence of understanding. Using the authoring tools, we design and implement a version of the virtual human Hassan and compare to previous architectures for the character.

Jonathan Gratch: “Research Workshop: Virtual Human Toolkit”

The Institute for Creative Technologies is organizing a small, invitation-only “Virtual Human School”, intended for graduate students, junior researchers and Army personnel. It is designed to provide a broad overview of relevant theories and practical tools needed to construct graphically-embodied conversational agents.

The workshop will consist of a set of lectures by leading researchers at the University of Southern California on speech and gesture recognition, dialogue management, emotional modeling and multi-modal behavior generation. It will also provide hands-on experience with a virtual human toolkit consisting of a variety of currently available tools for the construction of virtual interactive characters.

Subjects that will be covered include automated speech recognition, natural language understanding, verbal and non-verbal behavior, procedural animation and gestures, cognitive modeling, ontologies, data management, and large system integration.

ICT to Create Virtual Human Guides with Museum of Science, Boston

The collaboration receives $2 million NSF grant to build life-like computer characters showcasing museum exhibits and the technologies enabling these digital docents to think, feel and interact.  Visitors to the Museum of Science, Boston will soon receive personalized attention from guides who can ask and answer questions, direct them to specific areas, and assist them in understanding particular exhibits and how to use them.

These engaging educators won’t be real people but virtual ones – programmed by the University of Southern California Institute for Creative Technologies with such advanced artificial intelligence, natural language and computer animation systems that learning how they work will become an interactive exhibit unto itself.

Funded by a three-year $2 million grant from the National Science Foundation, the project includes the creation of life-sized computer generated characters who not only look human but have the ability to express knowledge, thoughts, feelings and even memories. Patrons may eventually be able to build relationships with these virtual humans, who will learn their names and remember their interests from previous museum visits.

Read the USC News story.

Louis-Philippe Morency, Jacob Whitehill, Javier Movellan: “Generalized Adaptive View-based Appearance Model”

Accurately estimating the person’s head position and orientation is an important task for a wide range of applications such as driver awareness and human-robot interaction.Over the past two decades, many approaches have been suggested to solve this problem, each with its own advantages and disadvantages. In this paper, we present a probabilistic framework called Generalized Adaptive Viewbased Appearance Model (GAVAM) which integrates the advantages from three of these approaches: (1) the automatic initialization and stability of static head pose estimation, (2) the relative precision and user-independence of differential registration, and (3) the robustness and bounded drift of keyframe tracking. In our experiments, we show how the GAVAM model can be used to estimate head position and orientation in real-time using a simple monocular camera. Our experiments on two previously published datasets show that the GAVAM framework can accurately track for a long period of time (>2 minutes) with an average accuracy of 3.5 and 0.75in with an inertial sensor and a 3D magnetic sensor.

Congresswoman Jane Harman Visits ICT

Congresswoman Jane Harman (D-Calif) visited ICT to learn more about our work using interactive characters, virtual reality and immersive technologies for training, education and health. ICT Executive Director Randall W. Hill, Jr. presented an overview of ICT projects and Harman viewed examples of our virtual human negotiation training scenario and Justina, the virtual patient developed to explore how interactive digital characters can be used to train mental health clinicians. She also asked questions of Sgt. Star, a life-sized virtual character with natural language and artificial intelligence capabilities and donned a virtual reality headset to experience the ICT-developed virtual reality exposure therapy for treating PTSD.  This was Congresswoman Harman’s second visit to ICT and she remains supportive of the work taking place here.

Anton Treskunov, Andrei Sherstyuk: “Fast Geometry Acquisition for Mixed Reality Applications Using Motion Tracking”

Mixing real and virtual elements into one environment often involves creating geometry models of physical objects. Traditional approaches include manual modeling by 3D artists or use of dedicated devices. Both approaches require special skills or special hardware and may be costly. We propose a new method for fast semi-automatic 3D geometry acquisition, based upon unconventional use of motion tracking equipment. The proposed method is intended for quick surface prototyping for Virtual, Augmented and Mixed reality applications where quality of visualization of objects is not required or is of low priority.

David Traum: “Research Workshop: Multi-modal Grounding for Virtual Humans and Robots”

Grounding is the process of adding to the “Common Ground” between interacting participants. There have been several approaches to these
issues, including computational models of grounding by way of dialogue moves such as acknowledgments and corresponding gestures (e.g. head nods). However, most of these models have focused on coordinating the addition of asserted propositions to the common ground.

There has been comparatively less focus on grounding of other kinds of information, such as planned action, visual information, emotion and attitudes, and physical and social relationships. As robots are populating the physical world, and virtual humans the social world, this kind of coordination between artificial agents and humans is crucial. Moreover, even when only humans are involved, non-traditional communication methods (e.g., radio and other communication devices, communication mediated by translators or visual aids) can require additional effort and provide additionalresources for reaching common ground.

In this workshop, we will bring together some of the leading scholars who are working on aspects of these problems, or on related issues that require solutions to these problems, and discuss current proposals and open problems that require further work.

San Diego Union Tribune Covers ICT’s Virtual Reality Exposure Therapy for Treating PTSD

Skip Rizzo was interviewed for this story which reports positive results for ICT’s virtual reality exposure therapy for treating PTSD. The San Deigo Naval Medical Center and Camp Pendleton both participated in studies testing the effectiveness of the treatment. In one of the studies, 16 of 20 patients no longer showed signs of PTSD after 10 sessions with the “Virtual Iraq” computer simulation, said Rizzo, the ICT psychologist who developed the program.

Read the story.

Watch In-Depth Coverage of the ICT Graphics Lab on Mahalo Daily

The work of Paul Debevec and the ICT Graphics Lab was featured on Mahalo Daily, an online video program about all things new and exciting across the web. Correspondent Leah D’Emilio visited ICT for a firsthand look at the latest techniques for creating and displaying photorealistic computer graphics of people, objects, and environments.

Watch the segment.

Thomas Parsons, Skip Rizzo: “Neuropsychological Assessment using the Virtual Reality Cognitive Performance Assessment Test”

The traditional approach to assessing neurocognitive performance makes use of paper and pencil neuropsychological assessments. This received approach has been criticized as limited in the area of ecological validity. The newly developed Virtual Reality Cognitive Performance Assessment Test (VRCPAT) focuses upon enhanced ecological validity using virtual environment scenarios to assess neurocognitive processing. The VRCPAT battery and a europsychological assessment were conducted with a sample of healthy adults. Findings suggest 1) good construct validity for the Memory Module; and 2) that increase in stimulus complexity and stimulus intensity can manipulate attention performance within the Attention Module.

Thomas Parsons, Patrick Kenny, Skip Rizzo: “Virtual Human Patients for Training of Clinical Interview and Communication Skills”

Although schools commonly make use of standardized patients to teach interview skills, the diversity of the scenarios standardized patients can characterize is limited by availability of human actors. Virtual Human Agent technology has evolved to a point where esearchers may begin developing mental health applications that make use of virtual reality patients. The work presented here is a preliminary attempt at what we believe to be a large application area. Herein we describe an ongoing study of our virtual patients. We present an approach that allows novice mental health clinicians to conduct an interview with virtual character that emulates 1) an adolescent male with conduct disorder; and 2) an adolescent female who has recently been physically traumatized.

ICT Research Leading to More Natural and Fluid Virtual Human Conversation Wins Award

ICT research scientist Louis-Philippe Morency won the best paper award at the highly selective Intelligent Virtual Agent’s Conference in Japan. In the paper, Morency and colleagues Iwan de Kok and Jonathan Gratch propose a new model for learning to recognize and generate meaningful multimodal behaviors from examples of face-to-face interactions including facial expressions, posture shifts, and other interactional signals. The work has importance not only as a means to improve the interactivity and expressiveness of virtual humans but as an fundamental tool for uncovering hidden patterns in human social behavior.  Two other USC papers were finalists for Best paper: Evaluation of Justina: A Virtual Patient with PTSD, Patrick Kenny, Thomas Parsons, Jonathan Gratch, Albert Rizzo and best student paper: The Relation between Gaze Behavior and Emotional State: An Empirical Study, Brent Lance, Stacy Marsella.

Read the paper.

Information About the Conference

Patrick Kenny, Thomas Parsons, Jonathan Gratch, Skip Rizzo: “Evaluation of Justina: A Virtual Patient with PTSD”

Recent research has established the potential for virtual characters to act as virtual standardized patients VP for the assessment and training of novice clinicians. We hypothesize that the responses of a VP simulating Post Traumatic Stress Disorder (PTSD) in an adolescent female could elicit a number of diagnostic mental health specific questions (from novice clinicians) that are necessary for differential diagnosis of the condition. Composites were developed to reflect the relation between novice clinician questions and VP responses. The primary goal in this study was evaluative: can a VP generate responses that elicit user questions relevant for PTSD categorization? A secondary goal was to investigate the impact of psychological variables upon the resulting VP Question/Response composites and the overall believability of the system.

Andrew Gordon, Reid Swanson: “Envisioning With Weblogs”

In this position paper we present a vision of how the stories that people tell in Internet weblogs can be used directly for automated commonsense reasoning, specifically to support the core envisionment functions of event prediction, explanation, and imagination.

Andrew Gordon, Reid Swanson: “Envisioning With Weblogs”

In this position paper we present a vision of how the stories that people tell in Internet weblogs can be used directly for automated commonsense reasoning, specifically to support the core envisionment functions of event prediction, explanation, and imagination.

Andrew Gordon, Reid Swanson: “Envisioning with Weblogs”

We present a vision of how the stories that people tell in Internet weblogs can be used directly for automated commonsense reasoning, specifically to support the core envisionment functions of event prediction, explanation, and imagination.

Andrew Gordon: “Story Management Technologies for Organizational Learning”

The stories told among members of an organization are an effective instrument for knowledge socialization, the sharing of experiences through social mechanisms. However, the utility of stories for organizational learning is limited due to the difficulties in acquiring stories that are relevant to the practices of an organization, identifying the learning goals that these stories serve, and delivering these stories to the right people and the right time in a manner that best facilitates learning. In this paper we outline a vision for story-based organizational learning in the future, and describe three areas where intelligent technologies can be applied to automate story management practices in support of organizational learning. First, we describe automated story capture technologies that identify narratives of people’s experiences within the context of a larger discourse. Second, we describe automated retrieval technologies that identify stories that are relevant to specific educational needs. Third, we describe how stories can be transformed into effective story-based learning environments with minimal development costs.

Andrew Gordon: “Story Management Technologies for Organizational Learning”

The stories told among members of an organization are an effective instrument for knowledge socialization, the sharing of experiences through social mechanisms. However, the utility of stories for organizational learning is limited due to the difficulties in acquiring stories that are relevant to the practices of an organization, identifying the learning goals that these stories serve, and delivering these stories to the right people and the right time in a manner that best facilitates learning. In this paper we outline a vision for story-based organizational learning in the future, and describe three areas where intelligent technologies can be applied to automate story management practices in support of organizational learning. First, we describe automated story capture technologies that identify narratives of people’s experiences within the context of a larger discourse. Second, we describe automated retrieval technologies that identify stories that are relevant to specific educational needs. Third, we describe how stories can be transformed into effective story-based learning environments with minimal development costs.

Sin-hwa Kang, Jonathan Gratch, Ning Wang, James H. Watt: “Agreeable people like agreeable virtual humans”

This study explored associations between the five-factor personality traits of human subjects and their feelings of rapport when they interacted with virtual agent or real humans. The agent, the Rapport Agent, responded to real human speakers’ storytelling behavior, using contingent, but only nonverbal feedback. We further investigated how interactants’ personalities were related to the three components of rapport: positivity, attentiveness, and coordination. The results revealed that more agreeable people showed strong self-reported rapport and weak behavioral-measured rapport in the disfluency dimension with the Rapport Agent, while showing no significant associations between agreeableness and both peoples’ self-reported rapport and the disfluency dimension with real humans. The conclusions provide fundamental data to further develop the rapport theory that would contribute to evaluating and enhancing the interactional fidelity of an agent on the design of virtual humans for social skills training and therapy.

Louis-Philippe Morency, Jonathan Gratch, Iwan de Kok: “Predicting Listener Backchannels: “A Probabilistic Multimodal Approach”

During face-to-face interactions, listeners use backchannel feedback such as head nods as a signal to the speaker that the communication is working and that they should continue speaking. Predicting these backchannel opportunities is an important milestone for building engaging and natural virtual humans. In this paper we show how sequential probabilistic models (e.g., Hidden Markov Model or Conditional Random Fields) can automatically learn from a database of human-to-human interactions to predict listener backchannels using the speaker multimodal output features (e.g., prosody, spoken words and eye gaze). The main challenges addressed in this paper are automatic selection of the relevant features and optimal feature representation for probabilistic models. For prediction of visual backchannel cues (i.e., head nods), our prediction model shows a statistically significant improvement over a previously published approach based on hand-crafted rules.

Martin van Velsen: “Towards real-time authoring of believable agents in interactive narrative”

In this paper we present an authoring tool called Narratoria that allows non-technical experts in the field of digital entertainment to create interactive narratives with 3D graphics and multimedia. Narratoria allows experts in digital entertainment to participate in the generation of story-based military training applications. Users of the tools can create story-arcs, screenplays, pedagogical goals and AI models using a single software application. Using commercial game engines, which provide direct visual output in a real-time feedback-loop, users can view the final product as they edit.

Mei Si, Stacy Marsella, David Pynadath: “Modeling appraisal in Theory of Mind Reasoning”

Cognitive appraisal theories, which link human emotional experience to their interpretations of events happening in the environment, are leading approaches to model emotions. Cognitive appraisal theories have often been used both for simulating “real emotions” in virtual characters and for predicting the human user’s emotional experience to facilitate human-computer interaction. In this work, we investigate the computational modeling of appraisal in a multi-agent decision-theoretic framework using Partially Observable Markov Decision Process-based (POMDP) agents. Domain-independent approaches are developed for five key appraisal dimensions (motivational relevance, motivation congruence, accountability, control and novelty). We also discuss how the modeling of theory of mind (recursive beliefs about self and others) is realized in the agents and is critical for simulating social emotions. Our model of appraisal is applied to three different scenarios to illustrate its usages. This work not only provides a solution for computationally modeling emotion in POMDPbased agents, but also illustrates the tight relationship between emotion and cognition — the appraisal dimensions are derived from the processes and information required for the agent’s decision-making and beliefmaintenance processes, which suggests a uniform cognitive structure for emotion and cognition.

Celso de Melo, Jonathan Gratch: “Evolving Expression of Emotions in Virtual Humans using Lights and Pixels”

Inspired by the arts, this paper addresses the challenge of expressing emotions in virtual humans using the environment’s lights and the screen’s pixels. An evolutionary approach is proposed which relies on genetic algorithms to learn how to map emotions into these forms of expression. The algorithm evolves populations of hypotheses, where each hypothesis represents a configuration of lighting and screen expression. Hypotheses are evaluated by a critic ensemble composed of artificial and human critics. The need for human critics is motivated by a study which reveals the limitations of an approach that relies only on artificial critics that follow principles from art literature. We also address the need for the model to improve with experience and to adapt to the individual, social and cultural values in the arts. Finally, a second study is described where subjects successfully evolved mappings for joy and sadness.

David Traum, Jonathan Gratch, Arno Hartholt, Stacy Marsella, Jina Lee: “Multi-party, Multi-issue, Multi-strategy Negotiation for Multi-modal Virtual Agents”

We present a model of negotiation for virtual agents that extends previous work to be more human-like and applicable to a broader range of situations, including more than two negotiators with different goals, and negotiating over multiple options. The agents can dynamically change their negotiating strategies based on the current values of several parameters and factors that can be updated in the course of the negotiation.We have implemented this model and done preliminary evaluation within a prototype training system and a three-party negotiation with two virtual humans and one human.

Army Team Demos ICT’s Sgt. Star Installation in Army Interactive Van

Sgt. Star, the life-sized virtual Army guide created by the ICT mixed reality research and development group, paid a visit to ICT in the 18-wheel Army Interactive Van that is one of his current homes. Personnel from the Army’s Interactive Van team led a demonstration of the Sgt.’s role as a fully engaging member of the van’s exhibition team. ICT’s virtual Sgt. Star answers questions on topics including Army careers, training, education and money for college. He can also handle queries about the technology behind his development and explain how his creation fits in with plans for future Army training environments. The stop at ICT was part of a West Coast road trip that also included visits to high schools and colleges in the region.

Virtual Reality as a Rehabilitation Tool

ICT research was highlighted in this Philidelphia Inquirer story about the research and use of virtual reality systems for stroke victims. The piece mentions ICT’s work using virtual reality therpaies for treating PTSD and as a rehabitation aide for regaining motor skills. The story also gives a broad overview of the growing research and application of using virtual reality for victims of stroke and traumatic brain injury.

Mark Bolas, Belinda Lange, Skip Rizzo, Ian Dallas, Anna Huerta: “Engaging breathing exercises: developing an interactive XNA-based air flow sensing and control system”

The aim of this project was to make breathing exercises for children with Cystic Fibrosis fun. We developed a prototype device that uses breathing to control specifically designed video games.

More Coverage for ICT’s Facial Animation Collaboration with Image Metrics

KCAL News featured a story about the Emily project which showcases ICT and Image Metrics’ advances in creating believable facial animation and tech writer David Pescovitz blogged about the project on boingboing.net.

boing.boing.net’s post.

Louis-Philippe Morency, Xu Sun, Daisuke Okanoharay, Jun’ichi Tsujii: “Modeling Latent-Dynamic in Shallow Parsing: A Latent Conditional Model with Improved Inference”

Shallow parsing is one of many NLP tasks that can be reduced to a sequence labeling problem. In this paper we show that the latent-dynamics (i.e., hidden substructure of shallow phrases) constitutes a problem in shallow parsing, and we show that modeling this intermediate structure is useful. By analyzing the automatically learned hidden states, we show how the latent conditional model explicitly learn latent-dynamics. We propose in this paper the Best Label Path (BLP) inference algorithm, which is able to produce the most probable label sequence on latent conditional models. It outperforms two existing inference algorithms. With the BLP inference, the LDCRF model significantly outperforms CRF models on word features, and achieves comparable performance of the most successful shallow parsers on the CoNLL data when further using part-ofspeech features.

William Swartout: “Toward the Holodeck: Integrating Graphics, AI, Entertainment and Learning”

Using the Holodeck from Star Trek: Next Generation as our inspiration, researchers at the USC Institute for Creative Technologies have been pushing back the boundaries of the possible with the goal of creating immersive experiences so compelling that people will react to them as if they were real. In this talk I will describe our research in photo-real computer graphics, interactive virtual humans and immersive virtual reality that moves us closer to realizing the vision of the Holodeck. I will also discuss how entertainment content in the form of engaging stories and characters can heighten these experiences, and how such experiences can be used for learning.

ICT Graphics Lab Collaboration Lauded for Advancing Realistic Facial Animation

A collaboration between Paul Debevec’s Graphics Lab and facial animation solutions provider Image Metrics has produced a computer-generated human character so convincing that it is easy to mistake her for being real.

Read the Daily Mail story.

Read VFXWorld’s coverage.

Open House Tour of ICT Graphics Lab for PROCAMS Attendees

PROCAMS attendees are invited to tour the USC ICT graphics lab. The PROCAMS workshop series serves as an annual gathering place for researchers and practitioners who use, build, and design projector-camera systems for a wide variety of applications and purposes. PROCAMS 2008 will co-located in Los Angeles with Siggraph 2008. Workshops will be held in Marina del Rey, near ICT.

Patrick Kenny, Thomas Parsons, Jonathan Gratch, Skip Rizzo: “Virtual Humans for Assisted Health Care”

There is a growing need for applications that can dynamically interact with aging populations to gather information, monitor their health care, provide information, or even act as companions. Virtual human agents or virtual characters offer a technology that can enable human users to overcome the confusing interfaces found in current human-computer interactions. These artificially intelligent virtual characters have speech recognition, natural language and vision that will allow human users to interact with their computers in a more natural way. Additionally, sensors may be used to monitor the environment for specific behaviors that can be fused into a virtual human system. As a result, the virtual human may respond to a patient or elderly person in a manner that will have a powerful affect on their living situation. This paper will describe the virtual human technology developed and some current applications that apply the technology to virtual patients for mental health diagnosis and clinician training. Additionally the paper will discuss possible ways in which the virtual humans may be utilized for assisted health care and for the integration of multi-modal input to enhance the virtual human system.

Andrew Gordon, Jerry R. Hobbs, Michael T. Cox: “Anthropomorphic self-models for metareasoning agents”

Representations of an AI agent’s mental states and processes are necessary to enable metareasoning, i.e. thinking about thinking. However, the formulation of suitable representations remains an outstanding AI research challenge, with no clear consensus on how to proceed. This paper outlines an approach involving the formulation of anthropomorphic self-models, where the representations that are used for metareasoning are based on formalizations of commonsense psychology. We describe two research activities that support this approach, the formalization of broad-coverage commonsense psychology theories and use of representations in the monitoring and control of object-level reasoning. We focus specifically on metareasoning about memory, but argue that anthropomorphic self-models support the development of integrated, reusable, broad-coverage representations for use in metareasoning systems.

Andrew Gordon: “Anthropomorphic Self-models for Metareasoning Agents”

Representations of an AI agent’s mental states and processes are necessary to enable metareasoning, i.e. thinking about thinking. However, the formulation of suitable representations remains an outstanding AI research challenge, with no clear consensus on how to proceed. This paper outlines an approach involving the formulation of anthropomorphic self-models, where the representations that are used for metareasoning are based on formalizations of commonsense psychology. We describe two research activities that support this approach, the formalization of broad-coverage commonsense psychology theories and use of representations in the monitoring and control of object-level reasoning. We focus specifically on metareasoning about memory, but argue that anthropomorphic self-models support the development of integrated, reusable, broad-coverage representations for use in metareasoning systems.

Steve Solomon, Jonathan Gratch, Vadim Bulitko, Michael Christopher van Lent: “Modeling Culturally and Emotionally Affected Behavior”

Culture and emotions have a profound impact on human behavior. Consequently, high-fidelity simulated interactive environments (e.g., trainers and computer games) that involve virtual humans must model socio-cultural and emotional affects on agent behavior. In this paper we discuss two recently fielded systems that do so independently: Culturally Affected Behavior (CAB) and EMotion and Adaptation (EMA). We then propose a simple language that combines the two systems in a natural way thereby enabling simultaneous simulation of culturally and emotionally affected behavior. The proposed language is based on matrix algebra and can be easily implemented on single- or multi-core hardware with a standard matrix package (e.g., MATLAB or a C++ library). We then show how to extend the combined culture and emotion model with an explicit representation of religion and personality profiles.

ICT’s Intelligent Forces Project Featured in Popular Mechanics

A story in the May issue of Popular Mechanics titled, “10 Genius Inventions We’re Still Waiting For,” describes the goal of ICT’s Intelligent Forces project, which is to use artificial intelligence to “simulate the behavior of civilian populations—and insurgent forces—on a citywide scale.” The story adds that researchers can even plug in different cultures. “In the coming decades, this type of technology could be used by urban planners, video-game designers and maybe even indecisive civilians who face tough life decisions,” according to the magazine.

Read the whole story.

Paul Debevec: “Electronic Theater Screening”

Mundos Digitales is pleased to announce the screening of the entire 2007 Electronic Theatre. The show will be introduced by SIGGRAPH 2007 Computer Animation Festival Chair Paul Debevec and shows in full high definition. The Electronic Theatre is the premier computer graphics film and video showcase that is part of the Computer Animation Festival each year at the SIGGRAPH conference.

ICT’s 3D Display Featured in ESPN The Magazine

A story exploring what the future of sports will look like calls the ICT Graphics Lab’s 3D Display, “something as fantastic and futuristic as anything in Tinseltown.” Graphics Lab Director Paul Debevec says in the next decade or two technology will allow for the projection of real-time 3-D transmissions of all your games above something resembling a coffee table.

Read the story.

Patrick Kenny, Thomas Parsons, Skip Rizzo, Caroly Pataki, Jeffery Sugar, Michele Pato, Cheryl St. George: “Virtual Justina: A PTSD Virtual Patient for Clinical Classroom Training”

The effects of trauma exposure manifest themselves in a wide range of symptoms: anxiety, post-traumatic stress disorder, fear, and various behavior problems. Effective interview skills are a core competency for the clinicians who will be working with children and adolescents exposed to trauma. The current project aims to improve child and adolescent psychiatry residents, and medical students’ interviewing skills and diagnostic acumen through practice with a female adolescent virtual human with post-traumatic stress disorder. This interaction with a virtual patient provides a context where immediate feedback can be provided regarding trainees’ interviewing skills in terms of psychiatric knowledge, sensitivity, and effectiveness. Results suggest that a virtual standardized patient can generate responses that elicit user questions relevant for PTSD categorization. We conclude with a discussion of the ways in which these capabilities allow virtual patients to serve as unique training tools whose special knowledge and reactions can be continually fed back to trainees. Our initial goal is to focus on a virtual patient with PTSD, but a similar strategy could be applied to teaching a broad variety of psychiatric diagnoses to trainees at every level from medical students, to psychiatry residents, to child and adolescent psychiatry residents.

Sudeep Gandhe, David Traum: “Evaluation Understudy for Dialogue Coherence Models”

Evaluating a dialogue system is seen as a major challenge within the dialogue research community. Due to the very nature of the task, most of the evaluation methods need a substantial amount of human involvement. Following the tradition in machine translation, summarization and discourse coherence modeling, we introduce the the idea of evaluation understudy for dialogue coherence models. Following (Lapata, 2006), we use the information ordering task as a testbed for evaluating dialogue coherence models. This paper reports findings about the reliability of the information ordering task as applied to dialogues. We find that simple n-gram co-occurrence statistics similar in spirit to BLEU (Papineni et al., 2001) correlate very well with human judgments for dialogue coherence.

Tim Peek, Sudeep Gandhe, David Maxwell Chickering: “Rapidly Deploying Grammar-Based Speech Applications with Active Learning and Back-off Grammars”

Grammar-based approaches to spoken lan-guage understanding are utilized to a great ex-tent in industry, particularly when developers are confronted with data sparsity. In order to ensure wide grammar coverage, developers typically modify their grammars in an itera-tive process of deploying the application, col-lecting and transcribing user utterances, and adjusting the grammar. In this paper, we ex-plore enhancing this iterative process by leve-raging active learning with back-off grammars. Because the back-off grammars expand coverage of user utterances, develop-ers have a safety net for deploying applica-tions earlier. Furthermore, the statistics related to the back-off can be used for active learning, thus reducing the effort and cost of data tran-scription. In experiments conducted on a commercially deployed application, the ap-proach achieved levels of semantic accuracy comparable to transcribing all failed utterances with 87% less transcriptions.

Antonio Roque, David Traum: “Degrees of Grounding Based on Evidence of Understanding”

We introduce the Degrees of Grounding model, which defines the extent to which material being discussed in a dialogue has been grounded. This model has been developed and evaluated by a corpus analysis, and includes a set of types of evidence of understanding, a set of degrees of groundedness, a set of grounding criteria, and methods for identifying each of these. We describe how this model can be used for dialogue management.

ICT’s Diane Piepol To Chair LA Chapter of ACM SIGGRAPH

Diane Piepol, project director at ICT’s Flatworld was elected chair of the Los Angeles chapter of ACM Siggraph for the 2008-2009 year. As a professional chapter of the Association for Computing Machinery’s Special Interest Group in Computer Graphics, their goal is to bring LA-area members in depth coverage of the latest developments and achievements in the field of computer graphics including animation, visual effects, motion capture, multi-media, Internet and web design, virtual reality, and others.

Visit the ACM SIGGRAPH LA Chapter website.

David Traum: “Computational Models of Non-cooperative Dialogue”

This talk will outline some cases of noncooperative communication behavior and computational dialogue mechanisms that can support these kinds of behavior, including generating, understanding, and deciding on strategies of when to engage in uncooperative behavios. Behaviors of interest include: unilateral topic shifts or topic maintenance avoidance, competition, unhelpful criticism, withholding of information or services, lying & deception, competition, antagonism, rejection of empathy.

Arno Hartholt, Thomas Russ, David Traum, Eduard Hovy and Susan Robinson at LREC 2008

Arno Hartholt presents ‘A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture; Arno Hartholt, Thomas Russ, David Traum, Eduard Hovy and Susan Robinson’

When dealing with large, distributed systems that use state-of-the-art components, individual components are usually developed in parallel. As development continues, the decoupling invariably leads to a mismatch between how these components internally represent concepts and how they communicate these representations to other components: representations can get out of synch, contain localized errors, or become manageable only by a small group of experts for each module. In this paper, we describe the use of an ontology as part of a complex distributed virtual human architecture in order to enable better communication between modules while improving the overall flexibility needed to change or extend the system. We focus on the natural language understanding capabilities of this architecture and the relationship between language and concepts within the entire system in general and the ontology in particular.

Arno Hartholt, David Traum, Susan Robinson, Thomas Russ, Eduard Hovy: “A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented Virtual Human Architecture”

When dealing with large, distributed systems that use state-of-the-art components, individual components are usually developed in parallel. As development continues, the decoupling invariably leads to a mismatch between how these components internally represent concepts and how they communicate these representations to other components: representations can get out of synch, contain localized errors, or become manageable only by a small group of experts for each module. In this paper, we describe the use of an ontology as part of a complex distributed virtual human architecture in order to enable better communication between modules while improving the overall flexibility needed to change or extend the system. We focus on the natural language understanding capabilities of this architecture and the relationship between language and concepts within the entire system in general and the ontology in particular.

Ron Artstein, Massimo Poesio: “Anaphoric annotation in the ARRAU corpus”

Arrau is a new corpus annotated for anaphoric relations, with information about agreement and explicit representation of multiple antecedents for ambiguous anaphoric expressions and discourse antecedents for expressions which refer to abstract entities such as events, actions and plans. The corpus contains texts from different genres: task-oriented dialogues from the Trains-91 and Trains-93 corpus, narratives from the English Pear Stories corpus, newspaper articles from the Wall Street Journal portion of the Penn Treebank, and mixed text from the Gnome corpus.

ICT’s Paul Debevec in VFX World Magazine

The work of Paul Debevec and the ICT Graphics Lab is featured in this visual effects, 3D animation and computer graphics magazine. Under the headline, “Captain of the Light”, the article shines a spotlight on Debevec and his work, calling ICT’s associate director for graphics research a “true pioneer in many technologies that, once discovered, have become part of the pallet of every 3D artist and digital filmmaker.”

Read the article. »

ICT’s PTSD Virtual Therapy Featured on NPR Programs

Skip Rizzo was interviewed on “Talk of the Nation” about his research. Rizzo has led the development of Virtual Iraq, an immersion therapy video game for psychologists treating veterans struggling with post-traumatic stress disorder. Skip and his work were alos featured on NPR’s “Bryant Park Project”.

Listen to the “Talk of the Nation” segment. »

Listen to the “Bryant Park Project” segment. »

Ron Artstein, Sudeep Gandhe, Anton Leuski and David Traum: “Field Testing of an Interactive Question-Answering Character”

We tested a life-size embodied question-answering character at a convention where he responded to questions from the audience. The character’s responses were then rated for coherence. The ratings, combined with speech transcripts, speech recognition results and the character’s responses, allowed us to identify where the character needs to improve, namely in speech recognition and providing off-topic responses.

Arno Hartholt, Thomas Russ, David Traum, Eduard Hovy and Susan Robinson: “A Common Ground for Virtual Humans: Using an Ontology in a Natural Language Oriented VH Architecture”

When dealing with large, distributed systems that use state-of-the-art components, individual components are usually developed in parallel. As development continues, the decoupling invariably leads to a mismatch between how these components internally represent concepts and how they communicate these representations to other components: representations can get out of synch, contain localized errors, or become manageable only by a small group of experts for each module. We describe the use of an ontology as part of a complex distributed virtual human architecture in order to enable better communication between modules while improving the overall flexibility needed to change or extend the system. We focus on the natural language understanding capabilities of this architecture and the relationship between language and concepts within the entire system in general and the ontology in particular.

Andrew Gordon, Jerry R. Hobbs: “The Deep Lexical Semantics of Emotions”

Jonathan Gratch, Nicole Novielli, Peter Carnevale: “Cooperation Attitude In Negotiation Dialogs”

We propose an annotation scheme for a corpus of negotiation dialogs that was collected in the scope of a study about the effect of negotiation attitudes and time pressure on dialog patterns.

David Traum, Joe Henderer, Susan Robinson: “What Would You Ask a Conversational Agent? Observations of Human-Agent Dialogues in a Museum Setting”

Embodied Conversational Agents have typically been constructed for use in limited domain applications, and tested in very specialized environments. Only in recent years have there been more cases of moving agents into wider public applications (e.g. Bell et al., 2003; Kopp et al., 2005). Yet little analysis has been done to determine the differing needs, expectations, and behavior of human users in these environments. With an increasing trend for virtual characters to ‘go public’, we need to expand our understanding of what this entails for the design and capabilities of our characters. We explore these issues through an analysis of a corpus that has been collected since December 2006, from interactions with the virtual character Sgt Blackwell at the Cooper Hewitt Museum in New York. The analysis includes 82 hierarchical categories of user utterances, as well as specific observations on user preferences and behaviors drawn from interactions with Blackwell.

Susan Robinson, David Traum, Midhun Ittycheriah, Joe Henderer: “What would you ask a conversational agent?”

Embodied Conversational Agents have typically been constructed for use in limited domain applications, and tested in very specialized environments. Only in recent years have there been more cases of moving agents into wider public applications (e.g. Bell et al., 2003; Kopp et al., 2005). Yet little analysis has been done to determine the differing needs, expectations, and behavior of human users in these environments. With an increasing trend for virtual characters to �go public�, we need to expand our understanding of what this entails for the design and capabilities of our characters. This paper explores these issues through an analysis of a corpus that has been collected since December 2006, from interactions with the virtual character Sgt Blackwell at the Cooper Hewitt Museum in New York. The analysis includes 82 hierarchical categories of user utterances, as well as specific observations on user preferences and behaviors drawn from interactions with Blackwell.

ICT Workshop on Dialogue Research

Other presenters include David Devault – Rutgers, David Herrera – UTEP, Raquel Fernandez – Stanford, Robin Cooper, Staffan Larsson – Goteborg Univeristy

ICT’s Virtual Humans and Real Researchers Featured on NBC News

Reporter Patrick Healy of Los Angeles NBC station KNBC reports on our virtual human work. The story features Sgt. Star, the SASO-EN negotiation scenario and our virtual patient. Patrick Kenny, Diane Piepol, Skip Rizzo and David Traum all explain our various projects, their impacts and potential.

See the segment.

ICT’s PTSD Treatment Featured in the New Yorker

This week’s New Yorker has a lengthy feature about ICT’s virtual reality PTSD treatment, with quotes and ancecdotes from ICT psychologist Skip Rizzo. The article also mentions ICT’s FlatWorld demo. Writer Sue Halpern visited ICT and several sites where the therapy is being tested. She reports that preliminary results are encouraging and that the virtual reality treatment may be more appealing and less stigmatizing than traditional talk therapy for soldiers seeking help.

Read the New Yorker story. »

Sin-hwa Kang, Jonathan Gratch, Ning Wang, James H. Watt: “Does Contingency of Agents’ Nonverbal Feedback Affect Users’ Social Anxiety?”

We explored the association between users’ social anxiety and the interactional fidelity of an agent (also referred to as a virtual human), specifically addressing whether the contingency of agents’ nonverbal feedback affects the relationship between users’ social anxiety and their feelings of rapport, performance, or judgment on interaction partners. This subject was examined across four experimental conditions where participants interacted with three different types of agents and a real human. The three types of agents included the Non-Contingent Agent, the Responsive Agent (opposite to the Non-Contingent Agent), and the Mediated Agent (controlled by a real human). The results indicated that people having greater social anxiety would feel less rapport and show worse performance while feeling more embarrassment if they experience the untimely feedback of the Non-Contingent Agent. The results also showed people having more anxiety would trust real humans less as their interaction partners. We discuss the implication of this relationship between social anxiety in a human subject and the interactional fidelity of an agent on the design of virtual characters for social skills training and therapy.

ICT’s Jacquelyn Ford Morie to Chair ACM SIGGRAPH Digital Arts Community Committee

Dr. Jacquelyn Ford Morie was named Chair of the ACM SIGGRAPH Digital Arts Community Committee. She will lead SIGGRAPH’s new efforts to address the creative spirit and needs of all artists in the digital arts community. The mission of this group is to foster the evolution of a strong digital arts community within the international ACM SIGGRAPH organization, and to promote a dialogue between visual artists and the larger SIGGRAPH community. It will host a central online site for artists to share resources, information, artwork and opportunities, providing a practical way for all ACM SIGGRAPH members to follow developments in the arts, stay connected, and identify potential collaborators.

Paul Debevec: “Building an Interactive 360º Autostereoscopic Light Field 3D Display”

This talk will present the hardware and software algorithms used to create USC ICT’s 3D display which is able to present interactive 3D graphics to multiple simultaneous viewers 360 degrees around the display. The display consists of a high-speed video projector, a spinning mirror covered by a holographic diffuser, and FPGA circuitry to decode specially rendered DVI video signals rendering up to 5,000 frames per second. The display can show both interactive OpenGL graphics using a custom vertex shader as well as 3D photographic imagery of real objects and people. To motivate the design of the display,the talk will describe a selection of other 3D projection techniques as well as related work in our laboratory for spinning-mirror-based multiview image acquisition. It will conclude by presenting future directions for advancing the capabilities of the display.

Paul Debevec: “Realistic Acquisition of Facial Geometry, Reflectance, and Motion”

This talk presents recent work in USC ICT’s graphics laboratory for acquiring and rendering realistic models of human faces, bodies, and performances. It overviews traditional image-based relighting techniques for matching the lighting on an actor’s performance to that of photographic background plates, and for creating 3D virtual actors using these techniques as seen in films such as “Spider Man 2” and “Superman Returns”. It describes initial work with our lab’s Light Stage 6 system to combine image-based relighting with free-viewpoint video to capture and render full-body performances. It also covers a newly developed 3D face scanning process that captures high-resolution skin detail by estimating surface orientations (normal maps) from the skin’s reflection of polarized spherical gradient illumination, and a new efficient skin rendering technique which uses independently measured surface normals for the specular and diffuse reflection components. It concludes with by presenting recent work to capture and synthesize high-resolution geometry and reflectance of real-time facial performances.

Paul Debevec: “Demonstration: Interactive 360º Light Field Display”

The Graphics Lab at the University of Southern California has designed an easily reproducible, low-cost 3D display system with a form factor that offers a number of advantages for displaying 3D objects in 3D. The display is:

autostereoscopic – requires no special viewing glasses
omnidirectional – generates simultaneous views accomodating large numbers of viewers
interactive – can update content at 200Hz
The system works by projecting high-speed video onto a rapidly spinning mirror. As the mirror turns, it reflects a different and accurate image to each potential viewer. Our rendering algorithm can recreate both virtual and real scenes with correct occlusion, horizontal and vertical perspective, and shading.

While flat electronic displays represent a majority of user experiences, it is important to realize that flat surfaces represent only a small portion of our physical world. Our real world is made of objects, in all their three-dimensional glory. The next generation of displays will begin to represent the physical world around us, but this progression will not succeed unless it is completely invisible to the user: no special glasses, no fuzzy pictures, and no small viewing zones.

Researchers Nominated for Best Paper Award

The paper on ICT’s SmartBody research project is one of three papers nominated for best paper at the upcoming AAMAS conference (Autonomous Agents and Multi-Agent Systems), the premiere Agent conference in Portugal, May 12 – 16. The paper, “SmartBody: Behavior Realization for Embodied Conversational Agents”, was authored by Marcus Thiebaux, Andrew Marshall, Stacy Marsella and Marcelo Kallmann.

Patricia Lange, PhD: “Morality, Anonymity, Learning, and Friendship in a Fractalized World”

From online text games to YouTube, ethnographic insights have empirically challenged common sense assumptions about mediated interaction. These common sense notions include: 1) anonymity is the source of hostile interaction online; 2) children do not learn much by playing games or exchanging pointless, degraded, viral videos on YouTube; 3) the public/private dichotomy has collapsed in a post-Fordist, Internet era, such that notions of privacy are untenable. First, the talk will demonstrate that in many cases it is concerns about the known moral positions of interlocutors rather than pure anonymity that drives online hostility. Second, it will also show that children may learn a lot by participating in online groups, playing games, and circulating personal videos. Specifically, people learn how to accomplish forms of self-directed learning, how to create a mediated self-presentation in an increasingly wired world, and how to evaluate media by learning how it is put together. Finally, the talk will also illustrate how the study of video sharing practices has revealed more nuanced understanding of the public/private dichotomy. Applying theories from semiotics, it argues that the public/private dichotomy is actually a fractalized relationship, and it proposes intermediate forms that I call “publicly private” and “privately public” forms of video sharing. Based on ten years of ethnographic research that combines observation, participation, interviews, and analysis of discourse and artifacts, this talk will re-examine oft-cited conclusions about mediated interaction and will provide suggestions for future research in these areas.

ICT Computer Scientist Honored for Bright Ideas

Abhijeet Ghosh, a computer scientist with the Graphics Lab at USC’s Institute for Creative Technologies, has been named the winner of the 2007 Alain Fournier Thesis Award. The prize recognizes the top Ph.D dissertation in the area of computer graphics earned at a Canadian university last year.

In announcing the award, the committee cited Ghosh’s outstanding, novel and significant research contributions to the field of computer graphics.

Ghosh’s dissertation focused on the realistic rendering pipeline – the steps involved in measuring, computing, and displaying properties of light and reflectance in order to make convincing digital copies of real world objects.  His innovations allow for more accurate modeling so that computer generated images can appear more real.

“We congratulate Abhijeet on this tremendous honor and feel fortunate to have him as part of our team,” said Randall W. Hill Jr, executive director of ICT. ”The fact this top international talent has chosen to continue his pioneering work here at USC is a testament to the cutting edge and relevant work being performed at the ICT Graphics Lab.”

Alain Fournier is considered the father of computer graphics in Canada and was a leading international figure in the field. He began the computer graphics program at the University of British Columbia, where Ghosh earned his doctorate. After Fournier’s death in 2000, this award was established to “celebrate his life, to commemorate his accomplishments and honor his memory.” Ghosh is the third annual winner.

“Alan Fournier is a very inspirational figure in the field and it is an honor to be associated with an award with his name on it”, said Ghosh, who joined Paul Debevec at the ICT Graphics Lab in the fall.

Ghosh had completed an internship here in 2003 and credits his earlier time at ICT for shaping his award-winning dissertation.

“Being here changed my research direction,” he said. “I had been working in a different area but became fascinated with the discipline of realistic rendering, making beautiful images based on physics and optics.”

The ICT Graphics Lab develops lighting techniques (involving photographic and computer-based methods) to enable virtual reality simulations and their characters to look more real. In addition to lighting techniques, the Graphics Lab also works on methods to digitize light reflectance properties of objects.  The lab has collaborated with Hollywood film producers and visual effects supervisors to create realistic lighting for computer graphics characters in such movies as The Matrix and SpiderMan 2.

“Among the accomplishments in his thesis, Abhijeet’s work presents a fundamentally new way of measuring how surfaces reflect light to and from all directions—surfaces such as car paint, wood, brushed metal, leaves, and skin—making it much easier to digitize realistic materials to be used in simulations, visual effects, and interactive entertainment,” said Paul Debevec, director of the ICT Graphics Lab.

Here at ICT, Ghosh is now focusing on realistic modeling of scenes and people so that they can be successfully composited in virtual spaces. He says working with human faces is one of the big challenges.

“People are used to looking at faces, hence subtle differences stand out,” he said. “One really has it get it right otherwise people will recognize it is computer generated.”

But Ghosh has no need to worry. So far the only recognition he is getting is for the quality of his work.

Read the USC News story. »

Hsiu-Chin Lin: “Creating Agents that Entertain”

Game AI research is moving away from the creation of dominant computer agents to the idea of creating agents that focus on entertainment. The development of a realistic definition of entertainment, or “fun”, requires a qualitative measure on subjective assessments from a large, diverse field of participants. We present our work in the early stages of creating a web-based system for testing different opponent behaviors. We have augmented the Link4 game to compete again human players using three different opponent strategies. Human players will rank the entertainment value of these agents, as well as personal information that may affect their ranking. This paper describes the simple behaviors we have coded for testing purposes and the challenge of capturing user the experience while not detracting from it.

Paul Debevec: “New Techniques for Acquiring, Rendering, and Displaying Human Performances”

This talk presents recent work for acquiring, rendering, and displaying photoreal models of people, objects, and dynamic performances. It overviews image-based lighting techniques for photorealistic compositing and reflectance acquisition techniques which have been used to create realistic digital actors in films such as “Spider Man 2” and “Superman Returns”. It also describes initial work with the ICT Graphics Lab’s Light Stage 6 system to combine image-based relighting with free-viewpoint video to capture and render full-body performances. It will also cover a new 3D face scanning process that captures high-resolution skin detail by estimating surface orientation from the skin’s reflection of polarized spherical gradient illumination. The talk concludes by describing a new 3D display that leverages 5,000 frames per second video projection to show autostereoscopic, interactive 3D imagery to any number of viewers simultaneously.

Executive Director Randall W. Hill Jr. Talks about ICT on Radio Legend Michael Jackson’s Program

Randall W. Hill, Jr spoke about ICT’s work in virtual reality and PTSD exposure therapy on a radio talk show about some of the difficulties soldiers face upon returning from Iraq. The program aired live during the 10:00 hour of the Michael Jackson show on Los Angeles radio station KGIL 1260.

ICT’s PTSD Therapy Wins Major European Virtual Reality Award

ICT’s virtual reality exposure therapy treatment for PTSD was honored with the Best Medical Application award at the 10th Virtual Reality International Conference in Laval, France.

Visit the conference website.

Paul Debevec: “Rendering for an Interactive 360º Light Field Display”

Computer graphics have transformed many aspects of life. But while a great deal of imagery is modeled and rendered in 3D, almost all of it is shown on a 2D display such as a computer monitor, movie screen, or television. Researchers at the USC Institute for Creative Technologies (ICT) along with their collaborators have devised a reproducible, low-cost 3D display system that requires no special glasses, and is viewable from all angles by multiple users. This system allows computer generated 3D objects to be seen in new ways, and will impact the future of interactive systems.

The Interactive 360 degree Light Field Display (3D Display) was demonstrated at Siggraph 2007, and won the award for “Best Emerging Technology.” Siggraph is the pre-eminent conference for computer graphics and interactive technologies, and the Emerging Technologies exhibit is in collaboration with the International Conference on Virtual Reality. The award brought with it an invitation to show the 3D Display technology at the 10th Virtual Reality International Conference in Laval, France.

The research team includes Paul Debevec and Andrew Jones from the ICT, Ian McDowall from Fakespace Labs, Inc., Hideshi Yamada from Sony Corporation and Mark Bolas from the USC School of Cinematic Arts. Debevec is Associate Director for Graphics Research at the ICT, and is a pioneer in the field of realistic and innovative/interactive computer images.

Jillian Gerten: “Linguistics at ICT and Technologies that Use Linguistics”

This talk focuses on natural language dialogue projects at ICT.  It also discusses linguistic-related education, skills, and tasks required to create successful voice interaction technologies.

USC’s ICT and Cinematic Arts Photosharing Project in the New York Times

A recent story featured Viewfinder, a new photo-placing system developed by the USC School of Cinematic Arts’ Interactive Media Division and the USC Institute for Creative Technologies. The system, created with a grant from Google, allows people to place images into 3-dimensional mapping software like Google Earth at the proper location and at the original angle at which they were taken. A stated goal of the team, which includes ICT’s Paul Debevec, is that a 10-year-old should be able to find the pose of a photo in less than a minute.

Read the New York Times article.

ICT and USC Keck School Collaborate on Virtual Patient Project

ICT and USC’s Keck School of Medicine teamed up to apply ICT’s virtual human technology, orginally developed for Army training, to develop a program to enhance and elevate the quality of psychiatric education. Funded by a Provost seed grant for Teaching with Technology, “Justina”, a virtual patient suffering from Post Traumatic Stress Disorder, helps clinicians develop interviewing and diagnostic techniques. “Students can interact with Justina anytime and there is no limit on the amount of time they can practice with the program,” said Patrick Kenny, ICT research scientist. “Virtual patients also have the capability to present different conditions and the system records every evaluation. The data can be used to assess the students in a timely manner.”

Read the USC News story.

ICT Featured in Law Enforcement Technology Magazine

A story about using games and simulations for supplementing real-life training featured ICT’s law enforcement incident command training program. ICT Project Director Todd Richmond was interviewed about the training, which aims to create a sympathetic connection using virtual characters, allowing officers to actually see crowds and watch the police deal with them.  “What the entertainment industry has known for years is that emotional connections are how you hook people,” Richmond says. “As the personal involvement increases, it becomes more difficult for trainees to dismiss the whole exercise. If you can get people to care, you can get them to learn because they are tuned into it.”

Read the article.

Reid Swanson and Andrew Gordon: “StoryUpgrade: Finding Stories in Internet Weblogs:

The phenomenal rise of Internet weblogging has created new opportunities for people to tell personal stories of their life experience, and the potential to share these stories with those who can most benefit from reading them. One barrier to this new mode of storytelling is the lack of accessibility; existing Internet search tools are not tailored to the unique characteristics of this textual genre. In this paper we describe our efforts to develop a search engine specifically for the stories that appear in Internet weblogs, called StoryUpgrade. This application utilizes statistical text classification technologies to separate story content from other text in weblog entries, and facilitates searches for stories that are related to particular activities of interest.

Healing Psychological Trauma

Highlighted as the cover story in National Defense Magazine, soldiers relive sights and sounds of war in “Virtual Iraq” as a way to heal their mental scars. Researchers at the University of Southern California Institute for Creative Technologies have created a way to deal with PTSD by using gaming technology to put soldiers back in an environment where they can relive their experiences under the guidance and care of trained clinicians.

Read the article.

Reid Swanson: “Supporting Musical Creativity with Unsupervised Syntactic Parsing”

Music and language are two human activities that fit well with a traditional notion of creativity and are particularly suited to computational exploration. In this paper we will argue for the necessity of syntactic processing in musical applications. Unsupervised methods offer uniquely interesting approaches to supporting creativity. We will demonstrate using the Constituent Context Model that syntactic structure of musical melodies can be learned automatically without annotated training data. Using a corpus built from the Well Tempered Clavier by Bach we describe a simple classification experiment that shows the relative quality of the induced parse trees for musical melodies.

ICT’s PTSD Virtual Therapy Featured on NPR

ICT’s Skip Rizzo was interviewed on NPR’s “News & Notes” about developing and using virtual reality exposure therapy to treat PTSD.  “In order to treat PTSD effectively, we often rely on what’s called exposure therapy, and that is to get people to revisit the trauma and to emotionally process those memories,” Rizzo said. “So that is why we go to the trouble and expense to create a virtual Iraq environment in which the person can ever so gradually re-experience what they went through in Iraq, but in a safe, supportive environment in the clinician’s office,” he added. Roughly 20 sites across the U.S. are testing the therapy, the story reported.

Listen to NPR’s story.

Interactive Ethics Course Gets High Marks

An ICT collaboration with the USC Marshall School of Business leveraged ICT’s AXL application to bring issues of leadership and ethics to life for MBA students. The project is one of several ICT partnerships across USC.

Read the USC News story.

Ron Artstein: “The Arrau Corpus of Anaphoric Relations”

The Arrau corpus of anaphoric relations was created at the University of Essex between 2004 and 2007. It introduces an annotation scheme specifically targeted at marking two phenomena which had been difficult to annotate: ambiguous expressions which may refer to more than one object from previous discourse, and expressions which refer to abstract entities such as events, actions and plans. The corpus consists of a mixture of genres: task-oriented dialogues from the Trains-91 and Trains-93 corpus, narratives from the Gnome corpus and English Pear Stories corpus, and newswire from the Wall Street Journal portion of the Penn Treebank.

The corpus was created using the MMAX2 tool (Mueller and Strube 2003) which allows marking text units at different levels. Each noun phrase is marked as either anaphoric, discourse-new, or non-referential. Antecedents of anaphoric NPs are marked by pointers, and anaphoric ambiguity is indicated by multiple pointers from a single anaphoric expression (Poesio and Artstein 2005). Reference to an event, action or plan is marked by a pointer from the referring NP to the clause that introduces the abstract entity (Artstein and Poesio 2006).

The Arrau corpus differs from existing corpora like MUC and ACE since it marks all NPs, not only those that refer to entities of interest like people and organizations. The annotation is richer than a division of NPs into equivalence classes which refer to the same object, but it can be converted into equivalence classes by removing ambiguous links. The corpus has been used in the development of the anaphora resolution system at the 2007 Johns Hopkins summer workshop on natural language engineering; we plan to release it to the public in the coming months.

Jacquelyn Ford Morie: “Identity: Performing the Self in the Digital Domain”

This workshop will introduce participants to the many types of roles that people play within digital arenas such as social networks and online virtual worlds. Erving Goffman, noted sociologist, maintained that we play many roles in our everyday lives and that our identity is constantly being redefined by both the situation and the other people with whom we interact. With the coming of the digital age, the possibilities for such performances of self have exploded. We now have more opportunities to explore aspects of our personalities that we might be reluctant to expose in real life situations. Participants will discuss these topics, create online characters, and experiment with role playing in the digital domain.

ICT Research Scientist Earns Prestigious Nod

Louis-Philippe Morency is named one of “AI’s 10 to Watch” by IEEE Intelligent Systems, a leading artificial intelligence journal. The honor recognizes Morency’s multi-disciplinary work providing computers an understanding of gestures and movement.

Read the article in USC News.

Andrew Gordon: “Computational Approaches to the Analysis of Textual Stories”

Narratives of real-world experiences (stories) are an effective vehicle for sharing information about events, but also tell us about the expectations of storytellers that were challenged by their experiences. Story-based learning environments and training simulations can capitalize on this quality to craft fictional experiences that challenge the expectations of the learner in same way. However, the potential of stories in knowledge sharing applications is limited by the degree to which the analysis of stories can be automated. In this talk, I will describe our ongoing efforts to automate the collection, analysis, and application of textual stories in large corpora (the web) using natural language processing technologies at different levels of representational abstraction. Machine learning methods operating at the word-level are used to segment stories from other text genres. Lexical-semantic analysis is used to model sequences of events and support commonsense inference. Finally, I will motivate the need for deep semantic interpretation, and describe our approach to the use of formal commonsense theories in story understanding.

ICT’s PTSD Virtual Therapy in the Wall Street Journal

The Wall Street Journal’s online health blog features ICT’s virtual reality exposure therapy. The treatment was created here at ICT and is being tested and used nationwide to treat Iraq war veterans with post-traumatic stress disorder. The article includes an interview with ICT’s Skip Rizzo, who developed the program, and includes video examples of what the treatment looks like.

Read the WSJ article.

ICT’s Virtual Patients in Psychology Today

The February 2008 issue of Psychology Today features ICT virtual characters Justin and Justina, two life-like patients being developed to help teach students at USC’s Keck School of Medicine to interview patients and diagnose their conditions. Part of ICT’s Virtual Humans Project, the “patients” were created by ICT’s Patrick Kenny, Tom Parsons and Skip Rizzo, with collaborators in USC’s Department of Psychiatry.  Says Rizzo in the article, “It’s going to revolutionize the way therapists are trained.”

Read the Psychology Today story.

Jacquelyn Ford Morie Adds to ICT Duties with Posts at UK Univeristy

Dr. Jacquelyn Ford Morie was named Adjunct Faculty and Post-Doc Research Fellow with the University of East London’s SMARTLab Research Centre, London, UK. Her SMARTLab work focuses on online virtual communities and identity and performance in game environments. As part of her duties there she will also supervise several United States-based PhD candidates.

ICT Computer Graphics Featured in Nova Documentary

Photorealistic imagery of the Parthenon created by Paul Debevec and his team at the ICT Graphics Lab can be seen in an upcoming Nova documentary, “Secrets of the Parthenon.”

The footage was built from digital photography, lighting measurements and three-dimensional laser scans captured in Athens, London, Paris and Basel to accurately record the structure and appearance of the Parthenon and its sculptures. The final imagery – realistically depicting the structure as it stands today and its appearance during the golden age of ancient Greece – relied on numerous technological innovations in high dynamic range imaging, realistic lighting simulation and surface reflectance measurement. “Secrets of the Parthenon” will air Tuesday, Jan. 29 at 8 p.m. on PBS.

Learn more information about the PBS program.

Coverage of ICT Technology at Camp Pendleton

Various news outlets covered Camp Pendleton’s new Infantry Immersion Trainer, a high-tech training simulation that leveraged ICT’s expertise designing and developing immersive mixed-reality systems. The project incorporates ICT’s digital projection techniques, immersive audio and multi-sensory effects, including realistic odors and vibrating floors.

Read the Wired story.

Watch the video.

Andrew Gordon: “Workshop on Common Sense Knowledge and Goal-Oriented Interfaces”

When interacting with user interfaces, users have to bridge the gap between their goals expressed in mental terms and the interface’s structures and functions expressed in physical terms. This gap has been characterized as the “Gulf of Execution”, and it may imply significant cognitive processing on the users’ side in order for them to successfully accomplish their goals.

If user interfaces could understand, at a high-level, our goals, our problems, and our social procedures, users could have cognitively accessible, dynamic interfaces accommodating their unique needs, beyond the range of applications anticipated by the designers.

For computers to realize such a goal-oriented paradigm, they must have access to information about the world that human users take for granted. This information, which forms the basis for goal-directed computer interactions, is common sense knowledge. They must provide for algorithms and techniques that are capable of acquiring and structuring knowledge about user goals. They must have adequate means to represent knowledge about user goals in a way that allows for reasoning and inference about them. They must provide for user interfaces that effectively map users’ goals on the computer’s functionalities and structures.

This workshop has the goal of bringing together researchers from different domains that share an interest in common sense knowledge and goal-oriented knowledge representations in the context of intelligent user interfaces.

Video games: A learning tool for modern warfare?

Michael Peck from Training and Simulation Journal examined the use of video games to prepare soldiers for war and their return to civilian life in an article in the January 2 issue of USA Today.

Read the article.