Institute for Creative Technologies

Machinima is a low-cost alternative to full production filmmaking. However, creating quality cinematic visualizations with existing machinima techniques still requires a high degree of talent and effort. We introduce a lightweight artificial intelligence system, Cambot, that canbe used to assist in machinima production. Cambot takes a script as input and produces a cinematic visualization. Unlike other virtual cinematography systems, Cambot favors an offline algorithm coupled with an extensible library of specific modular and reusable facets of cinematicknowledge. One of the advantages of this approach tovirtual cinematography is a tight coordination between the positions and movements of the camera and the actors.

Although the representation of physical environments and behaviors will continue to play an important role in simulation-based training, an emerging challenge is the representation of virtual humans with rich mental models (e.g., including emotions, trust) that interact through conversational as well as physical behaviors. The motivation for such simulations is training soft skills such as leadership, cultural awareness, and negotiation, where the majority of actions are conversational, and the problem solving involves consideration of the emotions, attitudes, and desires of others.The educational power of such simulations can be enhanced by the integration of an intelligent tutoring system to support learners? understanding of the effect of their actions on virtual humans and how they might improve their performance. In this paper, we discuss our efforts to build such virtual humans, along with an accompanying intelligent tutor, for the domain of negotiation and cultural awareness.

Real time computer graphics are limited in that they can only be displayed on projection screens and monitors. Monitors and projection screens cannot be used in live fire training or scenarios in which the displays could be physically damaged by trainees. To address this issue, we have developed projection systems using computer vision based color correction and image processing to project onto non-ideal surfaces such as painted walls, cinder blocks, and concrete floors. These projector-camera systems effectively paint the real world with digital light. Any surface can become an interactive projection screen allowing unprepared spaces to be transformed into an immersive environment. Virtual bullet holes, charring, and cracks can be added to real doors, walls, tables, chairs, cabinets, and windows. Distortion correction algorithms allow positioning of projection devices out of the field of view of trainees and their weapons. This paper describes our motivation and approach for implementing projector-camera systems for use within the FlatWorld wide area mixed reality system.

ELECT BiLAT is a prototype game-based simulation for Soldiers to practice conducting bilateral engagements in a cultural context. The prototype provides students with the experience of preparing for a meeting including familiarization with the cultural context, gathering intelligence, conducting a meeting and negotiating when possible, and following up on meeting agreements as appropriate. The ELECT BiLAT architecture is based on a commercial game engine that is integrated with research technologies to enable the use of virtual human characters, scenario customization, as well as coaching, feedback and tutoring.

Because the prototype application is intended to be a learning environment, pedagogy has been central throughout development. The project followed a five-phase process: (1) analyze the training domain; (2) develop a story board prototype; (3) implement a computer version of the training prototype; (4) refine training objectives and link their conditions and standards to game activities; and (5) develop training support content for students, instructors, and training developers. The goal is an authorable game-based environment that uses the pedagogy of guided discovery for training Soldiers in the conduct of bilateral engagements within a specific cultural context.

We present an evaluation of a spoken dialogue system that engages in dialogues with soldiers training in an immersive Call for Fire (CFF) simulation. We briefly describe aspects of the Joint Fires and Effects Trainer System, and the Radiobot-CFF dialogue system, which can engage in voice communications with a trainee in call for fire dialogues. An experiment is described to judge performance of the Radiobot CFF system compared with human radio operators. Results show that while the current version of the system is not quite at human-performance levels, it is already viable for training interaction and as an operator-controller aid.

We are interested in the problem of understanding the connections between human activities and the content of textual information generated in regard to those activities. Massive online collaborative environments, specifically online virtual worlds, where people meet, exchange messages, and perform actions can be a rich source for such an analysis. In this paper we study one of such virtual worlds and the activities of its inhabitants. We explore the existing dependencies between the activities and the content of the chat messages the world?s inhabitants exchange with each other. We outline three experimental tasks and show how language modeling and text clustering techniques allow us to explore those dependencies successfully

In this paper, we describe Fourth Frame Forums, an application that combines traditional four-frame comic strips with online web-based discussion forums. In this application, users are presented with a four-frame comic strip where the last dialogue balloon of the fourth frame is left blank. By typing a statement into this dialogue balloon, the user creates a new discussion thread in the forum, where the user?s dialogue choice can be critiqued and discussed by other users of the forum. We argue that Fourth Frame Forums provide an elegant and cost-effective solution for online education and training environments for communities of learners. We provide examples from the domain of US Army leadership development, and compare Fourth Frame Forums to alternative methods of story-directed simulation and training.

We report data from 9 participants with right hemisphere stroke on a new virtual reality (VR) wheelchair navigation test designed to assess lateralized spatial attention and neglect. The test consists of a virtual winding path along which participants must navigate (or be navigated by an experimenter) as they name objects encountered along the way. There are 4 VR task conditions, obtained by crossing the factors array complexity (Simple, Complex) and Driver (Participant, Experimenter). Participants performed the VR task, a real-life wheelchair navigation task, and a battery of tests assessing arousal, visual attention under secondary task demands, and neglect. The VR test showed sensitivity to both array complexity and driver, with best performance occurring in the Experimenter-Navigated, Simple Array condition. The VR test also showed high correlations with the wheelchair navigation test, and these correlations were in many instances higher than those between traditional clinical neglect tests and the wheelchair navigation task. Moreover, the VR test detected lateralized attention deficits in participants whose performance was within the normal range on other neglect tests. We conclude that the VR task is sensitive to factors likely to affect the severity of neglect in the daily environment, and shows promise as an efficient, easily administered measure of real-life wheelchair navigation.

Effective face-to-face conversations are highly interactive. Participants respond to each other, engaging in nonconscious behavioral mimicry and backchanneling feedback. Such behaviors produce a subjective sense of rapport and are correlated with effective communication, greater liking and trust, and greater influence between participants. Creating rapport requires a tight sense-act loop that has been traditionally lacking in embodied conversational agents. Here we describe a system, based on psycholinguistic theory, designed to create a sense of rapport between a human speaker and virtual human listener. We provide empirical evidence that it increases speaker fluency and engagement.

A dungeon role playing game intended to induce emotions such as boredom, surprise, joy, anger and disappointment is introduced. From the preliminary study, facial expressions indicating boredom and anger were observed. Individual differences were found on appraisal and facial expression of surprise, joy and disappointment

This paper describes an international effort to unify a multimodal behavior generation framework for Embodied Conversational Agents (ECAs). We propose a three stage model we call SAIBA where the stages represent intent planning, behavior planning and behavior realization. A Function Markup Language (FML), describing intent without referring to physical behavior, mediates between the first two stages and a Behavior Markup Language (BML)describing desired physical realization, mediates between the last two stages. In this paper we will focus on BML. The hope is that this abstraction and modularization will help ECA researchers pool their resources to build more sophisticated virtual humans.

Recording how scenes transform incident illumination into radiant light is an active topic in computational photography. Such techniques make it possible to create virtual images of a person or place from new viewpoints and in any form of illumination.

Believable nonverbal behaviors for embodied conversational agents (ECA) can create a more immersive experience for users and improve the effectiveness of communication. This paper describes a nonverbal behavior generator that analyzes the syntactic and semantic structure of the surface text as well as the affective state of the ECA and annotates the surface text with appropriate nonverbal behaviors. A number of video clips of people conversing were analyzed to extract the nonverbal behavior generation rules. The system works in real-time and is user-extensible so that users can easily modify or extend the current behavior generation rules.

We consider the problem of designing virtual characters that support speech-based interactions in a limited domain. Previously we have shown that classification can be an effective and robust tool for selecting appropriate in-domain responses. In this paper, we consider the problem of dealing with out-of-domain user questions. We introduce a taxonomy of out-of-domain response types. We consider three classification architectures for selecting the most appropriate out-of-domain responses. We evaluate these architectures and show that they significantly improve the quality of the response selection making the user?s interaction with the virtual character more natural and engaging.

This paper presents an empirical qualitative analysis of eliciting, giving and receiving empathy in discourse. The study identifies discursive and linguistic features, which realize cognitive, emotive, parallel and reactive empathy and suggests that imitation, simulation and representation could be non-exclusive processes in Theory of Mind reasoning.

The integration of sophisticated inference-based techniques into natural language processing applications first requires a reliable method of encoding the predicate-argument structure of the propositional content of text. Recent statistical approaches to automated predicate- argument annotation have utilized parse tree paths as predictive features, which encode the path between a verb predicate and a node in the parse tree that governs its argument. In this paper, we explore a number of alternatives for how these parse tree paths are encoded, focusing on the difference between automatically generated constituency parses and dependency parses. After describing five alternatives for encoding parse tree paths, we investigate how well each can be aligned with the argument substrings in annotated text corpora, their relative precision and recall performance, and their comparative learning curves. Results indicate that constituency parsers produce parse tree paths that can more easily be aligned to argument substrings, perform better in precision and recall, and have more favorable learning curves than those produced by a dependency parser.

We present a dialogue manager for ‘Call for Fire’ training dialogues. We describe the training environment, the domain, the features of its novel information state-based dialogue manager, the system it is a part of, and preliminary evaluation results.

This article summarizes recent progress in developing a validated computational account of the cognitive antecedents and consequences of emotion. We describe the potential of this work to impact a variety of AI problem domains.

We present a dialogue manager for ‘Call for Fire’ training dialogues. We describe the training environment, the domain, the features of its novel information state-based dialogue manager, the system it is a part of, and preliminary evaluation results.

Interactive narrative systems are storytelling systems in which the user can influence the content or ordering of story world events.  Conceptually, an interactive narrative can be represented as a branching graph of narrative elements, implying points at which an interactive user?s decisions influence the content or ordering of the remaining elements.  Generative approaches to interactive narrative construct narrative at runtime or pre-construct on a per-session basis highly interactive branching narrative structures.  One generative approach ? narrative mediation ? represents story as a linear progression of events with anticipated user actions and system-controlled agent actions together in a partially-ordered plan.  For every possible way the user can violate the story plan, an alternative story plan is generated.  If narrative mediation is powerful enough to express the same interactive stories as systems that use branching narrative structures, then linear narrative generation techniques can be applied to interactive narrative generation.  This paper lays out this argument and sketches a proof that narrative mediation is at least as powerful as acyclic branching story structures.

Reflection is critically important for time-constrained training simulations that do not permit extensive tutor-student interactions during an exercise. Here, we describe a reflective tutoring system for a virtual human simulation of negotiation. The tutor helps students review their exercise, elicits where and how they could have done better, and uses explainable artificial intelligence (XAI) to allow students the chance to ask questions about the virtual human’s behavior

This paper details the demonstration of an annotation and affordance-based software model intended to introduce cultural and social influences into a non-player character’s (NPC) decision-making process. We describe how recent research has supported the need to begin incorporating the effects of culture into the interactive digital domain. The technical approach is presented that describes the software techniques for embedding and utilizing culturally-specific information inside of a virtual environment, as well as the design and implementation of a deterministic Markov Decision Process (MDP) to model the affects of culture on the AI.

Planning and execution systems have been used in a wide varietyof systems to create practical and successful automation. Theyhave been used for everything from performing scientific researchon the surface of Mars to controlling enemy characters in video games to performing military air campaign planning. After reviewing past work on these various planning and executionsystems, we believe that most lack one or more key componentscontained in another system. To enable future researchers to build more complete systems, and avoid possible serious system failure, we identify the major technical problems any implementer of such a system would have to face. In addition wecite recent solutions to each of these technical problems. We limit our focus to planning and execution for virtual worlds and theunique problems faced therein.

Simulation is a common feature in computer entertainment.  However, in computer games simulation and story are often kept distinct by interleaving interactive play and cut scenes.  We describe a technique for an interactive narrative system that more closely integrates simulation and storyline.  The technique uses a combination of semi-autonomous character agents and high-level story direction.  The storyline is decomposed into directives to character agents to achieve particular world states.  Otherwise, character agents are allowed to behave autonomously.  When the player?s actions create inconsistency between the simulation state and storyline, the storyline is dynamically adapted and repaired to resolve any inconsistencies.

As an increasing part of the Army’s mission involves establishing rapport with diverse populations, training interpersonal skills becomes critically important. Here we describe a “Rapport Agent” that senses and responds to a speaker’s nonverbal behavior and provide empirical evidence that it increases speaker fluency and engagement. We argue such agent technology has potential, both as a training system to enhance communication skills, and to assess the key factors that influence rapport in face-to-face interactions. We conclude by discussing ways the nonverbal correlates of rapport vary between Arabic and English speakers and discuss the potential of such technology to advance research and training into rapport in cross-cultural settings.

In this paper, we introduce a methodology for analyzing judgment opinions. We define a judgment opinion as consisting of a valence, a holder, and a topic. We decompose the task of opinion analysis into four parts: 1) recognizing the opinion; 2) identifying the valence; 3) identifying the holder; and 4) identifying the topic. In this paper, we address the first three parts and evaluate our methodology using both intrinsic and extrinsic measures.

This paper describes the virtual humans developed as part of the Mission Rehearsal Exercise project, a virtual reality-based training system. This project is an ambitious exercise in integration, both in the sense of integrating technology with entertainment industry content, but also in that we have joined a number of component technologies that have not been integrated before. This integration has not only raised new research issues, but it has also suggested some new approaches to difficult problems. We describe the key capabilities of the virtual humans, including task representation and reasoning, natural language dialogue, and emotion reasoning, and show how these capabilities are integrated to provide more human-level intelligence than would otherwise be possible.

The objective of this study was to test the efficacy and suitability of virtual reality (VR) as apain distraction for pediatric intravenous (IV) placement. Twenty children (12 boys, 8 girls) requiring IV placement for a magnetic resonance imaging/computed tomography (MRI/CT) scan were randomly assigned to two conditions: (1) VR distraction using Street Luge(5DT), presented via a head-mounted display, or (2) standard of care (topical anesthetic) with no distraction. Children, their parents, and nurses completed self-report questionnaires that assessed numerous health-related outcomes. Responses from the Faces Pain Scale--Revisedindicated a fourfold increase in affective pain within the control condition; by contrast, nosignificant differences were detected within the VR condition. Significant associations between multiple measures of anticipatory anxiety, affective pain, IV pain intensity, and measures of past procedural pain provided support for the complex interplay of a multimodalassessment of pain perception. There was also a sufficient amount of evidence supportingthe efficacy of Street Luge as a pediatric pain distraction tool during IV placement: an adequate level of presence, no simulator sickness, and significantly more child-, parent-, and nurse-reported satisfaction with pain management. VR pain distraction was positively endorsed by all reporters and is a promising tool for decreasing pain, and anxiety in childrenundergoing acute medical interventions. However, further research with larger sample sizesand other routine medical procedures is warranted.

This paper details the design and implementation of an embedded environmental framework that introduces cultural and social influences into a simulation agent’s decision-making process. We describe the current limitations associated with accurately representing culture in virtual environments and military simulations, and how recent research in other academic fields have enabled computational techniques to begin incorporating the effects of culture into AI and behavior subsystems. The technical approach is presented that describes the design and implementation of a hierarchical data model, as well as the software techniques for embedding culturally-specific information inside of a virtual environment. Finally, future work is discussed for developing a more comprehensive and standardized approach for embedding this culturally-specific information inside of the virtual domain.

Intelligent agents are typically situated in a social environment and must reason about social cause and effect. Such reasoning is qualitatively different from physical causal reasoning that underlies most intelligent systems. Modeling social causal reasoning can enrich the capabilities of multi-agent systems and intelligent user interfaces. In this paper, we empirically evaluate a computational model of social causality and responsibility against human social judgments. Results from our experimental studies show that in general, the model’s predictions of internal variables and inference process are consistent with human responses, though they also suggest some possible refinement to the computational model.

The authoring of fictional stories is considered a creative process.  The purpose of most story authoring is not to invent a new style or genre of story that will be accepted by the population but to invent a single narrative that is novel enough to be tellable.  Computational story generation systems are more limited than human authors in the space of narratives that can be considered because it is often the case that story generation systems are constrained to operate within a fixed representation of the story world.  These limitations can impact whether a story generation system is considered creative or not.  In this paper, we describe a story planning system, Fabulist.  Fabulist however is constrained by the world model input by the system user.  We present two algorithms that enable story planning systems such as Fabulist to break outside the bounds of the initial world model in order to search a larger space of narratives.

Video-capture virtual reality (VR) systems are gaining popularity as intervention tools. Todate, these platforms offer visual and audio feedback but do not provide haptic feedback. Wecontend that adding haptic feedback may enhance the quality of intervention for various theoretical and empirical reasons. This study aims to integrate haptic-tactile feedback into avideo capture system (GX VR), which is currently applied for rehabilitation. The proposedmulti-modal system can deliver audio-visual as well as vibrotactile feedback. The latter isprovided via small vibratory discs attached to the patient’s limbs. This paper describes thesystem, the guidelines of its design, and the ongoing usability study.

Post Traumatic Stress Disorder is reported to be caused by traumatic events that are outside the range of usual human experiences including (but not limited to) military combat, violent personal assault, being kidnapped or taken hostage and terrorist attacks. Initial data suggests that 1 out of 6 Iraq War veterans are exhibiting symptoms of depression, anxiety and PTSD. Virtual Reality (VR) exposure treatment has been used in previous treatments of PTSD patients with reports of positive outcomes. The aim of the current paper is to specify the rationale, design and development of a Virtual Iraq PTSD VR application that has been created from the virtual assets that were initially developed for a combat tactical training simulation, which then served as the inspiration for the X-Box game entitled Full Spectrum Warrior.

Post Traumatic Stress Disorder (PTSD) is reported to be caused by traumatic events that are outside the range of usual human experiences including (but not limited to) military combat, violent personal assault, being kidnapped or taken hostage and terrorist attacks. Initial data suggests that 1 out of 6 Iraq War veterans are exhibiting symptoms of depression, anxiety and PTSD. Virtual Reality (VR) exposure treatment has been used in previous treatments of PTSD patients with reports of positive outcomes. The aim of the current paper is to present the rationale, technical specifications, application features and user-centered design process for the development of a Virtual Iraq PTSD VR therapy application. The VR treatment environment is being created via the recycling of virtual graphic assets that were initially built for the U.S. Army-funded combat tactical simulation scenario and commercially successful X-Box game, Full Spectrum Warrior, in addition to other available and newly created assets. Thus far we have created a series of customizable virtual scenarios designed to represent relevant contexts for exposure therapy to be conducted in VR, including a city and desert road convoy environment. User-Centered tests with the application are currently underway at the Naval Medical Center--San Diego and within an Army Combat Stress Control Team in Iraq with clinical trials scheduled to commence in February 2006.

The University of Southern California and its collaborative research partner, Fakespace Labs, are participating in a number of research programs to invent and implement new forms of display technologies for immersive and semi-immersive applications. This paper briefly describes three of these technologies and highlights a few emerging results from those efforts. The first system is a rear projected 300 degree field of view cylindrical display. It is driven by 11 projectors with geometry correction and edge blending hardware. A full scale prototype will be completed in March 2006. The second system is a 14 screen projected panoramic room environment used as an advanced teaching and meeting space. It can be driven by a cluster of personal computers or low-cost DVD players, or driven by a single personal computer. The third is a prototype stereoscopic head mounted display that can be worn in a fashion similar to standard dust protection goggles. It provides a field of view in excess of 150 degrees.

This paper presents an activity-based framework for empirical discourse analysis of mitigation in public environments such as Swedish and Bulgarian courtroom examinations. Mitigation is defined as a pragmatic, cognitive and linguistic behavior the main purpose of which is reduction of vulnerability. The suggested framework consists of mitigation processes, which involve mitigating argumentation lines, defense moves, and communicative acts. The functions of mitigation are described in terms of the participants’ actions and goals separately from politeness strategies. The conclusions and observations address two things: issues related to the pragmatic theory of communication especially mitigation and issues related to the trial as a social activity. For instance, non-turn-taking confirmations by examiners are often followed by volunteered utterances, which in some cases may be examples of ‘rehearsed’ testimonies. At the same time the witnesses’ tendency to volunteer information even on the behalf of their own credibility indicates that they also favor pro-party testimonies. Despite the objective judicial role of the prosecutor or judge and/or despite the examiners accommodating style the verbal behavior of the witnesses exhibits constant anticipation of danger.

Contemporary technology urges us to believe that it improves communication. One result of such a belief is increased negativity towards “failure” in communication, which is cured only by more communication or by force. A second consequence is a focus on the view that language is a vehicle for transmission of thoughts and a dislocation of the view that language and communication offer opportunity for ethical manifestation. This article proposes that ethics emerges through and in language: beyond the contents delivered and the linguistic structure it enforces, language inspires the fundamental response-ability between self and Other. It is exactly in the breakdown of communication that we communicate through the ethics of a caress. The contemporary dichotomies describing communication and meaning are due to a conceptual fusion/confusion, which in its turn is due to the post-world-wars’ breathless desire for speedy progress and recovery.