Institute for Creative Technologies

This paper presents a novel panning algorithm called Speaker-Placement Correction Amplitude Panning (SPCAP) which guarantees conservation of loudspeaker power output. The method is appropriate for any speaker arrangement (e.g. ITU 5.1, 10.2, etc.), and scales with the number of speakers. SPCAP works by correcting initial pan values based on speaker placement to achieve constant power output. Because panning occurs over an arbitrary number of speakers (i.e. is not pair-wise), SPCAP provides two significant advantages over discrete panning schemes. First, pan values for current and future surround-sound formats (e.g. 5.1 and 10.2) are guaranteed to conserve power under any lower-resolution setup, making dynamic up/down mixing in non-standard setups feasible. Second, SPCAP provides a framework for producing wide (non point-source) sounds.

Based on the assumption that a rational agent will adopt a plan that maximizes the expected utility, we present a utility-based approach to plan recognition problem in this paper. The approach explicitly takes the observed agent’s preferences into consideration, and computes the estimated expected utilities of plans to disambiguate competing hypotheses. Online plan recognition is realized by incrementally using plan knowledge and observations to change state probabilities. We also discuss the work and compare it with other probabilistic models in the paper.

Social judgment is a process of social explanation whereby one evaluates which entities deserve credit or blame for multiagent activities. Such explanations are a key aspect of inference in a social environment and a model of this process can advance several design compo-nents of multi-agent systems. Social judgment underlies social planning, social learning, natural language prag-matics and computational model of emotion. Based on psychological attribution theory, this paper presents a computational approach to forming social judgment based on an agent’s causal knowledge and communica-tive interactions with other agents.

Fluid simulations are notoriously difficult to predict and control. As a result, authoring fluid flows often involves a tedious trial and error process. There is to date no convenient way of editing a fluid after it has been simulated. In particular, the Eulerian approach to fluid simulation is not suitable for flow editing since it does not provide a convenient spatio-temporal parameterization of the simulated flows. In this research, we develop a new technique to learn such parameterization. This technique is based on a new representation, the Advected Radial Basis Function. It is a time-varying kernel that models the local properties of the fluid. We describe this representation and demonstrate its use for interactive three-dimensional flow editing.

Data-driven approaches have been successfully used for realistic visual speech synthesis. However, little effort has been devoted to real-time lip-synching for interactive applications. In particular, algorithms that are based on a graph of motions are notorious for their exponential complexity. In this paper, we present a greedy graph search algorithm that yields vastly superior performance and allows real-time motion synthesis from a large database of motions. The time complexity of the algorithm is linear with respect to the size of an input utterance. In our experiments, the synthesis time for an input sentence of average length is under a second.

In an unbounded physical domain, simulating a turbulent fluid on an Eulerian grid is rather tricky. Since it is difficult to predict the motion of the fluid, it is also difficult to guess which computational domain would allow the simulation of the fluid without crossing the computational boundaries. To address this dilemma, we have developed a novel adaptive framework where the simulation grid follows the motion of the flow. Our technique is based onn the principle of Galilean Invariance and the culling of simulation cells using a metric derived from continuative boundary conditions. We describe our framework and showcase its advantages over traditional techniques. Timing results and visual comparisons are presented.

In this work, we present a technique for capturing a time-varying human performance in such a way that it can be re-illuminated in postproduction.  The key idea is to illuminate the subject with a variety of rapidly changing time-multiplexed basis lighting conditions, and to record these lighting conditions with a fast enough video camera so that several or many different basis lighting conditions are recorded during the span of the final video’s desired frame rate.  In this poster we present two versions of such a system and propose plans for creating a complete, production-ready device.

We describe the emotion and dialogue aspects of the virtual agents used in the MRE project at USC. The models of emotion and dialogue started independently, though each makes crucial use of a central task model. In this paper we describe the task model, dialogue model, and emotion model, and the interactions between them.

We describe a dialogue evaluation plan for a multi-character virtual reality training simulation. A multi-component evaluation plan is presented, including user satisfaction, intended task completion, recognition rate, and a new annotation scheme for appropriateness. Preliminary results for formative tests are also presented.

This paper describes the development of a multi-modal corpus based on multi-party multi-task driven common goal oriented spoken language interaction.  The data consists of approximately 10 hours of audio human simulation radio data and nearly 5 hours of video and audio face-to-face sessions between human trainees and virtual agents.

This paper reviews nine available transcription and annotation tools, considering in particular the special difficulties arising from transcribing and annotating multi-party, multi-modal dialogue. Tools are evaluated as to the ability to support the user’s annotation scheme, ability to visualize the form of the data, compatibility with other tools, flexibility of data representation, and general user-friendliness.

This article details preliminary studies undertaken to integrate the most relevant advancements across multiple disciplines in an effort to construct a highly realistic neurosurgical simulator based on a distributed computer architecture. Techniques based on modified computational modeling paradigms incorporating finite element analysis are presented, as are current and projected efforts directed toward the implementation of a novel bidirectional haptic device.

This technical report elaborates on the technical details of the EMA model of emotional appraisal and coping. It should be seen as an appendix to the journal article on this topic (Gratch & Marsella, to appear)

This report describes the addition of an emotion dialogue to the Mission Rehearsal Exercise (MRE) system. The goal of the MRE system is to provide an immersive learning environment for army officer recruits. The user can engage in conversation with several intelligent agents in order to accomplish the goals within a certain scenario. Although these agents did already posses emotions, they were unable to express them verbally. A question - answer dialogue has been implemented to this purpose. The implementation makes use of proposition states for modelling knowledge, keyword scanning for natural language understanding and templates for natural language generation. The system is implemented using Soar and TCL. An agent can understand emotion related questions in four different domains, type, intensity, state, and the combination of responsible-agent and blameworthiness. Some limitations arise due to the techniques used and to the relative short time frame in which the assignment was to be executed. Main issues are that the existing natural language understanding and generation modules could not be fully used, that very little context about the conversation is available and that the emotion states simplify the emotional state of an agent. These limitations and other thoughts give rise to the following recommendations for further work: * Make full use of references. * Use coping strategies for generating agent’s utterances. * Use focus mechanisms for generating agent’s utterances. * Extend known utterances. * Use NLU and NLG module. * Use emotion dialogue and states to influence emotions. * Fix known bugs.

The virtual world of the Mission Rehearsal Exercise project is three dimensional, but the scenery is projected on a 2D screen. As the user moves around in the virtual theatre, the view on the scenery should change accordingly. The goal of this assignment is to gather experience on camera movement to make the projection on the screen realistic.

In this report, first we give a survey of the work in plan recognition field, including the evolution of different approaches, their strength and weaknesses. Then we propose two decision-theoretic approaches to plan recognition problem, which explicitly take outcome utilities into consideration. One is an extension within the probabilistic reasoning framework, by adding utility nodes to belief nets. The other is based on maximizing the estimated expected utility of possible plan. Illustrative examples are given to explain the approaches. Finally, we compare the two approaches presented in the report and summarize the work.

The location, condition, and number of the Parthenon sculptures present a considerable challenge to archeologists and researchers studying this monument.  Although the Parthenon proudly stands on the Athenian Acropolis after nearly 2,500 years, many of its sculptures have been damaged or lost. Since the end of the 18th century, its surviving sculptural decorations have been scattered to museums around the world.  We propose a strategy for digitally capturing a large number of sculptures while minimizing impact on site and working under time and resource constraints. Our system employs a custom structured light scanner and adapted techniques for organizing, aligning and merging the data. In particular this paper details our effort to digitally record the Parthenon sculpture collection in the Basel Skulpturhalle museum, which exhibits plaster casts of most of the known existing pediments, metopes, and frieze. We demonstrate our results by virtually placing the scanned sculptures on the Parthenon.

This paper presents an overview of techniques for generating photoreal computer graphics models of real-world places and objects. Our group’s early efforts in modeling scenes involved the development of Facade, an interactive photogrammetric modeling system that uses geometric primitives to model the scene, and projective texture mapping to produce the scene appearance properties. Subsequent work has produced techniques to model the incident illumination within scenes, which we have shown to be useful for realistically adding computer-generated objects to image-based models. More recently, our work has focussed on recovering lighting-independent models of scenes and objects, capturing how each point on an object reflects light. Our latest work combines three-dimensional range scans, digital photographs, and incident illumination measurements to produce lighting-independent models of complex objects and environments.

Social credit assignment is a process of social judgment whereby one singles out individuals to blame or credit for multi-agent activities.  Such judgments are a key aspect of social intelligence and underlie social planning, social learning, natural language pragmatics and computational models of emotion. Based on psychological attribution theory, this paper presents a preliminary computational approach to forming such judgments based on an agent’s causal knowledge and conversation interactions.

This paper presents a technique for estimating the spatially-varying reflectance properties of a surface based on its appearance during a single pass of a linear light source. By using a linear light rather than a point light source as the illuminant, we are able to reliably observe and estimate the diffuse color, specular color, and specular roughness of each point of the surface. The reflectometry apparatus we use is simple and inexpensive to build, requiring a single direction of motion for the light source and a fixed camera viewpoint. Our model fitting technique first renders a reflectance table of how diffuse and specular reflectance lobes would appear under moving linear light source illumination. Then, for each pixel we compare its series of intensity values to the tabulated reflectance lobes to determine which reflectance model parameters most closely produce the observed reflectance values. Using two passes of the linear light source at different angles, we can also estimate per-pixel surface normals as well as the reflectance parameters. Additionally our system records a per-pixel height map for the object and estimates its per-pixel translucency. We produce real-time renderings of the captured objects using a custom hardware shading algorithm. We apply the technique to a test object exhibiting a variety of materials as well as to an illuminated manuscript with gold lettering. To demonstrate the technique’s accuracy, we compare renderings of the captured models to real photographs of the original objects.

Think Like a Commander - Excellence in Leadership (TLAC-XL) is an application designed for learning leadership skills both from the experiences of others and through a structured dialogue about issues raised in a vignette. The participant watches a movie, interacts with a synthetic mentor and interviews characters in the story. The goal is to enable leaders to learn the human dimensions of leadership, addressing a gap in the training tools currently available to the U.S. Army. The TLAC-XL application employs a number of Artificial Intelligence technologies, including the use of a coordination architecture, a machine learning approach to natural language processing, and an algorithm for the automated animation of rendered human faces.

This paper presents a process for capturing spatially and directionally varying illumination from a real-world scene and using this lighting to illuminate computer-generated objects. We use two devices for capturing such illumination. In the first we photograph an array of mirrored spheres in high dynamic range to capture the spatially varying illumination. In the second, we obtain higher resolution data by capturing images with an high dynamic range omnidirectional camera as it traverses across a plane. For both methods we apply the light �02eld technique to extrapolate the incident illumination to a volume. We render computer-generated objects as illuminated by this captured illumination using a custom shader within an existing global illumination rendering system. To demonstrate our technique we capture several spatially-varying lighting environments with spotlights, shadows, and dappled lighting and use them to illuminate synthetic scenes. We also show comparisons to real objects under the same illumination.

The Sensory Environments Evaluation (SEE) Project seeks to formulate a new design methodology for virtual environments that utilize multiple sensory inputs to induce presence. Deviating from virtual reality’s twenty-year focus on photo-realism, our “feels-real” design alternative aims directly at understanding the interdependencies of sensory stimuli in their creation of mental constructs, and the subsequent degree of realism perceived.

The majority of research in the field of virtual reality to date has focused on increasing the fidelity of the environments created and trying to determine the quality of the participant experience. Efforts have been made to quantify such aspects, especially in regards to visuals and sound, and to a lesser extent to the user experience. Recent thinking has tended towards the assumption that ever-greater fidelity would ensure a better user experience. However, such emphasis on photo-realism and audio-realism does not take into account the collective results of our multimodal sensory inputs with their intertwined effects. Our design philosophy for the creation of virtual environments attempts to replicate the human experience, and asks the question: Is there an underlying fidelity of feels-real through which the quality of the participant experience could be improved?

Social credit assignment is a process of social judgment whereby one singles out individuals to blame or credit for multi-agent activities. Such judgments are a key aspect of social intelligence and underlie social planning, social learning, natural language pragmatics and computational models of emotion. Based on psychological attribution theory, this report presents a preliminary computational approach to forming such judgments using an agent’s causal knowledge and conversation interactions.

This paper presents automated methods for facilitating after action review in team training exercises. Much of the learning from team training arises from frank after-the-fact discussions of the exercise, combining individual attributions of blame or credit into a more objective view of what transpired.  These individual attributions are social judgments involving not only causality but also explanations of individual responsibility, free will and mitigating circumstances. Such judgments are a key aspect of social intelligence and underlie social planning, social learning, natural language pragmatics and computational models of emotion. Here we introduce a computational model of this judgment process based on psychological Attribution Theory and discuss its potential to facilitate after action review in team training.