By Dr. Jonathan Gratch, Director, Affective Computing Group
Affective Computing is a cross-disciplinary field of research aimed at creating machines that can recognize, interpret, simulate and stimulate human emotions. The study of computational emotion has undergone a substantial transformation over the past quarter-century. In my twenty-five years of research into this topic at USC’s Institute for Creative Technologies (ICT), the field has changed, evolved, and matured. This retrospective highlights some key research projects at ICT that have contributed to this evolution.
Computational Models of Emotion: The Mission Rehearsal Exercise
My mentor, Paul Rosenbloom, first encouraged me to incorporate emotion into his SOAR cognitive architecture, but this idea took fruition in the context of ICT’s first major research prototype, the Mission Rehearsal Exercise. In collaboration with Stacy Marsella, we addressed a fundamental question: how can virtual humans respond with appropriate emotions within an interactive training simulation? In a movie, emotional behaviors can be scripted in advance, but when training a young soldier how to make a decision, these simulated humans must respond appropriately to however the trainee chooses to act. Our approach involved developing EMA (Emotion and Adaptation), a computational architecture that integrated psychological appraisal theory (to decide what emotions make sense in a situation) with coping theory (which helps explain how emotions can lead to irrational decisions like wishful thinking).
EMA operated on the premise that emotions serve regulatory functions in intelligent behavior, influencing attention allocation, memory consolidation, and action selection. The model demonstrated that artificial agents equipped with emotional mechanisms exhibited more adaptive behavior than those relying solely on deliberative reasoning processes. This work challenged prevailing assumptions in artificial intelligence regarding the separation of rational and affective processing.
The theoretical contributions of this research program were recognized through several academic honors, including election as a Fellow of the Association for the Advancement of Artificial Intelligence and the Cognitive Science Society. More significantly, the work established computational emotion as a legitimate area of scientific inquiry within the broader artificial intelligence research community.
Social Signal Processing and Interpersonal Dynamics: The Virtual Rapport Project
While this early work established computational models of emotion as scientifically legitimate, it also raised a deeper question: would people actually respond to simulated emotions as if they were real? The Virtual Rapport Project was an attempt to answer this question by examining the social consequences of agent nonverbal communication and whether agents could establish rapport and its associated benefits, such as encouraging trust and self-disclosure. This work, conducted primarily with Louis-Philippe Morency, now at Carnegie Mellon University and Meta FAIR (formerly Facebook AI Research), investigated whether computational systems could learn to employ active listening behaviors to facilitate human disclosure.
Using data from experiments on human subjects, we developed machine learning models that accurately predicted how people use nonverbal feedback to create a sense of rapport. We then connected these models to a virtual human and examined if these would also create rapport. Our experiments revealed that subjects felt a social connection with the virtual human and provided significantly more personal information to virtual agents that exhibited appropriate nonverbal feedback—head nods, eye gaze patterns, and vocal acknowledgments—compared to agents lacking these capabilities. The finding suggested that relatively simple behavioral modifications could substantially alter the dynamics of human-machine interaction.
This research program expanded into the SimSensei project project, co-managed by Morency and Skip Rizzo. The initiative involved David Traum, Gale Lucas, Jill Boberg, Ed Fast, Arno Hartholt, and additional collaborators. In addition to linking with the Virtual Rapport Project, my work with Gale Lucas focused on data collection and validating the system in naturalistic settings.
The project yielded the DAIC-WOZ dataset, which has become a standard resource for depression detection research. The corpus contains over 140 clinical interviews and has been cited extensively in the literature on computational mental health assessment. The dataset’s utility extends beyond its original purpose, serving as a benchmark for multimodal signal processing and affective computing research.
Our most significant finding emerged from a controlled study with military personnel, conducted with Gale Lucas. The results indicated that soldiers disclosed more sensitive information to artificial interviewers than to human clinicians. This surprising result reshapes how we think about clinical assessment and therapeutic technology design.
Science Meets Hollywood: The Gunslinger Project
Not all significant research contributions manifest through traditional academic channels. Kim LeMasters (former ICT creative director) and I launched the Gunslinger Project to explore how Hollywood storytelling insights could transform human-computer interaction research. The aim was to break down silos between cinematic methods (which evoke strong emotion, but rely on linear storytelling) and interactive technology (which allows users freedom to choose but often leads to meandering and, frankly, boring narratives)
LeMasters, who had contributed to the screenplay of the original Westworld film, assembled a team that included Hollywood cinematographers and production designers. The cinematographer from My Darling Clementine provided insights on how staging and lighting can set the emotions and expectations for a scene. ICT’s (then) director, Dick Lindheim, coordinated with Paramount Pictures to construct an authentic period saloon on their backlot. ICT staff lent their appearances to key virtual characters, with Randy Hill (now ICT’s executive director) serving as the antagonist, “Reo Lane”, and Skip Rizzo, the gruff bartender. Mark Bolas developed the mixed reality components that enabled user interaction within this environment, and Arno Hartholt’s integration team stitched everything together.
The project served multiple research objectives simultaneously. It provided a testbed for evaluating narrative engagement in virtual environments, examined the role of environmental authenticity in user experience, and investigated the integration of multiple sensory modalities in interactive storytelling systems. A key insight was the use of story elements to create affordances for the user. For example, early tests showed that users felt lost with how to proceed. The storytellers on the team addressed this by having the character suggest actions that would drive the story forward along predesigned paths while creating the illusion of free will.
The work gained broader recognition beyond academic circles, culminating in my participation in a panel discussion with the creators of HBO’s Westworld series. The panel addressed fundamental questions regarding machine consciousness and emotional authenticity—issues that remain central to contemporary artificial intelligence research. Although it was framed through entertainment, Gunslinger demonstrated how emotion, story, and interaction intertwine—a theme that also proved invaluable for scientific exploration.
Virtual Humans Advance Affective Science
Beyond applications, research like the Rapport Project illustrated that virtual humans are powerful tools for basic science, enabling controlled experiments on emotion, trust, and decision-making. Over the years, we used these computational models to perform controlled experiments to investigate various aspects of human social cognition and emotional behavior. These systems enabled systematic manipulation of social and emotional variables while maintaining ecological validity.
For example, the Battleship paradigm (and subsequent incarnations like MouseWars), developed with substantial contributions from Jill Boberg, examined competitive dynamics from the board game Battleship to study emotional expression and decision making in strategic interactions. The system revealed patterns in how individuals regulate their emotions during competitive tasks and provided insight into the relationship between affect and strategic decision-making, and early insights that expressed emotion is often disconnected from underlying feelings (a theme that has become prominent in my more recent work).
The Impossible Anagrams paradigm, conceived by Gale Lucas, investigated personality factors in task persistence and emotional regulation. The experiment showed that traits that are assumed to be necessarily positive, like grit, can make it difficult to disengage from fruitless tasks. More broadly, this work highlighted the central role of emotion regulation in human decision-making.
The Split or Steal game, adapted from the British television program Golden Balls, created controlled conditions for studying the role of emotional expressions in trust, cooperation, and betrayal. The paradigm provided quantitative measures of risk assessment and social decision-making under conditions of uncertainty about partner intentions. This work led my PhD student, Celso de Melo, to develop “Reverse Appraisal Theory,” which highlights the role of emotional expressions in theory of mind reasoning, and formed a central role in the dissertations of Jessie Hoegen and Su Lei.
Most recently, the negotiation research program emerged from these foundational studies and has generated practical applications. One collaboration with Harbor UCLA focused on developing negotiation training systems for female physicians to address documented salary disparities in medical practice. Research into emotion and negotiation was central to four of my PhD students’ dissertations, including Johnathan Mell, Emmanuel J. Dorley, Kushal Chawla (co-advised with Gale Lucas), and James Hale. More recently, this work has contributed to discussions at the Harvard AI Summit regarding the integration of artificial intelligence in negotiation contexts. Together, these paradigms illustrated the potential of computational models not only to build technology but also to deepen our scientific understanding of human behavior.
Scholarly Development and Academic Service
Sustaining this progress required training new scholars and shaping the broader field, roles I took on through supervision, mentoring, and academic service. Over twenty-five years, I have supervised ten doctoral students through dissertation completion, served on fifty dissertation committees, and mentored seven postdoctoral researchers. These supervision activities have encompassed diverse research areas within affective computing, including computational models of emotion, multimodal signal processing, virtual human development, and human-robot interaction, and they have gone on to major research roles at government labs (Army Research Lab), universities (Florida, Central Florida), and corporate research labs (Google, Meta, Spotify). In addition, I helped to recruit and hire key ICT researchers, including Louis-Philippe Morency, Ning Wang, Skip Rizzo, Morteza Deghani, Gale Lucas, and Mohammad Soleymani.
Over the years, the Affective Computing Group has hosted forty-three visiting students and researchers from institutions in the US, Germany, the Netherlands, Korea, and Japan. These collaborations have contributed to the global development of affective computing research and have resulted in numerous joint publications and ongoing research partnerships.
An important aspect of research is growing one’s field. My proudest example of this is the founding of IEEE Transactions on Affective Computing, where I served as the first editor-in-chief. It has subsequently grown into one of the most impactful journals in computer science. I’ve also organized numerous conferences, including the Affective Computing conference and Intelligent Virtual Agents conferences, and served associate editor positions at multiple journals, and guest editorial roles for special issues addressing ethics in affective computing and artificial intelligence applications in emotion research.
Research Impact and Applications
Research only matters if it someday contributes to meaningful applications. The practical applications of my research span multiple domains. Depression detection algorithms developed from the DCAPS dataset are being evaluated in clinical settings for screening and assessment purposes. Rapport-building techniques identified in our virtual agent research have informed the design of conversational AI systems deployed in customer service and educational applications. The negotiation training systems have been adapted for use in professional development contexts, addressing documented inequities in salary negotiation outcomes. Mental health applications of our virtual human technology are being investigated for therapeutic intervention and assessment protocols.
Beyond direct applications, this work has contributed to theoretical understanding of emotion’s computational properties and its role in intelligent behavior. The integration of appraisal theory with computational architectures has influenced subsequent research in affective computing and cognitive modeling.
Methodological Contributions
Methodologically, the field benefited from new datasets, experimental paradigms, and shared platforms that became standard across research groups. The research programs described above have advanced methodological approaches within affective computing research. The development of the DAIC-WOZ corpus established protocols for collecting multimodal interaction data in clinical contexts and the CaSiNo and KODIS corpora for studying negotiation and conflict resolution. The virtual human platforms developed for these studies have been adopted by other research groups and have become standard tools for investigating human-computer interaction.
Experimental paradigms developed through our game-based studies have been replicated and extended by other researchers, contributing to standardized approaches for investigating trust, cooperation, and emotional regulation in controlled laboratory settings.
Theoretical Implications
This body of work has contributed to several theoretical advances in understanding computational emotion and its applications. The demonstration that emotional processes enhance rather than impair rational decision-making has influenced AI system design principles. The finding that humans may disclose more readily to artificial agents than human interlocutors has implications for theories of trust and social cognition.
The integration of computational emotion with virtual human technology has advanced understanding of the necessary and sufficient conditions for establishing rapport and trust in human-computer interaction. These insights have practical implications for designing more effective interactive systems across multiple application domains.
Conclusion
Looking back, computational emotion research has advanced theory and applications. Looking forward, its interdisciplinary nature will be essential for tackling questions of emotion, intelligence, and AI’s place in society. Twenty-five years of research in computational emotion has produced substantial advances in both theoretical understanding and practical applications. The work described above represents one perspective on the development of affective computing as a scientific discipline. The field continues to evolve as new technologies and methodological approaches become available, presenting opportunities for addressing fundamental questions about the nature of emotion, intelligence, and human-computer interaction.
The collaborative nature of this research, involving colleagues from computer science, psychology, and creative industries, has been essential to its success. The interdisciplinary approach required for advancing affective computing research will likely become increasingly important as the field addresses more complex questions about the role of emotion in artificial intelligence systems and their integration into human society.
//