Simulations, Architectures & Intelligent Agents Lab
Research Lead: Arno Hartholt
The Simulations, Architectures & Intelligent Agents Lab (SAIA) creates interactive, autonomous, and controllable agents, the worlds they inhabit, and the architectures and tools to build both.
The SAIA Lab hardens research software, integrates academic and industry technologies into shared R&D platforms, and designs, develops, and transitions simulation-based intelligent systems.
The lab maintains the Rapid Integration & Development Environment (RIDE) and the Virtual Human Toolkit. These open integration platforms support rapid prototyping and evaluation of systems ranging from embodied conversational agents for training, education, and health to real-time military simulations on 3D geospatial terrain. Deployment targets include desktop, web, mobile, AR and VR, with integrated support for local and cloud AI services.
Projects
Virtual Human Toolkit
A modular R&D platform for creating real-time embodied conversational agents – virtual humans (VH) – that perceive and interact with users both verbally and nonverbally. Freely available for non-commercial use. Collaboration with all ICT VH labs.
RIDE
A rapid prototyping testbed unifying military simulation technologies to accelerate integration and experimentation for training and simulation systems, including human-machine teaming, drones, and autonomous vehicles. Freely available under Government Purpose Rights (GPR). Collaboration with all ICT labs.
VITA
Battle Buddy
A mobile health companion prototype that uses virtual human interaction to support veteran wellness and help prevent suicide. Winner of the VA Mission Daybreak initiative.
Personalized Avatars
A research initiative developing high-fidelity digital double avatars that reflect individual soldiers’ attributes to study realism, engagement, and effectiveness in military simulation and training environments. Collaboration with ICT VHTL.
Previous Work and Collaborations
Modeling & Simulation
Design and development of agent-based models, simulation environments, and experimental platforms for studying and evaluating complex, integrated systems.
Hybrid Computing
Leveraging RIDE to stand up and evaluate an on-prem GeForce Now streaming server to support scalable, low-latency, multi-user training and experimentation. Collaboration with NVIDIA and MITRE, under DEVCOM Soldier Center STTC.
SAFARI
Multi-Domain Operations Electronic Warfare prototype that used RIDE to integrate Navy radar models with industry real-time visualization, developed by Unity, led by the Air Force Agency for Modeling and Simulation (AFAMS).
Generative Programming
DEVCOM Soldier Center STTC project to enable simulation-agnostic model reuse by representing behaviors and logic in canonical forms that can be automatically translated into executable models across multiple simulation environments, including RIDE.
TSS SimReady Behaviors
Developing agent behavior frameworks and content for the Synthetic Training Environment (STE) Training Simulation Software (TSS), including behavior trees and reinforcement learning. Collaboration with Cole Engineering Services, Inc. (CESI), and Dynamic Animation Systems (DAS), under DEVCOM Soldier Center STTC.
USMA Camouflage Study
Creation of visual stimuli in support of studying the effectiveness of various camouflage patterns. Collaboration with the U.S. Military Academy West Point, directed by the Office of the Secretary of Defense (OSD).
Wargaming Coalition
Exploring wargaming simulation and course-of-action (COA) analysis through WW2 scenarios in RIDE.
Human Dimension Avatar
2010 internal prototype with high-fidelity, real-time digital doubles of real people.
RIDE Rapid Avatar
2021 internal prototype to showcase avatars captured in 10 minutes on 3D geospatial terrain.
VH Fidelity Research
Using machine learning techniques to semi-automatically categorize and analyze the complete field of virtual human fidelity academic research.
Training
Virtual humans and simulation-based systems that support operational readiness, skill acquisition, and performance assessment.
Immersive Training Environment
Life-sized conversational and kinetic virtual human prototype developed for USMC Camp Pendleton to train Marines. Developed in collaboration with Dignitas Technologies.
De-Escalation of Violence for Police
Virtual Reality prototype to train agents in de-escalation techniques. Developed in collaboration with RAND and the Los Angeles Police Department (LAPD).
INOTS
Deployed Navy virtual human and intelligent tutoring system for instruction, practice and assessment of interpersonal communication skills for leadership and counseling.
ELITE
Deployed Army virtual human and intelligent tutoring system for instruction, practice and assessment of interpersonal communication skills for leadership, counseling, and SHARP.
VITA Magic Leap
Early VITA job interviewing AR prototype, developed with Magic Leap on their pre-release hardware, in collaboration with the Dan Marino Foundation.
Elect BiLat
A portable-PC based training program designed to provide students an immersive and compelling training environment to practice their skills in conducting meetings and negotiations in a specific cultural context.
STAT
Laptop training capability to help K-12 school administrators and psychologists practice conducting threat assessment interviews with students.
Miles
Virtual patient in support of training social work counselors.
Cicero
Virtual classroom with configurable audience to teach and research public speaking.
Conflict Resolution Agent
Negotiation testbed in support of negotiation and conflict resolution research.
Mission Rehearsal Exercise (MRE)
The original USC ICT VH system, a pioneering, award-winning, immersive simulation that combined AI and interactive storytelling to train soldiers in decision-making and leadership in a simulated, high-pressure conflict scenario.
Stability and Support Operations (SASO)
The follow-up to MRE: an advanced system of reasoning agents who exhibit emotions and basic reasoning skills, based on the SOAR cognitive architecture.
Health
Virtual humans, interactive simulations, and AI-driven systems for healthcare training, behavioral health, information dissemination, assessment, and clinical support.
SimSensei
A basic research clinical decision support tool and interactive virtual agent-based healthcare dissemination system that is able to recognize and identify psychological distress from multimodal signals.
SimCoach
Web-based virtual humans designed to assist service members in breaking down barriers to care.
DocOn
Proof of concept mobile app with a virtual doctor avatar providing patient information. Collaboration with the USC Center for Body Computing.
Standard Patient Studio
A free online community where medical students, residents and physicians can improve their interview and diagnostic skills with a Virtual Standardized Patient.
STRIVE
Explores the efficacy of VR-based immersion in preparing users for the psychological challenges of combat before deployment through a series of narrative episodes set in Afghanistan.
RAM Replay
VR testbed in support of a DARPA-funded project aimed at improving memory and recall. Collaboration with Hughes Research Laboratories.
mHealth Coach
Proof of concept mobile app aimed at coaching young adult cancer survivors. Collaboration with the USC mHealth Collaboratory.
Education
Intelligent tutoring systems, learning simulations, and conversational agents that enhance instruction and learner engagement.
Boston Museum of Science Museum Guides
Deployed life-sized virtual museum guides who promote STEM education.
Virtual Research Navigator
Tablet-based VH who educated children and their caregivers about clinical trials. Collaboration with Children’s Hospital Los Angeles (CHLA).
Canvas
Application developed for the Smithsonian Institute to semi-automatically generate and author educational content for classrooms.
Project Syria
Immersive journalism VR application that enables audiences to experience the plight of Syrian children at the civil war, led and designed by Nonny de la Peña.
Coach Mike
A virtual coach designed to be an enthusiastic proponent for STEM education as he helped museumgoers learn basic programming skills with a robot.
PAL3
A Navy meta-tutor drone that accompanies a student or professional throughout their career, providing advice and recommendations along the way.
Workforce Transformation
AI-enabled tools and simulation platforms that support workforce development, reskilling, and organizational adaptation.
Telecommuting Autonomous Personal Assistant (TAPA)
A prototype of an autonomous, personalized agent to represent a remote government worker in support of advanced telecommuting technologies evaluation. Performed for the Defense Information Systems Agency (DISA).
Kingsman Telepresence
Evaluation of immersive academic and industry telepresence technologies in support of Army planning and acquisition, for DEVCOM Soldier Center STTC.
BlueShark
Leveraged AR, VR, VH, and AI to create a vision for what future Navy workspaces and collaboration will look like.
PRIMER
A VR prototype designed to show the potential of immersive dialogue agents and virtual environments that adapt and respond to both direct verbal input and indirect emotional input.
Entertainment
Interactive experiences with virtual humans for games, theme parks, and museums.
Gunslinger
Combines VH technology with Hollywood storytelling and stagecraft, creating a mixed-reality experience where a single participant interacts with multiple virtual characters set in the Wild West.
Monticello
Internal project to showcase integrated, multi-platform virtual humans for 2016 hardware, including mobile, web, desktop, AR (both headset and mobile), and VR, featuring a virtual Thomas Jefferson.