By Dr. Volkan Ustun, Director, Human-inspired Adaptive Teaming Systems (HATs) Lab, and Research Scientist, ICT
Dr. Volkan Ustun is the Director of the Human-inspired Adaptive Teaming Systems (HATs) Lab and Research Scientist at ICT, focusing on multi-agent reinforcement learning, agent directed simulation, cognitive architectures, and artificial intelligence. Dr. Ustun joined ICT in 2012 after conducting postdoctoral research at Rice University and Georgia Tech, having received his PhD (Industrial and Systems Engineering) from Auburn University, following a MSc, Industrial Engineering from Orta Doğu Teknik Üniversitesi, Turkey.
Synthetic entities in military simulations often fall short of expectations. They rely on rule-based or reactive computational models, which are minimally intelligent and incapable of adapting based on experience. Even at that current level, they are costly to create and take time to develop.
To create more effective synthetic entities, we need adaptive models that demonstrate human-like behavior—entities that can communicate, perceive their environment, reason, and choose actions dynamically. Experiential learning is key to building such credible behavior; trying to program realism entirely through pre-defined rules would be highly impractical.
This is what we do at the Human-inspired Adaptive Teaming Systems (HATs) Lab.
We focus on creating autonomous synthetic characters—commonly referred to as Non-Player Characters (NPCs)—that are both aware of human trainees’ needs and capable of providing realistic, challenging learning experiences.
Our research at ICT combines Multi-agent Reinforcement Learning (MARL) with Graph Neural Networks (GNNs) and Large Language Models (LLMs), drawing inspiration from fields as diverse as operations research, cognitive science, game theory, and artificial intelligence.
This interdisciplinary approach enables us to tackle complex challenges and stay at the cutting edge of research.
From Cognitive Architectures to Reinforcement Learning
In the past, our lab relied heavily on cognitive architectures, like the Sigma Cognitive Architecture, to design synthetic minds for NPCs. However, as our focus shifted toward experiential learning, we began embracing machine learning techniques more. In addition to generating capable synthetic characters as opponents and teammates, our models can now act as interactive decision support systems, aiding commanders in military simulations via generating tactical recommendations and alternative courses of action. By integrating deep learning MARL into our research, we’ve expanded our capacity to train NPCs for adaptive and dynamic environments.
For instance, we’ve developed behavior models for various military training scenarios using geo-specific terrains and abstractions in Unity gaming engine. These environments allow us to conduct MARL experiments on realistic environments while incorporating critical domain knowledge, such as military hierarchies tactics. Such an integration not only accelerates our experiments but also enhances the realism of our synthetic agents.
Our work isn’t confined to theoretical exploration; we actively share our tools and findings. You can explore our sample simulation environments and libraries for MARL experiments on our public GitHub repository.
Key Research Highlights
Here are some recent papers from our lab that highlight our progress in the field:
1. Leveraging Organizational Hierarchy to Simplify Reward Design in Cooperative MARL
This study focuses on military training simulations, which are complex, hierarchical, and doctrine-driven. By incorporating organizational hierarchy into MARL, we streamlined reward engineering and improved team coordination. Using two-tiered commander-subordinate networks, we observed enhanced learning efficiency, with commanders prioritizing team goals and subordinates exploring more effectively under soft constraints.
2. Leveraging Graph Networks to Model Environments in RL
We explored the use of GNNs to model environments, improving policy performance in RL. Our experiments, conducted in a non-stationary, partially observable environment, demonstrated that GNNs could create better sparring partners for human trainees. These findings underscore the value of combining GNNs with RL in scenarios requiring adaptive behavior. Code available on GitHub.
3. Improving RL Experiments in Unity through Waypoint Utilization
MARL experiments can be computationally intensive, particularly in complex environments. We introduced a waypoint-based movement system in Unity to simplify these environments while maintaining strategic depth. Our experiments showed that waypoint navigation not only reduced computational costs but also facilitated better policy learning for heterogeneous team roles. Explore the project on GitHub.
4. Multi-agent Reinforcement Learning with a Scout Mission Scenario in RIDE
To address the lack of realistic Opposing Forces (OPFOR) behavior in military simulations, we trained RL agents to act as adversaries in a Scout Mission Scenario. The agent and environment representations presented in this approach have the potential to scale and significantly reduce the need for humans in OPFOR roles in interactive training simulations.
Looking Ahead
Our mission at the HATs Lab is to advance the frontier of synthetic character design, focusing on creating agents that are not just reactive but adaptive and human-inspired. By integrating MARL, GNNs, and domain-specific knowledge, we aim to revolutionize military simulations, providing trainees with environments that are as dynamic and challenging as real-world scenarios.
The journey from cognitive architecture to MARL has been transformative, and we’re excited about what lies ahead. Our work is a testament to the power of interdisciplinary research and collaboration. For those interested in contributing or learning more, I encourage you to explore our research and tools online. Together, we can shape the future of synthetic intelligence.
//