By David Nelson, Director, Mixed Reality Lab / Modeling and Simulation
David Nelson is an award-winning Producer and Director, who has worked for 20th Century Fox, Disney, HBO and Sony Pictures Entertainment, winning a Hugo Award for Excellence in Television Documentary for “Positively Naked” (2005). As the Director of both the Mixed Reality Lab (MxR) and Modeling & Simulation Lab at USC’s Institute for Creative Technologies, Nelson has delivered multiple UX research-based technology prototypes, adaptive user interfaces, and immersive training systems, for clients including US Army, US Navy, Military and Sealift Command Fleet and ARL-West. He spearheaded a major interdisciplinary research study on the “Language of Immersion” and its impact on the future of narrative storytelling. He is currently leading an effort looking at immersive technologies applied toward next-generation command and control. Raised in Brooklyn, Nelson graduated from NYU Tisch School of the Arts, with a BFA, Dramatic Writing and Film Production. In this essay, Nelson talks about how both of his labs create immersive experiences which improve the acquisition of fundamental skills for the US military.
The future of military training and operations is being shaped today in our labs. As the Director of both the Mixed Reality Lab (MxR) and the Modeling & Simulation Lab at USC’s Institute for Creative Technologies (ICT), I’ve had the privilege of working at the frontier of immersive simulation technologies, developing cutting-edge tools that empower our warfighters. Through our work in virtual reality (VR), augmented reality (AR), and artificial intelligence (AI), we are creating the next generation of training environments and decision-support systems tailored to the evolving needs of the Department of Defense (DoD).
The Evolution of Immersive Technology at MxR
Established in 2008, the MxR Lab was founded with the mission of exploring emerging human-machine interactions, pushing VR and AR beyond the lab and into practical applications. Over the years, we have pioneered advancements in Extended Reality (XR) User Interfaces & Experiences, Spatial Data Visualization, Adaptive Human-Machine Interfaces (HMI), Generative AI Content Creation, and Immersive Story-Based Learning.
Our innovations have had a profound impact on both military and civilian sectors. For instance, our work directly influenced the development of high-end VR headsets like the Oculus Rift and inspired the creation of low-cost alternatives like Google Cardboard. Today, MxR continues to lead the charge in developing rapid-prototype solutions with a human-centered design ethos.
Modeling and Simulation for the Modern Warfighter
The Modeling & Simulation (M&S) Group at ICT is dedicated to creating engaging and informative experiences that enhance military training and operations. Our team comprises researchers, military domain experts, creative writers, and professional game designers, all collaborating to develop immersive simulations that are both educational and effective.
Through strategic partnerships across academia and the DoD, we have successfully transitioned advanced prototypes to military applications, including DisasterSim, as well as a range of strategic training games for the Naval Postgraduate School. Our work is helping the Army, Navy, and other defense agencies adopt cutting-edge technology to enhance warfighter readiness.
Training Through Storytelling: The Power of Narrative in Military Learning
At MxR, we believe that storytelling plays a critical role in training and knowledge retention.
One of our latest projects, the Leaders Enhanced & Applied Doctrine System (LEADS), is being evaluated at the Command and General Staff College to enhance comprehension of FM 3-0 Operations doctrine. By integrating interactive, scenario-based learning, we aim to accelerate mastery of doctrinal concepts, ensuring that commanders and their staff are better prepared for modern multi-domain operations.
Likewise, our Emergent Leader Immersive Training Environment (ELITE) has become a prerequisite for Army Warrant Officer School, reinforcing essential leadership skills through interactive role-playing scenarios.
Advancing Command and Control with Adaptive Interfaces
One of the most pressing challenges facing the military today is the integration of massive amounts of data into a coherent and actionable format. Recent feedback from Marine Forces Special Operations Command specialists emphasized the need for adaptive data fusion systems that streamline decision-making. In response, our Adaptive HMD Interfaces (AHMDi) research seeks to develop and define next-generation, head-mounted display (HMD) capability requirements by prototyping intelligent adaptive interfaces that enhance situational awareness and reduce cognitive load.
By investigating how adaptive interfaces can improve user experience and efficacy, we aim to create tools that support faster, more effective decision-making in high-stakes environments. Our research is particularly focused on the mission command environment, where leaders must process complex data sets while maintaining a clear operational picture.
Bridging Research and Real-World Application
Our work is not confined to theoretical exploration—it is actively shaping the future of military training and operations. At I/ITSEC 2024, we presented our Watercraft and Ship Simulator of the Future (WSSOF), a next-generation portable and modular simulation system designed to support the assessment of littoral, near shore maritime operations. This collaborative effort with the U.S. Army Engineer Research and Development Center (ERDC) Coastal Hydraulics Lab (CHL) will be a revolutionary mission rehearsal and planning tool.
Additionally, our ITEMS (Intelligent Training for Equipment Maintenance System) prototype created in collaboration with ICT’s Learning Sciences team is investigating AI-supported diagnostics to armored vehicle maintenance at the Army National Training Center.
These projects represent just a fraction of the groundbreaking work being done at MxR and M&S. With the support of our military partners, we continue to push the boundaries of what’s possible, ensuring that our warfighters have access to the most advanced training and decision-support systems available.
Future Visions
As we look to the future, immersive technology will only become more integral to military operations. Whether through adaptive training simulations, AI-driven decision-support systems, or innovative mixed-reality interfaces, our labs are committed to developing solutions that enhance warfighter effectiveness and preparedness.
We are standing at the precipice of a technological revolution—one that will redefine how our military trains, learns, and operates in the digital age. And while some of the Institute’s most exciting projects remain classified, one thing is certain: the work we do today will shape the battlefield of tomorrow.
//