Publications
Search
41.
Gurney, Nikolos; Pynadath, David V.; Miller, John H.
Willingness to work as a predictor of human-agent team success Journal Article
In: Front. Comput. Sci., vol. 7, pp. 1405436, 2025, ISSN: 2624-9898.
@article{gurney_willingness_2025,
title = {Willingness to work as a predictor of human-agent team success},
author = {Nikolos Gurney and David V. Pynadath and John H. Miller},
url = {https://www.frontiersin.org/articles/10.3389/fcomp.2025.1405436/full},
doi = {10.3389/fcomp.2025.1405436},
issn = {2624-9898},
year = {2025},
date = {2025-03-01},
urldate = {2025-04-15},
journal = {Front. Comput. Sci.},
volume = {7},
pages = {1405436},
abstract = {Research shows that the effectiveness of human-agent teams depends heavily on human team members' prior experiences, whether from direct teaming activities or relevant domain knowledge. While researchers have proposed various mechanisms to explain this relationship, we present a simpler alternative explanation: experience serves primarily as an indicator of a person's fundamental willingness to engage in teaming tasks. We introduce a measure called “willingness to work” that quantifies this underlying disposition. Our empirical analysis demonstrates that this straightforward metric robustly predicts human-agent team performance. Beyond its practical value as a predictive tool, this reconceptualization of the experience-performance relationship necessitates a fresh examination of existing findings in the field. The results suggest that a team member's basic willingness to invest effort may be more fundamental to success than previously recognized mechanisms.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Research shows that the effectiveness of human-agent teams depends heavily on human team members' prior experiences, whether from direct teaming activities or relevant domain knowledge. While researchers have proposed various mechanisms to explain this relationship, we present a simpler alternative explanation: experience serves primarily as an indicator of a person's fundamental willingness to engage in teaming tasks. We introduce a measure called “willingness to work” that quantifies this underlying disposition. Our empirical analysis demonstrates that this straightforward metric robustly predicts human-agent team performance. Beyond its practical value as a predictive tool, this reconceptualization of the experience-performance relationship necessitates a fresh examination of existing findings in the field. The results suggest that a team member's basic willingness to invest effort may be more fundamental to success than previously recognized mechanisms.
42.
Ustun, Volkan; Hans, Soham; Kumar, Rajay; Wang, Yunzhe
Abstracting Geo-specific Terrains to Scale Up Reinforcement Learning Miscellaneous
2025, (arXiv:2503.20078 [cs]).
@misc{ustun_abstracting_2025,
title = {Abstracting Geo-specific Terrains to Scale Up Reinforcement Learning},
author = {Volkan Ustun and Soham Hans and Rajay Kumar and Yunzhe Wang},
url = {http://arxiv.org/abs/2503.20078},
doi = {10.48550/arXiv.2503.20078},
year = {2025},
date = {2025-03-01},
urldate = {2025-04-15},
publisher = {arXiv},
abstract = {Multi-agent reinforcement learning (MARL) is increasingly ubiquitous in training dynamic and adaptive synthetic characters for interactive simulations on geo-specific terrains. Frameworks such as Unity's ML-Agents help to make such reinforcement learning experiments more accessible to the simulation community. Military training simulations also benefit from advances in MARL, but they have immense computational requirements due to their complex, continuous, stochastic, partially observable, non-stationary, and doctrine-based nature. Furthermore, these simulations require geo-specific terrains, further exacerbating the computational resources problem. In our research, we leverage Unity's waypoints to automatically generate multi-layered representation abstractions of the geo-specific terrains to scale up reinforcement learning while still allowing the transfer of learned policies between different representations. Our early exploratory results on a novel MARL scenario, where each side has differing objectives, indicate that waypoint-based navigation enables faster and more efficient learning while producing trajectories similar to those taken by expert human players in CSGO gaming environments. This research points out the potential of waypoint-based navigation for reducing the computational costs of developing and training MARL models for military training simulations, where geo-specific terrains and differing objectives are crucial.},
note = {arXiv:2503.20078 [cs]},
keywords = {},
pubstate = {published},
tppubtype = {misc}
}
Multi-agent reinforcement learning (MARL) is increasingly ubiquitous in training dynamic and adaptive synthetic characters for interactive simulations on geo-specific terrains. Frameworks such as Unity's ML-Agents help to make such reinforcement learning experiments more accessible to the simulation community. Military training simulations also benefit from advances in MARL, but they have immense computational requirements due to their complex, continuous, stochastic, partially observable, non-stationary, and doctrine-based nature. Furthermore, these simulations require geo-specific terrains, further exacerbating the computational resources problem. In our research, we leverage Unity's waypoints to automatically generate multi-layered representation abstractions of the geo-specific terrains to scale up reinforcement learning while still allowing the transfer of learned policies between different representations. Our early exploratory results on a novel MARL scenario, where each side has differing objectives, indicate that waypoint-based navigation enables faster and more efficient learning while producing trajectories similar to those taken by expert human players in CSGO gaming environments. This research points out the potential of waypoint-based navigation for reducing the computational costs of developing and training MARL models for military training simulations, where geo-specific terrains and differing objectives are crucial.
43.
Liu, Ruying; Becerik-Gerber, Burcin; Pynadath, David V.; Marti, Deniz; Lucas, Gale M.
Elicitation and verification of learning via experts (EVOLVE) for creating a theoretical framework for active shooter incidents Journal Article
In: Developments in the Built Environment, vol. 21, pp. 100635, 2025, ISSN: 26661659.
@article{liu_elicitation_2025,
title = {Elicitation and verification of learning via experts (EVOLVE) for creating a theoretical framework for active shooter incidents},
author = {Ruying Liu and Burcin Becerik-Gerber and David V. Pynadath and Deniz Marti and Gale M. Lucas},
url = {https://linkinghub.elsevier.com/retrieve/pii/S2666165925000353},
doi = {10.1016/j.dibe.2025.100635},
issn = {26661659},
year = {2025},
date = {2025-03-01},
urldate = {2025-03-18},
journal = {Developments in the Built Environment},
volume = {21},
pages = {100635},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
44.
Jalal-Kamali, Ali; Gurney, Nikolos; Pynadath, David
Predicting Team Performance from Communications in Simulated Search-and-Rescue Miscellaneous
2025, (arXiv:2503.03791 [cs]).
@misc{jalal-kamali_predicting_2025,
title = {Predicting Team Performance from Communications in Simulated Search-and-Rescue},
author = {Ali Jalal-Kamali and Nikolos Gurney and David Pynadath},
url = {http://arxiv.org/abs/2503.03791},
doi = {10.48550/arXiv.2503.03791},
year = {2025},
date = {2025-03-01},
urldate = {2025-03-18},
publisher = {arXiv},
abstract = {Understanding how individual traits influence team performance is valuable, but these traits are not always directly observable. Prior research has inferred traits like trust from behavioral data. We analyze conversational data to identify team traits and their correlation with teaming outcomes. Using transcripts from a Minecraft-based search-and-rescue experiment, we apply topic modeling and clustering to uncover key interaction patterns. Our findings show that variations in teaming outcomes can be explained through these inferences, with different levels of predictive power derived from individual traits and team dynamics.},
note = {arXiv:2503.03791 [cs]},
keywords = {},
pubstate = {published},
tppubtype = {misc}
}
Understanding how individual traits influence team performance is valuable, but these traits are not always directly observable. Prior research has inferred traits like trust from behavioral data. We analyze conversational data to identify team traits and their correlation with teaming outcomes. Using transcripts from a Minecraft-based search-and-rescue experiment, we apply topic modeling and clustering to uncover key interaction patterns. Our findings show that variations in teaming outcomes can be explained through these inferences, with different levels of predictive power derived from individual traits and team dynamics.
45.
Kwon, Deuksin; Hae, Jiwon; Clift, Emma; Shamsoddini, Daniel; Gratch, Jonathan; Lucas, Gale M.
ASTRA: A Negotiation Agent with Adaptive and Strategic Reasoning through Action in Dynamic Offer Optimization Miscellaneous
2025, (arXiv:2503.07129 [cs]).
@misc{kwon_astra_2025,
title = {ASTRA: A Negotiation Agent with Adaptive and Strategic Reasoning through Action in Dynamic Offer Optimization},
author = {Deuksin Kwon and Jiwon Hae and Emma Clift and Daniel Shamsoddini and Jonathan Gratch and Gale M. Lucas},
url = {http://arxiv.org/abs/2503.07129},
doi = {10.48550/arXiv.2503.07129},
year = {2025},
date = {2025-03-01},
urldate = {2025-03-18},
publisher = {arXiv},
abstract = {Negotiation requires dynamically balancing self-interest and cooperation to maximize one's own utility. Yet, existing agents struggle due to bounded rationality in human data, low adaptability to counterpart behavior, and limited strategic reasoning. To address this, we introduce principle-driven negotiation agents, powered by ASTRA, a novel framework for turn-level offer optimization grounded in two core principles: opponent modeling and Tit-for-Tat reciprocity. ASTRA operates in three stages: (1) interpreting counterpart behavior, (2) optimizing counteroffers via a linear programming (LP) solver, and (3) selecting offers based on negotiation tactics and the partner's acceptance probability. Through simulations and human evaluations, our agent effectively adapts to an opponent's shifting stance and achieves favorable outcomes through enhanced adaptability and strategic reasoning. Beyond improving negotiation performance, it also serves as a powerful coaching tool, offering interpretable strategic feedback and optimal offer recommendations.},
note = {arXiv:2503.07129 [cs]},
keywords = {},
pubstate = {published},
tppubtype = {misc}
}
Negotiation requires dynamically balancing self-interest and cooperation to maximize one's own utility. Yet, existing agents struggle due to bounded rationality in human data, low adaptability to counterpart behavior, and limited strategic reasoning. To address this, we introduce principle-driven negotiation agents, powered by ASTRA, a novel framework for turn-level offer optimization grounded in two core principles: opponent modeling and Tit-for-Tat reciprocity. ASTRA operates in three stages: (1) interpreting counterpart behavior, (2) optimizing counteroffers via a linear programming (LP) solver, and (3) selecting offers based on negotiation tactics and the partner's acceptance probability. Through simulations and human evaluations, our agent effectively adapts to an opponent's shifting stance and achieves favorable outcomes through enhanced adaptability and strategic reasoning. Beyond improving negotiation performance, it also serves as a powerful coaching tool, offering interpretable strategic feedback and optimal offer recommendations.
46.
Fonseca, Henrique Correia Da; Melo, Celso M. De; Terada, Kazunori; Gratch, Jonathan; Paiva, Ana S.; Santos, Francisco C.
Evolution of indirect reciprocity under emotion expression Journal Article
In: Sci Rep, vol. 15, no. 1, pp. 9151, 2025, ISSN: 2045-2322.
@article{correia_da_fonseca_evolution_2025,
title = {Evolution of indirect reciprocity under emotion expression},
author = {Henrique Correia Da Fonseca and Celso M. De Melo and Kazunori Terada and Jonathan Gratch and Ana S. Paiva and Francisco C. Santos},
url = {https://www.nature.com/articles/s41598-025-89588-8},
doi = {10.1038/s41598-025-89588-8},
issn = {2045-2322},
year = {2025},
date = {2025-03-01},
urldate = {2025-03-20},
journal = {Sci Rep},
volume = {15},
number = {1},
pages = {9151},
abstract = {Abstract
Do emotion expressions impact the evolution of cooperation? Indirect Reciprocity offers a solution to the cooperation dilemma with prior work focusing on the role of social norms in propagating others’ reputations and contributing to evolutionarily stable cooperation. Recent experimental studies, however, show that emotion expressions shape pro-social behaviour, communicate one’s intentions to others, and serve an error-correcting function; yet, the role of emotion signals in the evolution of cooperation remains unexplored. We present the first model of IR based on evolutionary game theory that exposes how emotion expressions positively influence the evolution of cooperation, particularly in scenarios of frequent errors. Our findings provide evolutionary support for the existence of emotion-based social norms, which help foster cooperation among unrelated individuals.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Abstract
Do emotion expressions impact the evolution of cooperation? Indirect Reciprocity offers a solution to the cooperation dilemma with prior work focusing on the role of social norms in propagating others’ reputations and contributing to evolutionarily stable cooperation. Recent experimental studies, however, show that emotion expressions shape pro-social behaviour, communicate one’s intentions to others, and serve an error-correcting function; yet, the role of emotion signals in the evolution of cooperation remains unexplored. We present the first model of IR based on evolutionary game theory that exposes how emotion expressions positively influence the evolution of cooperation, particularly in scenarios of frequent errors. Our findings provide evolutionary support for the existence of emotion-based social norms, which help foster cooperation among unrelated individuals.
Do emotion expressions impact the evolution of cooperation? Indirect Reciprocity offers a solution to the cooperation dilemma with prior work focusing on the role of social norms in propagating others’ reputations and contributing to evolutionarily stable cooperation. Recent experimental studies, however, show that emotion expressions shape pro-social behaviour, communicate one’s intentions to others, and serve an error-correcting function; yet, the role of emotion signals in the evolution of cooperation remains unexplored. We present the first model of IR based on evolutionary game theory that exposes how emotion expressions positively influence the evolution of cooperation, particularly in scenarios of frequent errors. Our findings provide evolutionary support for the existence of emotion-based social norms, which help foster cooperation among unrelated individuals.
47.
Jin, Zhangyu; Feng, Andrew; Chemburkar, Ankur; Melo, Celso M. De
PromptGAR: Flexible Promptive Group Activity Recognition Miscellaneous
2025, (arXiv:2503.08933 [cs]).
@misc{jin_promptgar_2025,
title = {PromptGAR: Flexible Promptive Group Activity Recognition},
author = {Zhangyu Jin and Andrew Feng and Ankur Chemburkar and Celso M. De Melo},
url = {http://arxiv.org/abs/2503.08933},
doi = {10.48550/arXiv.2503.08933},
year = {2025},
date = {2025-03-01},
urldate = {2025-03-20},
publisher = {arXiv},
abstract = {We present PromptGAR, a novel framework that addresses the limitations of current Group Activity Recognition (GAR) approaches by leveraging multi-modal prompts to achieve both input flexibility and high recognition accuracy. The existing approaches suffer from limited real-world applicability due to their reliance on full prompt annotations, the lack of long-term actor consistency, and under-exploration of multi-group scenarios. To bridge the gap, we proposed PromptGAR, which is the first GAR model to provide input flexibility across prompts, frames, and instances without the need for retraining. Specifically, we unify bounding boxes, skeletal keypoints, and areas as point prompts and employ a recognition decoder for cross-updating class and prompt tokens. To ensure long-term consistency for extended activity durations, we also introduce a relative instance attention mechanism that directly encodes instance IDs. Finally, PromptGAR explores the use of area prompts to enable the selective recognition of the particular group activity within videos that contain multiple concurrent groups. Comprehensive evaluations demonstrate that PromptGAR achieves competitive performances both on full prompts and diverse prompt inputs, establishing its effectiveness on input flexibility and generalization ability for real-world applications.},
note = {arXiv:2503.08933 [cs]},
keywords = {},
pubstate = {published},
tppubtype = {misc}
}
We present PromptGAR, a novel framework that addresses the limitations of current Group Activity Recognition (GAR) approaches by leveraging multi-modal prompts to achieve both input flexibility and high recognition accuracy. The existing approaches suffer from limited real-world applicability due to their reliance on full prompt annotations, the lack of long-term actor consistency, and under-exploration of multi-group scenarios. To bridge the gap, we proposed PromptGAR, which is the first GAR model to provide input flexibility across prompts, frames, and instances without the need for retraining. Specifically, we unify bounding boxes, skeletal keypoints, and areas as point prompts and employ a recognition decoder for cross-updating class and prompt tokens. To ensure long-term consistency for extended activity durations, we also introduce a relative instance attention mechanism that directly encodes instance IDs. Finally, PromptGAR explores the use of area prompts to enable the selective recognition of the particular group activity within videos that contain multiple concurrent groups. Comprehensive evaluations demonstrate that PromptGAR achieves competitive performances both on full prompts and diverse prompt inputs, establishing its effectiveness on input flexibility and generalization ability for real-world applications.
48.
Liu, Ruying; Becerik-Gerber, Burçin; Lucas, Gale M.
Investigating Role of Personal Factors in Shaping Responses to Active Shooter Incident using Machine Learning Miscellaneous
2025, (arXiv:2503.05719 [cs]).
@misc{liu_investigating_2025,
title = {Investigating Role of Personal Factors in Shaping Responses to Active Shooter Incident using Machine Learning},
author = {Ruying Liu and Burçin Becerik-Gerber and Gale M. Lucas},
url = {http://arxiv.org/abs/2503.05719},
doi = {10.48550/arXiv.2503.05719},
year = {2025},
date = {2025-02-01},
urldate = {2025-03-18},
publisher = {arXiv},
abstract = {This study bridges the knowledge gap on how personal factors affect building occupants' responses in active shooter situations by applying interpretable machine learning methods to data from 107 participants. The personal factors studied are training methods, prior training experience, sense of direction, and gender. The response performance measurements consist of decisions (run, hide, multiple), vulnerability (corresponding to the time a participant is visible to a shooter), and pre-evacuation time. The results indicate that the propensity to run significantly determines overall response strategies, overshadowing vulnerability, and pre-evacuation time. The training method is a critical factor where VR-based training leads to better responses than video-based training. A better sense of direction and previous training experience are correlated with a greater propensity to run and less vulnerability. Gender slightly influences decisions and vulnerability but significantly impacts pre-evacuation time, with females evacuating slower, potentially due to higher risk perception. This study underscores the importance of personal factors in shaping responses to active shooter incidents.},
note = {arXiv:2503.05719 [cs]},
keywords = {},
pubstate = {published},
tppubtype = {misc}
}
This study bridges the knowledge gap on how personal factors affect building occupants' responses in active shooter situations by applying interpretable machine learning methods to data from 107 participants. The personal factors studied are training methods, prior training experience, sense of direction, and gender. The response performance measurements consist of decisions (run, hide, multiple), vulnerability (corresponding to the time a participant is visible to a shooter), and pre-evacuation time. The results indicate that the propensity to run significantly determines overall response strategies, overshadowing vulnerability, and pre-evacuation time. The training method is a critical factor where VR-based training leads to better responses than video-based training. A better sense of direction and previous training experience are correlated with a greater propensity to run and less vulnerability. Gender slightly influences decisions and vulnerability but significantly impacts pre-evacuation time, with females evacuating slower, potentially due to higher risk perception. This study underscores the importance of personal factors in shaping responses to active shooter incidents.
49.
Huang, Huajian; Chen, Yingshu; Li, Longwei; Cheng, Hui; Braud, Tristan; Zhao, Yajie; Yeung, Sai-Kit
SC-OmniGS: Self-Calibrating Omnidirectional Gaussian Splatting Miscellaneous
2025, (arXiv:2502.04734 [cs]).
@misc{huang_sc-omnigs_2025,
title = {SC-OmniGS: Self-Calibrating Omnidirectional Gaussian Splatting},
author = {Huajian Huang and Yingshu Chen and Longwei Li and Hui Cheng and Tristan Braud and Yajie Zhao and Sai-Kit Yeung},
url = {http://arxiv.org/abs/2502.04734},
doi = {10.48550/arXiv.2502.04734},
year = {2025},
date = {2025-02-01},
urldate = {2025-03-18},
publisher = {arXiv},
abstract = {360-degree cameras streamline data collection for radiance field 3D reconstruction by capturing comprehensive scene data. However, traditional radiance field methods do not address the specific challenges inherent to 360-degree images. We present SC-OmniGS, a novel self-calibrating omnidirectional Gaussian splatting system for fast and accurate omnidirectional radiance field reconstruction using 360-degree images. Rather than converting 360-degree images to cube maps and performing perspective image calibration, we treat 360-degree images as a whole sphere and derive a mathematical framework that enables direct omnidirectional camera pose calibration accompanied by 3D Gaussians optimization. Furthermore, we introduce a differentiable omnidirectional camera model in order to rectify the distortion of real-world data for performance enhancement. Overall, the omnidirectional camera intrinsic model, extrinsic poses, and 3D Gaussians are jointly optimized by minimizing weighted spherical photometric loss. Extensive experiments have demonstrated that our proposed SC-OmniGS is able to recover a high-quality radiance field from noisy camera poses or even no pose prior in challenging scenarios characterized by wide baselines and non-object-centric configurations. The noticeable performance gain in the real-world dataset captured by consumer-grade omnidirectional cameras verifies the effectiveness of our general omnidirectional camera model in reducing the distortion of 360-degree images.},
note = {arXiv:2502.04734 [cs]},
keywords = {},
pubstate = {published},
tppubtype = {misc}
}
360-degree cameras streamline data collection for radiance field 3D reconstruction by capturing comprehensive scene data. However, traditional radiance field methods do not address the specific challenges inherent to 360-degree images. We present SC-OmniGS, a novel self-calibrating omnidirectional Gaussian splatting system for fast and accurate omnidirectional radiance field reconstruction using 360-degree images. Rather than converting 360-degree images to cube maps and performing perspective image calibration, we treat 360-degree images as a whole sphere and derive a mathematical framework that enables direct omnidirectional camera pose calibration accompanied by 3D Gaussians optimization. Furthermore, we introduce a differentiable omnidirectional camera model in order to rectify the distortion of real-world data for performance enhancement. Overall, the omnidirectional camera intrinsic model, extrinsic poses, and 3D Gaussians are jointly optimized by minimizing weighted spherical photometric loss. Extensive experiments have demonstrated that our proposed SC-OmniGS is able to recover a high-quality radiance field from noisy camera poses or even no pose prior in challenging scenarios characterized by wide baselines and non-object-centric configurations. The noticeable performance gain in the real-world dataset captured by consumer-grade omnidirectional cameras verifies the effectiveness of our general omnidirectional camera model in reducing the distortion of 360-degree images.
50.
Roth, Holger R.; Xu, Ziyue; Chen, Chester; Xu, Daguang; Dogra, Prerna; Flores, Mona; Cheng, Yan; Feng, Andrew
Overview of real-world applications of federated learning with NVIDIA FLARE Journal Article
In: Journal of Biopharmaceutical Statistics, pp. 1–11, 2025, ISSN: 1054-3406, 1520-5711.
@article{roth_overview_2025,
title = {Overview of real-world applications of federated learning with NVIDIA FLARE},
author = {Holger R. Roth and Ziyue Xu and Chester Chen and Daguang Xu and Prerna Dogra and Mona Flores and Yan Cheng and Andrew Feng},
url = {https://www.tandfonline.com/doi/full/10.1080/10543406.2025.2456174},
doi = {10.1080/10543406.2025.2456174},
issn = {1054-3406, 1520-5711},
year = {2025},
date = {2025-02-01},
urldate = {2025-03-20},
journal = {Journal of Biopharmaceutical Statistics},
pages = {1–11},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
51.
Tak, Ala N.; Banayeeanzade, Amin; Bolourani, Anahita; Kian, Mina; Jia, Robin; Gratch, Jonathan
Mechanistic Interpretability of Emotion Inference in Large Language Models Miscellaneous
2025, (arXiv:2502.05489 [cs]).
@misc{tak_mechanistic_2025,
title = {Mechanistic Interpretability of Emotion Inference in Large Language Models},
author = {Ala N. Tak and Amin Banayeeanzade and Anahita Bolourani and Mina Kian and Robin Jia and Jonathan Gratch},
url = {http://arxiv.org/abs/2502.05489},
doi = {10.48550/arXiv.2502.05489},
year = {2025},
date = {2025-02-01},
urldate = {2025-02-20},
publisher = {arXiv},
abstract = {Large language models (LLMs) show promising capabilities in predicting human emotions from text. However, the mechanisms through which these models process emotional stimuli remain largely unexplored. Our study addresses this gap by investigating how autoregressive LLMs infer emotions, showing that emotion representations are functionally localized to specific regions in the model. Our evaluation includes diverse model families and sizes and is supported by robustness checks. We then show that the identified representations are psychologically plausible by drawing on cognitive appraisal theory, a well-established psychological framework positing that emotions emerge from evaluations (appraisals) of environmental stimuli. By causally intervening on construed appraisal concepts, we steer the generation and show that the outputs align with theoretical and intuitive expectations. This work highlights a novel way to causally intervene and precisely shape emotional text generation, potentially benefiting safety and alignment in sensitive affective domains.},
note = {arXiv:2502.05489 [cs]},
keywords = {},
pubstate = {published},
tppubtype = {misc}
}
Large language models (LLMs) show promising capabilities in predicting human emotions from text. However, the mechanisms through which these models process emotional stimuli remain largely unexplored. Our study addresses this gap by investigating how autoregressive LLMs infer emotions, showing that emotion representations are functionally localized to specific regions in the model. Our evaluation includes diverse model families and sizes and is supported by robustness checks. We then show that the identified representations are psychologically plausible by drawing on cognitive appraisal theory, a well-established psychological framework positing that emotions emerge from evaluations (appraisals) of environmental stimuli. By causally intervening on construed appraisal concepts, we steer the generation and show that the outputs align with theoretical and intuitive expectations. This work highlights a novel way to causally intervene and precisely shape emotional text generation, potentially benefiting safety and alignment in sensitive affective domains.
52.
Liu, Ruying; Becerik-Gerber, Burcin; Lucas, Gale M.; Busta, Kelly
Impact of behavior-based virtual training on active shooter incident preparedness in healthcare facilities Journal Article
In: International Journal of Disaster Risk Reduction, vol. 118, pp. 105225, 2025, ISSN: 22124209.
@article{liu_impact_2025,
title = {Impact of behavior-based virtual training on active shooter incident preparedness in healthcare facilities},
author = {Ruying Liu and Burcin Becerik-Gerber and Gale M. Lucas and Kelly Busta},
url = {https://linkinghub.elsevier.com/retrieve/pii/S2212420925000494},
doi = {10.1016/j.ijdrr.2025.105225},
issn = {22124209},
year = {2025},
date = {2025-02-01},
urldate = {2025-02-20},
journal = {International Journal of Disaster Risk Reduction},
volume = {118},
pages = {105225},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
53.
Brun, Antonin; Liu, Ruying; Shukla, Aryan; Watson, Frances; Gratch, Jonathan
Exploring Emotion-Sensitive LLM-Based Conversational AI Miscellaneous
2025, (arXiv:2502.08920 [cs]).
@misc{brun_exploring_2025,
title = {Exploring Emotion-Sensitive LLM-Based Conversational AI},
author = {Antonin Brun and Ruying Liu and Aryan Shukla and Frances Watson and Jonathan Gratch},
url = {http://arxiv.org/abs/2502.08920},
doi = {10.48550/arXiv.2502.08920},
year = {2025},
date = {2025-02-01},
urldate = {2025-02-20},
publisher = {arXiv},
abstract = {Conversational AI chatbots have become increasingly common within the customer service industry. Despite improvements in their emotional development, they often lack the authenticity of real customer service interactions or the competence of service providers. By comparing emotion-sensitive and emotion-insensitive LLM-based chatbots across 30 participants, we aim to explore how emotional sensitivity in chatbots influences perceived competence and overall customer satisfaction in service interactions. Additionally, we employ sentiment analysis techniques to analyze and interpret the emotional content of user inputs. We highlight that perceptions of chatbot trustworthiness and competence were higher in the case of the emotion-sensitive chatbot, even if issue resolution rates were not affected. We discuss implications of improved user satisfaction from emotion-sensitive chatbots and potential applications in support services.},
note = {arXiv:2502.08920 [cs]},
keywords = {},
pubstate = {published},
tppubtype = {misc}
}
Conversational AI chatbots have become increasingly common within the customer service industry. Despite improvements in their emotional development, they often lack the authenticity of real customer service interactions or the competence of service providers. By comparing emotion-sensitive and emotion-insensitive LLM-based chatbots across 30 participants, we aim to explore how emotional sensitivity in chatbots influences perceived competence and overall customer satisfaction in service interactions. Additionally, we employ sentiment analysis techniques to analyze and interpret the emotional content of user inputs. We highlight that perceptions of chatbot trustworthiness and competence were higher in the case of the emotion-sensitive chatbot, even if issue resolution rates were not affected. We discuss implications of improved user satisfaction from emotion-sensitive chatbots and potential applications in support services.
54.
Terada, Kazunori; Melo, Celso De; Santos, Francisco C.; Gratch, Jonathan
A Bayesian Model of Mind Reading from Decisions and Emotions in Social Dilemmas Journal Article
In: Proceedings of the Annual Meeting of the Cognitive Science Society, vol. 47, 2025.
@article{terada_bayesian_2025,
title = {A Bayesian Model of Mind Reading from Decisions and Emotions in Social Dilemmas},
author = {Kazunori Terada and Celso De Melo and Francisco C. Santos and Jonathan Gratch},
url = {escholarship.org/uc/item/12f7f7f8#main},
year = {2025},
date = {2025-01-01},
journal = {Proceedings of the Annual Meeting of the Cognitive Science Society},
volume = {47},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
55.
Tak, Ala N.; Gratch, Jonathan; Scherer, Klaus R.
Aware yet Biased: Investigating Emotional Reasoning and Appraisal Bias in Large Language Models Journal Article
In: IEEE Trans. Affective Comput., pp. 1–11, 2025, ISSN: 1949-3045, 2371-9850.
@article{tak_aware_2025,
title = {Aware yet Biased: Investigating Emotional Reasoning and Appraisal Bias in Large Language Models},
author = {Ala N. Tak and Jonathan Gratch and Klaus R. Scherer},
url = {https://ieeexplore.ieee.org/document/11045290/},
doi = {10.1109/TAFFC.2025.3581461},
issn = {1949-3045, 2371-9850},
year = {2025},
date = {2025-01-01},
urldate = {2025-08-19},
journal = {IEEE Trans. Affective Comput.},
pages = {1–11},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
56.
Walsh, Joel; Mamidanna, Siddarth; Nye, Benjamin; Core, Mark; Auerbach, Daniel
Fine-tuning for Better Few Shot Prompting: An Empirical Comparison for Short Answer Grading Miscellaneous
2025, (Version Number: 1).
@misc{walsh_fine-tuning_2025,
title = {Fine-tuning for Better Few Shot Prompting: An Empirical Comparison for Short Answer Grading},
author = {Joel Walsh and Siddarth Mamidanna and Benjamin Nye and Mark Core and Daniel Auerbach},
url = {https://arxiv.org/abs/2508.04063},
doi = {10.48550/ARXIV.2508.04063},
year = {2025},
date = {2025-01-01},
urldate = {2025-08-19},
publisher = {arXiv},
abstract = {Research to improve Automated Short Answer Grading has recently focused on Large Language Models (LLMs) with prompt engineering and no- or few-shot prompting to achieve best results. This is in contrast to the fine-tuning approach, which has historically required large-scale compute clusters inaccessible to most users. New closed-model approaches such as OpenAI's fine-tuning service promise results with as few as 100 examples, while methods using open weights such as quantized low-rank adaptive (QLORA) can be used to fine-tune models on consumer GPUs. We evaluate both of these fine-tuning methods, measuring their interaction with few-shot prompting for automated short answer grading (ASAG) with structured (JSON) outputs. Our results show that finetuning with small amounts of data has limited utility for Llama open-weight models, but that fine-tuning methods can outperform few-shot baseline instruction-tuned LLMs for OpenAI's closed models. While our evaluation set is limited, we find some evidence that the observed benefits of finetuning may be impacted by the domain subject matter. Lastly, we observed dramatic improvement with the LLama 3.1 8B-Instruct open-weight model by seeding the initial training examples with a significant amount of cheaply generated synthetic training data.},
note = {Version Number: 1},
keywords = {},
pubstate = {published},
tppubtype = {misc}
}
Research to improve Automated Short Answer Grading has recently focused on Large Language Models (LLMs) with prompt engineering and no- or few-shot prompting to achieve best results. This is in contrast to the fine-tuning approach, which has historically required large-scale compute clusters inaccessible to most users. New closed-model approaches such as OpenAI's fine-tuning service promise results with as few as 100 examples, while methods using open weights such as quantized low-rank adaptive (QLORA) can be used to fine-tune models on consumer GPUs. We evaluate both of these fine-tuning methods, measuring their interaction with few-shot prompting for automated short answer grading (ASAG) with structured (JSON) outputs. Our results show that finetuning with small amounts of data has limited utility for Llama open-weight models, but that fine-tuning methods can outperform few-shot baseline instruction-tuned LLMs for OpenAI's closed models. While our evaluation set is limited, we find some evidence that the observed benefits of finetuning may be impacted by the domain subject matter. Lastly, we observed dramatic improvement with the LLama 3.1 8B-Instruct open-weight model by seeding the initial training examples with a significant amount of cheaply generated synthetic training data.
57.
Xue, Jintang; Zhao, Ganning; Yao, Jie-En; Chen, Hong-En; Hu, Yue; Chen, Meida; You, Suya; Kuo, C. -C. Jay
Descrip3D: Enhancing Large Language Model-based 3D Scene Understanding with Object-Level Text Descriptions Miscellaneous
2025, (Version Number: 1).
@misc{xue_descrip3d_2025,
title = {Descrip3D: Enhancing Large Language Model-based 3D Scene Understanding with Object-Level Text Descriptions},
author = {Jintang Xue and Ganning Zhao and Jie-En Yao and Hong-En Chen and Yue Hu and Meida Chen and Suya You and C. -C. Jay Kuo},
url = {https://arxiv.org/abs/2507.14555},
doi = {10.48550/ARXIV.2507.14555},
year = {2025},
date = {2025-01-01},
urldate = {2025-08-19},
publisher = {arXiv},
abstract = {Understanding 3D scenes goes beyond simply recognizing objects; it requires reasoning about the spatial and semantic relationships between them. Current 3D scene-language models often struggle with this relational understanding, particularly when visual embeddings alone do not adequately convey the roles and interactions of objects. In this paper, we introduce Descrip3D, a novel and powerful framework that explicitly encodes the relationships between objects using natural language. Unlike previous methods that rely only on 2D and 3D embeddings, Descrip3D enhances each object with a textual description that captures both its intrinsic attributes and contextual relationships. These relational cues are incorporated into the model through a dual-level integration: embedding fusion and prompt-level injection. This allows for unified reasoning across various tasks such as grounding, captioning, and question answering, all without the need for task-specific heads or additional supervision. When evaluated on five benchmark datasets, including ScanRefer, Multi3DRefer, ScanQA, SQA3D, and Scan2Cap, Descrip3D consistently outperforms strong baseline models, demonstrating the effectiveness of language-guided relational representation for understanding complex indoor scenes.},
note = {Version Number: 1},
keywords = {},
pubstate = {published},
tppubtype = {misc}
}
Understanding 3D scenes goes beyond simply recognizing objects; it requires reasoning about the spatial and semantic relationships between them. Current 3D scene-language models often struggle with this relational understanding, particularly when visual embeddings alone do not adequately convey the roles and interactions of objects. In this paper, we introduce Descrip3D, a novel and powerful framework that explicitly encodes the relationships between objects using natural language. Unlike previous methods that rely only on 2D and 3D embeddings, Descrip3D enhances each object with a textual description that captures both its intrinsic attributes and contextual relationships. These relational cues are incorporated into the model through a dual-level integration: embedding fusion and prompt-level injection. This allows for unified reasoning across various tasks such as grounding, captioning, and question answering, all without the need for task-specific heads or additional supervision. When evaluated on five benchmark datasets, including ScanRefer, Multi3DRefer, ScanQA, SQA3D, and Scan2Cap, Descrip3D consistently outperforms strong baseline models, demonstrating the effectiveness of language-guided relational representation for understanding complex indoor scenes.
58.
Behzad, Tina; Gurney, Nikolos; Wang, Ning; Pynadath, David V.
Beyond Predictions: A Study of AI Strength and Weakness Transparency Communication on Human-AI Collaboration Miscellaneous
2025, (Version Number: 1).
@misc{behzad_beyond_2025,
title = {Beyond Predictions: A Study of AI Strength and Weakness Transparency Communication on Human-AI Collaboration},
author = {Tina Behzad and Nikolos Gurney and Ning Wang and David V. Pynadath},
url = {https://arxiv.org/abs/2508.09033},
doi = {10.48550/ARXIV.2508.09033},
year = {2025},
date = {2025-01-01},
urldate = {2025-08-19},
publisher = {arXiv},
abstract = {The promise of human-AI teaming lies in humans and AI working together to achieve performance levels neither could accomplish alone. Effective communication between AI and humans is crucial for teamwork, enabling users to efficiently benefit from AI assistance. This paper investigates how AI communication impacts human-AI team performance. We examine AI explanations that convey an awareness of its strengths and limitations. To achieve this, we train a decision tree on the model's mistakes, allowing it to recognize and explain where and why it might err. Through a user study on an income prediction task, we assess the impact of varying levels of information and explanations about AI predictions. Our results show that AI performance insights enhance task performance, and conveying AI awareness of its strengths and weaknesses improves trust calibration. These findings highlight the importance of considering how information delivery influences user trust and reliance in AI-assisted decision-making.},
note = {Version Number: 1},
keywords = {},
pubstate = {published},
tppubtype = {misc}
}
The promise of human-AI teaming lies in humans and AI working together to achieve performance levels neither could accomplish alone. Effective communication between AI and humans is crucial for teamwork, enabling users to efficiently benefit from AI assistance. This paper investigates how AI communication impacts human-AI team performance. We examine AI explanations that convey an awareness of its strengths and limitations. To achieve this, we train a decision tree on the model's mistakes, allowing it to recognize and explain where and why it might err. Through a user study on an income prediction task, we assess the impact of varying levels of information and explanations about AI predictions. Our results show that AI performance insights enhance task performance, and conveying AI awareness of its strengths and weaknesses improves trust calibration. These findings highlight the importance of considering how information delivery influences user trust and reliance in AI-assisted decision-making.
59.
Rizzo, Albert; Mozgai, Sharon; Sigaras, Alexandros; Rubin, John E.; Jotwani, Rohan
In: Journal of Medical Extended Reality, vol. 2, no. 1, pp. 209–222, 2025, (_eprint: https://www.liebertpub.com/doi/pdf/10.1177/29941520251369450).
@article{rizzo_expert_2025,
title = {Expert Consensus Best Practices for the Safe, Ethical, and Effective Design and Implementation of Artificially Intelligent Conversational Agent (i.e., Chatbot/Virtual Human) Systems in Health Care Applications},
author = {Albert Rizzo and Sharon Mozgai and Alexandros Sigaras and John E. Rubin and Rohan Jotwani},
url = {https://www.liebertpub.com/doi/abs/10.1177/29941520251369450},
doi = {10.1177/29941520251369450},
year = {2025},
date = {2025-01-01},
journal = {Journal of Medical Extended Reality},
volume = {2},
number = {1},
pages = {209–222},
abstract = {The integration of artificially intelligent conversational agents (AICAs), variously referred to as chatbots and virtual humans (VHs), is transforming health care delivery and education. This article explores our perspective on best practices for the evolution, potential, and ethical considerations of AICAs in clinical and educational contexts. Early applications of simulation technology in health care focused on productivity improvements, teletherapy, and virtual reality therapy applications. Recent technological advancements have enabled the development of high-fidelity extended reality systems and AICAs capable of engaging users in credible interactions. These systems leverage natural language processing, machine learning, large language models, and advanced VH authoring software to create interactive, personalized, and engaging experiences. Recent efforts in the creation of AICAs suggest significant potential benefits, including enhanced patient engagement, improved access to self-care resources, and low-stigma interaction environments. They have demonstrated promise in mental health support, providing a sense of safety and encouraging open disclosure. However, the rapid adoption of AICAs raises critical challenges, including safeguarding user privacy, ensuring system reliability, and addressing ethical concerns. Incidents of harm, such as inappropriate interactions and psychological distress, highlight the need for rigorous design and implementation best practices. This article outlines key principles for developing safe, effective, and equitable AICAs, emphasizing transparency in artificial intelligence (AI) identity, accountability, cultural sensitivity, and informed consent. Additionally, the authors advocate for robust privacy measures, adaptive learning capabilities, and evidence-based content validation to optimize user experience and maintain trust. To mitigate risks, a “human-in-the-loop” approach is recommended, ensuring health care professionals oversee AI-supported decisions. By adhering to these best practices, AICAs can enhance health care accessibility, support clinical training, and complement human professionals. This work aims to provide a foundation for the ethical and effective integration of AICAs, maximizing their potential while minimizing risks, ultimately advancing patient care and education in the digital age.},
note = {_eprint: https://www.liebertpub.com/doi/pdf/10.1177/29941520251369450},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
The integration of artificially intelligent conversational agents (AICAs), variously referred to as chatbots and virtual humans (VHs), is transforming health care delivery and education. This article explores our perspective on best practices for the evolution, potential, and ethical considerations of AICAs in clinical and educational contexts. Early applications of simulation technology in health care focused on productivity improvements, teletherapy, and virtual reality therapy applications. Recent technological advancements have enabled the development of high-fidelity extended reality systems and AICAs capable of engaging users in credible interactions. These systems leverage natural language processing, machine learning, large language models, and advanced VH authoring software to create interactive, personalized, and engaging experiences. Recent efforts in the creation of AICAs suggest significant potential benefits, including enhanced patient engagement, improved access to self-care resources, and low-stigma interaction environments. They have demonstrated promise in mental health support, providing a sense of safety and encouraging open disclosure. However, the rapid adoption of AICAs raises critical challenges, including safeguarding user privacy, ensuring system reliability, and addressing ethical concerns. Incidents of harm, such as inappropriate interactions and psychological distress, highlight the need for rigorous design and implementation best practices. This article outlines key principles for developing safe, effective, and equitable AICAs, emphasizing transparency in artificial intelligence (AI) identity, accountability, cultural sensitivity, and informed consent. Additionally, the authors advocate for robust privacy measures, adaptive learning capabilities, and evidence-based content validation to optimize user experience and maintain trust. To mitigate risks, a “human-in-the-loop” approach is recommended, ensuring health care professionals oversee AI-supported decisions. By adhering to these best practices, AICAs can enhance health care accessibility, support clinical training, and complement human professionals. This work aims to provide a foundation for the ethical and effective integration of AICAs, maximizing their potential while minimizing risks, ultimately advancing patient care and education in the digital age.
60.
Hartholt, Arno; Fast, Ed; Kim, Kevin; Sookiassian, Edwin; Leeds, Andrew
TAC-Twin: A Rapid Framework for Personalized Doppelgänger Avatar Creation Using a Modular Virtual Human Pipeline Proceedings Article
In: 2025.
@inproceedings{hartholt_tac-twin_2025,
title = {TAC-Twin: A Rapid Framework for Personalized Doppelgänger Avatar Creation Using a Modular Virtual Human Pipeline},
author = {Arno Hartholt and Ed Fast and Kevin Kim and Edwin Sookiassian and Andrew Leeds},
url = {https://openaccess.cms-conferences.org/publications/book/978-1-964867-74-8/article/978-1-964867-74-8_51},
doi = {10.54941/ahfe1006807},
year = {2025},
date = {2025-01-01},
urldate = {2025-09-18},
abstract = {We present an end-to-end framework for rapidly creating interactive, personalized avatars for scalable training and simulation applications. Built as an extension of the Virtual Human Toolkit, the framework integrates technologies for audio-visual sensing, speech recognition, natural language processing, nonverbal behavior generation, and high-fidelity text-to-speech synthesis. A personalized avatar is defined here as a real-time, embodied digital representation of an actual individual rather than a generic character. The creation pipeline requires only a single facial photograph, processed through a photorealistic character generation workflow, then refined, customized, and deployed in a real-time 3D environment for integration with conversational AI and synthetic voice generation. The system also supports rapid generation of generic avatars from high-quality synthetic headshots produced by generative AI, enabling the creation of diverse, realistic or stylized cohorts within minutes. Our initial use case examines whether personalized avatars enhance engagement, motivation, and performance compared to generic avatars, with the hypothesis that personalization increases relevance, identification, and learning outcomes. We describe the architecture, avatar creation pipeline, and role of generative AI in accelerating development, and share early implementation insights.},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
We present an end-to-end framework for rapidly creating interactive, personalized avatars for scalable training and simulation applications. Built as an extension of the Virtual Human Toolkit, the framework integrates technologies for audio-visual sensing, speech recognition, natural language processing, nonverbal behavior generation, and high-fidelity text-to-speech synthesis. A personalized avatar is defined here as a real-time, embodied digital representation of an actual individual rather than a generic character. The creation pipeline requires only a single facial photograph, processed through a photorealistic character generation workflow, then refined, customized, and deployed in a real-time 3D environment for integration with conversational AI and synthetic voice generation. The system also supports rapid generation of generic avatars from high-quality synthetic headshots produced by generative AI, enabling the creation of diverse, realistic or stylized cohorts within minutes. Our initial use case examines whether personalized avatars enhance engagement, motivation, and performance compared to generic avatars, with the hypothesis that personalization increases relevance, identification, and learning outcomes. We describe the architecture, avatar creation pipeline, and role of generative AI in accelerating development, and share early implementation insights.
Filter
2024
Tak, Ala N.; Gratch, Jonathan
GPT-4 Emulates Average-Human Emotional Cognition from a Third-Person Perspective Miscellaneous
2024, (arXiv:2408.13718 [cs]).
Abstract | Links | BibTeX | Tags: DTIC, Emotions
@misc{tak_gpt-4_2024,
title = {GPT-4 Emulates Average-Human Emotional Cognition from a Third-Person Perspective},
author = {Ala N. Tak and Jonathan Gratch},
url = {http://arxiv.org/abs/2408.13718},
year = {2024},
date = {2024-08-01},
urldate = {2024-09-17},
publisher = {arXiv},
abstract = {This paper extends recent investigations on the emotional reasoning abilities of Large Language Models (LLMs). Current research on LLMs has not directly evaluated the distinction between how LLMs predict the self-attribution of emotions and the perception of others' emotions. We first look at carefully crafted emotion-evoking stimuli, originally designed to find patterns of brain neural activity representing fine-grained inferred emotional attributions of others. We show that GPT-4 is especially accurate in reasoning about such stimuli. This suggests LLMs agree with humans' attributions of others' emotions in stereotypical scenarios remarkably more than self-attributions of emotions in idiosyncratic situations. To further explore this, our second study utilizes a dataset containing annotations from both the author and a third-person perspective. We find that GPT-4's interpretations align more closely with human judgments about the emotions of others than with self-assessments. Notably, conventional computational models of emotion primarily rely on self-reported ground truth as the gold standard. However, an average observer's standpoint, which LLMs appear to have adopted, might be more relevant for many downstream applications, at least in the absence of individual information and adequate safety considerations.},
note = {arXiv:2408.13718 [cs]},
keywords = {DTIC, Emotions},
pubstate = {published},
tppubtype = {misc}
}
This paper extends recent investigations on the emotional reasoning abilities of Large Language Models (LLMs). Current research on LLMs has not directly evaluated the distinction between how LLMs predict the self-attribution of emotions and the perception of others' emotions. We first look at carefully crafted emotion-evoking stimuli, originally designed to find patterns of brain neural activity representing fine-grained inferred emotional attributions of others. We show that GPT-4 is especially accurate in reasoning about such stimuli. This suggests LLMs agree with humans' attributions of others' emotions in stereotypical scenarios remarkably more than self-attributions of emotions in idiosyncratic situations. To further explore this, our second study utilizes a dataset containing annotations from both the author and a third-person perspective. We find that GPT-4's interpretations align more closely with human judgments about the emotions of others than with self-assessments. Notably, conventional computational models of emotion primarily rely on self-reported ground truth as the gold standard. However, an average observer's standpoint, which LLMs appear to have adopted, might be more relevant for many downstream applications, at least in the absence of individual information and adequate safety considerations.
Roth, Holger R.; Beutel, Daniel J.; Cheng, Yan; Marques, Javier Fernandez; Pan, Heng; Chen, Chester; Zhang, Zhihong; Wen, Yuhong; Yang, Sean; Isaac,; Yang,; Hsieh, Yuan-Ting; Xu, Ziyue; Xu, Daguang; Lane, Nicholas D.; Feng, Andrew
Supercharging Federated Learning with Flower and NVIDIA FLARE Miscellaneous
2024, (arXiv:2407.00031 [cs]).
Abstract | Links | BibTeX | Tags:
@misc{roth_supercharging_2024,
title = {Supercharging Federated Learning with Flower and NVIDIA FLARE},
author = {Holger R. Roth and Daniel J. Beutel and Yan Cheng and Javier Fernandez Marques and Heng Pan and Chester Chen and Zhihong Zhang and Yuhong Wen and Sean Yang and Isaac and Yang and Yuan-Ting Hsieh and Ziyue Xu and Daguang Xu and Nicholas D. Lane and Andrew Feng},
url = {http://arxiv.org/abs/2407.00031},
doi = {10.48550/arXiv.2407.00031},
year = {2024},
date = {2024-07-01},
urldate = {2025-01-16},
publisher = {arXiv},
abstract = {Several open-source systems, such as Flower and NVIDIA FLARE, have been developed in recent years while focusing on different aspects of federated learning (FL). Flower is dedicated to implementing a cohesive approach to FL, analytics, and evaluation. Over time, Flower has cultivated extensive strategies and algorithms tailored for FL application development, fostering a vibrant FL community in research and industry. Conversely, FLARE has prioritized the creation of an enterprise-ready, resilient runtime environment explicitly designed for FL applications in production environments. In this paper, we describe our initial integration of both frameworks and show how they can work together to supercharge the FL ecosystem as a whole. Through the seamless integration of Flower and FLARE, applications crafted within the Flower framework can effortlessly operate within the FLARE runtime environment without necessitating any modifications. This initial integration streamlines the process, eliminating complexities and ensuring smooth interoperability between the two platforms, thus enhancing the overall efficiency and accessibility of FL applications.},
note = {arXiv:2407.00031 [cs]},
keywords = {},
pubstate = {published},
tppubtype = {misc}
}
Several open-source systems, such as Flower and NVIDIA FLARE, have been developed in recent years while focusing on different aspects of federated learning (FL). Flower is dedicated to implementing a cohesive approach to FL, analytics, and evaluation. Over time, Flower has cultivated extensive strategies and algorithms tailored for FL application development, fostering a vibrant FL community in research and industry. Conversely, FLARE has prioritized the creation of an enterprise-ready, resilient runtime environment explicitly designed for FL applications in production environments. In this paper, we describe our initial integration of both frameworks and show how they can work together to supercharge the FL ecosystem as a whole. Through the seamless integration of Flower and FLARE, applications crafted within the Flower framework can effortlessly operate within the FLARE runtime environment without necessitating any modifications. This initial integration streamlines the process, eliminating complexities and ensuring smooth interoperability between the two platforms, thus enhancing the overall efficiency and accessibility of FL applications.
Diaz-Pinto, Andres; Alle, Sachidanand; Nath, Vishwesh; Tang, Yucheng; Ihsani, Alvin; Asad, Muhammad; Pérez-García, Fernando; Mehta, Pritesh; Li, Wenqi; Flores, Mona; Roth, Holger R.; Vercauteren, Tom; Xu, Daguang; Dogra, Prerna; Ourselin, Sebastien; Feng, Andrew; Cardoso, M. Jorge
MONAI Label: A framework for AI-assisted interactive labeling of 3D medical images Journal Article
In: Medical Image Analysis, vol. 95, pp. 103207, 2024, ISSN: 13618415.
@article{diaz-pinto_monai_2024,
title = {MONAI Label: A framework for AI-assisted interactive labeling of 3D medical images},
author = {Andres Diaz-Pinto and Sachidanand Alle and Vishwesh Nath and Yucheng Tang and Alvin Ihsani and Muhammad Asad and Fernando Pérez-García and Pritesh Mehta and Wenqi Li and Mona Flores and Holger R. Roth and Tom Vercauteren and Daguang Xu and Prerna Dogra and Sebastien Ourselin and Andrew Feng and M. Jorge Cardoso},
url = {https://linkinghub.elsevier.com/retrieve/pii/S1361841524001324},
doi = {10.1016/j.media.2024.103207},
issn = {13618415},
year = {2024},
date = {2024-07-01},
urldate = {2025-01-16},
journal = {Medical Image Analysis},
volume = {95},
pages = {103207},
keywords = {MedVR},
pubstate = {published},
tppubtype = {article}
}
Huang, Shuo; Jones, Fred; Gurney, Nikolos; Pynadath, David; Srivastava, Kunal; Trent, Stoney; Wu, Peggy; Zhu, Quanyan
PsybORG+: Modeling and Simulation for Detecting Cognitive Biases in Advanced Persistent Threats Miscellaneous
2024, (Version Number: 3).
Abstract | Links | BibTeX | Tags: DTIC
@misc{huang_psyborg_2024,
title = {PsybORG+: Modeling and Simulation for Detecting Cognitive Biases in Advanced Persistent Threats},
author = {Shuo Huang and Fred Jones and Nikolos Gurney and David Pynadath and Kunal Srivastava and Stoney Trent and Peggy Wu and Quanyan Zhu},
url = {https://arxiv.org/abs/2408.01310},
doi = {10.48550/ARXIV.2408.01310},
year = {2024},
date = {2024-07-01},
urldate = {2024-12-05},
publisher = {arXiv},
abstract = {Advanced Persistent Threats (APTs) bring significant challenges to cybersecurity due to their sophisticated and stealthy nature. Traditional cybersecurity measures fail to defend against APTs. Cognitive vulnerabilities can significantly influence attackers' decision-making processes, which presents an opportunity for defenders to exploit. This work introduces PsybORG$ˆ+$, a multi-agent cybersecurity simulation environment designed to model APT behaviors influenced by cognitive vulnerabilities. A classification model is built for cognitive vulnerability inference and a simulator is designed for synthetic data generation. Results show that PsybORG$ˆ+$ can effectively model APT attackers with different loss aversion and confirmation bias levels. The classification model has at least a 0.83 accuracy rate in predicting cognitive vulnerabilities.},
note = {Version Number: 3},
keywords = {DTIC},
pubstate = {published},
tppubtype = {misc}
}
Advanced Persistent Threats (APTs) bring significant challenges to cybersecurity due to their sophisticated and stealthy nature. Traditional cybersecurity measures fail to defend against APTs. Cognitive vulnerabilities can significantly influence attackers' decision-making processes, which presents an opportunity for defenders to exploit. This work introduces PsybORG$ˆ+$, a multi-agent cybersecurity simulation environment designed to model APT behaviors influenced by cognitive vulnerabilities. A classification model is built for cognitive vulnerability inference and a simulator is designed for synthetic data generation. Results show that PsybORG$ˆ+$ can effectively model APT attackers with different loss aversion and confirmation bias levels. The classification model has at least a 0.83 accuracy rate in predicting cognitive vulnerabilities.
Owayyed, Mohammed Al; Tielman, Myrthe; Hartholt, Arno; Specht, Marcus; Brinkman, Willem-Paul
Agent-based social skills training systems: the ARTES architecture, interaction characteristics, learning theories and future outlooks Journal Article
In: Behaviour & Information Technology, pp. 1–28, 2024, ISSN: 0144-929X, 1362-3001.
Links | BibTeX | Tags: Social Simulation, Virtual Agents, Virtual Humans
@article{al_owayyed_agent-based_2024,
title = {Agent-based social skills training systems: the ARTES architecture, interaction characteristics, learning theories and future outlooks},
author = {Mohammed Al Owayyed and Myrthe Tielman and Arno Hartholt and Marcus Specht and Willem-Paul Brinkman},
url = {https://www.tandfonline.com/doi/full/10.1080/0144929X.2024.2374891},
doi = {10.1080/0144929X.2024.2374891},
issn = {0144-929X, 1362-3001},
year = {2024},
date = {2024-07-01},
urldate = {2024-08-15},
journal = {Behaviour & Information Technology},
pages = {1–28},
keywords = {Social Simulation, Virtual Agents, Virtual Humans},
pubstate = {published},
tppubtype = {article}
}
Bell, Imogen H.; Pot-Kolder, Roos; Rizzo, Albert; Rus-Calafell, Mar; Cardi, Valentina; Cella, Matteo; Ward, Thomas; Riches, Simon; Reinoso, Martin; Thompson, Andrew; Alvarez-Jimenez, Mario; Valmaggia, Lucia
Advances in the use of virtual reality to treat mental health conditions Journal Article
In: Nat Rev Psychol, 2024, ISSN: 2731-0574.
@article{bell_advances_2024,
title = {Advances in the use of virtual reality to treat mental health conditions},
author = {Imogen H. Bell and Roos Pot-Kolder and Albert Rizzo and Mar Rus-Calafell and Valentina Cardi and Matteo Cella and Thomas Ward and Simon Riches and Martin Reinoso and Andrew Thompson and Mario Alvarez-Jimenez and Lucia Valmaggia},
url = {https://www.nature.com/articles/s44159-024-00334-9},
doi = {10.1038/s44159-024-00334-9},
issn = {2731-0574},
year = {2024},
date = {2024-07-01},
urldate = {2024-07-11},
journal = {Nat Rev Psychol},
keywords = {MedVR},
pubstate = {published},
tppubtype = {article}
}
Gunasekara, Chulaka; Kim, Seokhwan; D'Haro, Luis Fernando; Rastogi, Abhinav; Chen, Yun-Nung; Eric, Mihail; Hedayatnia, Behnam; Gopalakrishnan, Karthik; Liu, Yang; Huang, Chao-Wei; Hakkani-Tür, Dilek; Li, Jinchao; Zhu, Qi; Luo, Lingxiao; Liden, Lars; Huang, Kaili; Shayandeh, Shahin; Liang, Runze; Peng, Baolin; Zhang, Zheng; Shukla, Swadheen; Huang, Minlie; Gao, Jianfeng; Mehri, Shikib; Feng, Yulan; Gordon, Carla; Alavi, Seyed Hossein; Traum, David; Eskenazi, Maxine; Beirami, Ahmad; Cho, Eunjoon; Crook, Paul A.; De, Ankita; Geramifard, Alborz; Kottur, Satwik; Moon, Seungwhan; Poddar, Shivani; Subba, Rajen
Overview of the Ninth Dialog System Technology Challenge: DSTC9 Journal Article
In: IEEE/ACM Trans. Audio Speech Lang. Process., pp. 1–10, 2024, ISSN: 2329-9290, 2329-9304.
Links | BibTeX | Tags: Natural Language
@article{gunasekara_overview_2024,
title = {Overview of the Ninth Dialog System Technology Challenge: DSTC9},
author = {Chulaka Gunasekara and Seokhwan Kim and Luis Fernando D'Haro and Abhinav Rastogi and Yun-Nung Chen and Mihail Eric and Behnam Hedayatnia and Karthik Gopalakrishnan and Yang Liu and Chao-Wei Huang and Dilek Hakkani-Tür and Jinchao Li and Qi Zhu and Lingxiao Luo and Lars Liden and Kaili Huang and Shahin Shayandeh and Runze Liang and Baolin Peng and Zheng Zhang and Swadheen Shukla and Minlie Huang and Jianfeng Gao and Shikib Mehri and Yulan Feng and Carla Gordon and Seyed Hossein Alavi and David Traum and Maxine Eskenazi and Ahmad Beirami and Eunjoon Cho and Paul A. Crook and Ankita De and Alborz Geramifard and Satwik Kottur and Seungwhan Moon and Shivani Poddar and Rajen Subba},
url = {https://ieeexplore.ieee.org/document/10595468/},
doi = {10.1109/TASLP.2024.3426331},
issn = {2329-9290, 2329-9304},
year = {2024},
date = {2024-07-01},
urldate = {2024-08-15},
journal = {IEEE/ACM Trans. Audio Speech Lang. Process.},
pages = {1–10},
keywords = {Natural Language},
pubstate = {published},
tppubtype = {article}
}
Han, Bin; Yau, Cleo; Lei, Su; Gratch, Jonathan
In-Depth Analysis of Emotion Recognition through Knowledge-Based Large Language Models Miscellaneous
2024, (arXiv:2408.00780 [cs]).
Abstract | Links | BibTeX | Tags: Virtual Humans
@misc{han_-depth_2024,
title = {In-Depth Analysis of Emotion Recognition through Knowledge-Based Large Language Models},
author = {Bin Han and Cleo Yau and Su Lei and Jonathan Gratch},
url = {http://arxiv.org/abs/2408.00780},
year = {2024},
date = {2024-07-01},
urldate = {2024-08-15},
publisher = {arXiv},
abstract = {Emotion recognition in social situations is a complex task that requires integrating information from both facial expressions and the situational context. While traditional approaches to automatic emotion recognition have focused on decontextualized signals, recent research emphasizes the importance of context in shaping emotion perceptions. This paper contributes to the emerging field of context-based emotion recognition by leveraging psychological theories of human emotion perception to inform the design of automated methods. We propose an approach that combines emotion recognition methods with Bayesian Cue Integration (BCI) to integrate emotion inferences from decontextualized facial expressions and contextual knowledge inferred via Large-language Models. We test this approach in the context of interpreting facial expressions during a social task, the prisoner's dilemma. Our results provide clear support for BCI across a range of automatic emotion recognition methods. The best automated method achieved results comparable to human observers, suggesting the potential for this approach to advance the field of affective computing.},
note = {arXiv:2408.00780 [cs]},
keywords = {Virtual Humans},
pubstate = {published},
tppubtype = {misc}
}
Emotion recognition in social situations is a complex task that requires integrating information from both facial expressions and the situational context. While traditional approaches to automatic emotion recognition have focused on decontextualized signals, recent research emphasizes the importance of context in shaping emotion perceptions. This paper contributes to the emerging field of context-based emotion recognition by leveraging psychological theories of human emotion perception to inform the design of automated methods. We propose an approach that combines emotion recognition methods with Bayesian Cue Integration (BCI) to integrate emotion inferences from decontextualized facial expressions and contextual knowledge inferred via Large-language Models. We test this approach in the context of interpreting facial expressions during a social task, the prisoner's dilemma. Our results provide clear support for BCI across a range of automatic emotion recognition methods. The best automated method achieved results comparable to human observers, suggesting the potential for this approach to advance the field of affective computing.
Xiao, Hanyuan; Chen, Yingshu; Huang, Huajian; Xiong, Haolin; Yang, Jing; Prasad, Pratusha; Zhao, Yajie
Localized Gaussian Splatting Editing with Contextual Awareness Miscellaneous
2024, (arXiv:2408.00083 [cs]).
Abstract | Links | BibTeX | Tags: DTIC, VGL
@misc{xiao_localized_2024,
title = {Localized Gaussian Splatting Editing with Contextual Awareness},
author = {Hanyuan Xiao and Yingshu Chen and Huajian Huang and Haolin Xiong and Jing Yang and Pratusha Prasad and Yajie Zhao},
url = {http://arxiv.org/abs/2408.00083},
year = {2024},
date = {2024-07-01},
urldate = {2024-08-16},
publisher = {arXiv},
abstract = {Recent text-guided generation of individual 3D object has achieved great success using diffusion priors. However, these methods are not suitable for object insertion and replacement tasks as they do not consider the background, leading to illumination mismatches within the environment. To bridge the gap, we introduce an illumination-aware 3D scene editing pipeline for 3D Gaussian Splatting (3DGS) representation. Our key observation is that inpainting by the state-of-the-art conditional 2D diffusion model is consistent with background in lighting. To leverage the prior knowledge from the well-trained diffusion models for 3D object generation, our approach employs a coarse-to-fine objection optimization pipeline with inpainted views. In the first coarse step, we achieve image-to-3D lifting given an ideal inpainted view. The process employs 3D-aware diffusion prior from a view-conditioned diffusion model, which preserves illumination present in the conditioning image. To acquire an ideal inpainted image, we introduce an Anchor View Proposal (AVP) algorithm to find a single view that best represents the scene illumination in target region. In the second Texture Enhancement step, we introduce a novel Depth-guided Inpainting Score Distillation Sampling (DI-SDS), which enhances geometry and texture details with the inpainting diffusion prior, beyond the scope of the 3D-aware diffusion prior knowledge in the first coarse step. DI-SDS not only provides fine-grained texture enhancement, but also urges optimization to respect scene lighting. Our approach efficiently achieves local editing with global illumination consistency without explicitly modeling light transport. We demonstrate robustness of our method by evaluating editing in real scenes containing explicit highlight and shadows, and compare against the state-of-the-art text-to-3D editing methods.},
note = {arXiv:2408.00083 [cs]},
keywords = {DTIC, VGL},
pubstate = {published},
tppubtype = {misc}
}
Recent text-guided generation of individual 3D object has achieved great success using diffusion priors. However, these methods are not suitable for object insertion and replacement tasks as they do not consider the background, leading to illumination mismatches within the environment. To bridge the gap, we introduce an illumination-aware 3D scene editing pipeline for 3D Gaussian Splatting (3DGS) representation. Our key observation is that inpainting by the state-of-the-art conditional 2D diffusion model is consistent with background in lighting. To leverage the prior knowledge from the well-trained diffusion models for 3D object generation, our approach employs a coarse-to-fine objection optimization pipeline with inpainted views. In the first coarse step, we achieve image-to-3D lifting given an ideal inpainted view. The process employs 3D-aware diffusion prior from a view-conditioned diffusion model, which preserves illumination present in the conditioning image. To acquire an ideal inpainted image, we introduce an Anchor View Proposal (AVP) algorithm to find a single view that best represents the scene illumination in target region. In the second Texture Enhancement step, we introduce a novel Depth-guided Inpainting Score Distillation Sampling (DI-SDS), which enhances geometry and texture details with the inpainting diffusion prior, beyond the scope of the 3D-aware diffusion prior knowledge in the first coarse step. DI-SDS not only provides fine-grained texture enhancement, but also urges optimization to respect scene lighting. Our approach efficiently achieves local editing with global illumination consistency without explicitly modeling light transport. We demonstrate robustness of our method by evaluating editing in real scenes containing explicit highlight and shadows, and compare against the state-of-the-art text-to-3D editing methods.
Liu, Ruying; Wu, Wanjing; Becerik-Gerber, Burcin; Lucas, Gale M.
2024, (arXiv:2407.10441 [cs]).
Abstract | Links | BibTeX | Tags: DTIC, Virtual Worlds
@misc{liu_enhancing_2024,
title = {Enhancing Building Safety Design for Active Shooter Incidents: Exploration of Building Exit Parameters using Reinforcement Learning-Based Simulations},
author = {Ruying Liu and Wanjing Wu and Burcin Becerik-Gerber and Gale M. Lucas},
url = {http://arxiv.org/abs/2407.10441},
year = {2024},
date = {2024-07-01},
urldate = {2024-09-17},
publisher = {arXiv},
abstract = {With the alarming rise in active shooter incidents (ASIs) in the United States, enhancing public safety through building design has become a pressing need. This study proposes a reinforcement learning-based simulation approach addressing gaps in existing research that has neglected the dynamic behaviours of shooters. We developed an autonomous agent to simulate an active shooter within a realistic office environment, aiming to offer insights into the interactions between building design parameters and ASI outcomes. A case study is conducted to quantitatively investigate the impact of building exit numbers (total count of accessible exits) and configuration (arrangement of which exits are available or not) on evacuation and harm rates. Findings demonstrate that greater exit availability significantly improves evacuation outcomes and reduces harm. Exits nearer to the shooter's initial position hold greater importance for accessibility than those farther away. By encompassing dynamic shooter behaviours, this study offers preliminary insights into effective building safety design against evolving threats.},
note = {arXiv:2407.10441 [cs]},
keywords = {DTIC, Virtual Worlds},
pubstate = {published},
tppubtype = {misc}
}
With the alarming rise in active shooter incidents (ASIs) in the United States, enhancing public safety through building design has become a pressing need. This study proposes a reinforcement learning-based simulation approach addressing gaps in existing research that has neglected the dynamic behaviours of shooters. We developed an autonomous agent to simulate an active shooter within a realistic office environment, aiming to offer insights into the interactions between building design parameters and ASI outcomes. A case study is conducted to quantitatively investigate the impact of building exit numbers (total count of accessible exits) and configuration (arrangement of which exits are available or not) on evacuation and harm rates. Findings demonstrate that greater exit availability significantly improves evacuation outcomes and reduces harm. Exits nearer to the shooter's initial position hold greater importance for accessibility than those farther away. By encompassing dynamic shooter behaviours, this study offers preliminary insights into effective building safety design against evolving threats.
Ke, Pei; Wen, Bosi; Feng, Andrew; Liu, Xiao; Lei, Xuanyu; Cheng, Jiale; Wang, Shengyuan; Zeng, Aohan; Dong, Yuxiao; Wang, Hongning; Tang, Jie; Huang, Minlie
CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generation Proceedings Article
In: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 13034–13054, Association for Computational Linguistics, Bangkok, Thailand, 2024.
Links | BibTeX | Tags: Natural Language
@inproceedings{ke_critiquellm_2024,
title = {CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generation},
author = {Pei Ke and Bosi Wen and Andrew Feng and Xiao Liu and Xuanyu Lei and Jiale Cheng and Shengyuan Wang and Aohan Zeng and Yuxiao Dong and Hongning Wang and Jie Tang and Minlie Huang},
url = {https://aclanthology.org/2024.acl-long.704},
doi = {10.18653/v1/2024.acl-long.704},
year = {2024},
date = {2024-06-01},
urldate = {2025-01-16},
booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
pages = {13034–13054},
publisher = {Association for Computational Linguistics},
address = {Bangkok, Thailand},
keywords = {Natural Language},
pubstate = {published},
tppubtype = {inproceedings}
}
Lu, Shuhong; Jin, Zhangyu; Rajendran, Vickram; Harari, Michal; Feng, Andrew; Melo, Celso M. De
Synthetic-to-real adaptation for complex action recognition in surveillance applications Proceedings Article
In: Manser, Kimberly E.; Melo, Celso De; Rao, Raghuveer M.; Howell, Christopher L. (Ed.): Synthetic Data for Artificial Intelligence and Machine Learning: Tools, Techniques, and Applications II, pp. 14, SPIE, National Harbor, United States, 2024, ISBN: 978-1-5106-7388-5 978-1-5106-7389-2.
@inproceedings{lu_synthetic–real_2024,
title = {Synthetic-to-real adaptation for complex action recognition in surveillance applications},
author = {Shuhong Lu and Zhangyu Jin and Vickram Rajendran and Michal Harari and Andrew Feng and Celso M. De Melo},
editor = {Kimberly E. Manser and Celso De Melo and Raghuveer M. Rao and Christopher L. Howell},
url = {https://www.spiedigitallibrary.org/conference-proceedings-of-spie/13035/3012393/Synthetic-to-real-adaptation-for-complex-action-recognition-in-surveillance/10.1117/12.3012393.full},
doi = {10.1117/12.3012393},
isbn = {978-1-5106-7388-5 978-1-5106-7389-2},
year = {2024},
date = {2024-06-01},
urldate = {2024-07-11},
booktitle = {Synthetic Data for Artificial Intelligence and Machine Learning: Tools, Techniques, and Applications II},
pages = {14},
publisher = {SPIE},
address = {National Harbor, United States},
keywords = {DTIC},
pubstate = {published},
tppubtype = {inproceedings}
}
Nurunnabi, Abdul; Teferle, Felicia; Laefer, Debra F.; Chen, Meida; Ali, Mir Masoom
Development of a Precise Tree Structure from LiDAR Point Clouds Journal Article
In: Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci., vol. XLVIII-2-2024, pp. 301–308, 2024, ISSN: 2194-9034.
Abstract | Links | BibTeX | Tags: Narrative, VGL
@article{nurunnabi_development_2024,
title = {Development of a Precise Tree Structure from LiDAR Point Clouds},
author = {Abdul Nurunnabi and Felicia Teferle and Debra F. Laefer and Meida Chen and Mir Masoom Ali},
url = {https://isprs-archives.copernicus.org/articles/XLVIII-2-2024/301/2024/},
doi = {10.5194/isprs-archives-XLVIII-2-2024-301-2024},
issn = {2194-9034},
year = {2024},
date = {2024-06-01},
urldate = {2024-07-11},
journal = {Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci.},
volume = {XLVIII-2-2024},
pages = {301–308},
abstract = {Abstract. A precise tree structure that represents the distribution of tree stem, branches, and leaves is crucial for accurately capturing the full representation of a tree. Light Detection and Ranging (LiDAR)-based three-dimensional (3D) point clouds (PCs) capture the geometry of scanned objects including forests stands and individual trees. PCs are irregular, unstructured, often noisy, and contaminated by outliers. Researchers have struggled to develop methods to separate leaves and wood without losing the tree geometry. This paper proposes a solution that employs only the spatial coordinates (x, y, z) of the PC. The new algorithm works as a filtering approach, utilizing multi-scale neighborhood-based geometric features (GFs) e.g., linearity, planarity, and verticality to classify linear (wood) and non-linear (leaf) points. This involves finding potential wood points and coupling them with an octree-based segmentation to develop a tree architecture. The main contributions of this paper are (i) investigating the potential of different GFs to split linear and non-linear points, (ii) introducing a novel method that pointwise classifies leaf and wood points, and (iii) developing a precise 3D tree structure. The performance of the new algorithm has been demonstrated through terrestrial laser scanning PCs. For a Scots pine tree, the new method classifies leaf and wood points with an overall accuracy of 97.9%.},
keywords = {Narrative, VGL},
pubstate = {published},
tppubtype = {article}
}
Abstract. A precise tree structure that represents the distribution of tree stem, branches, and leaves is crucial for accurately capturing the full representation of a tree. Light Detection and Ranging (LiDAR)-based three-dimensional (3D) point clouds (PCs) capture the geometry of scanned objects including forests stands and individual trees. PCs are irregular, unstructured, often noisy, and contaminated by outliers. Researchers have struggled to develop methods to separate leaves and wood without losing the tree geometry. This paper proposes a solution that employs only the spatial coordinates (x, y, z) of the PC. The new algorithm works as a filtering approach, utilizing multi-scale neighborhood-based geometric features (GFs) e.g., linearity, planarity, and verticality to classify linear (wood) and non-linear (leaf) points. This involves finding potential wood points and coupling them with an octree-based segmentation to develop a tree architecture. The main contributions of this paper are (i) investigating the potential of different GFs to split linear and non-linear points, (ii) introducing a novel method that pointwise classifies leaf and wood points, and (iii) developing a precise 3D tree structure. The performance of the new algorithm has been demonstrated through terrestrial laser scanning PCs. For a Scots pine tree, the new method classifies leaf and wood points with an overall accuracy of 97.9%.
Zhang, Mingyuan; Cai, Zhongang; Pan, Liang; Hong, Fangzhou; Guo, Xinying; Yang, Lei; Liu, Ziwei
MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model Journal Article
In: IEEE Trans. Pattern Anal. Mach. Intell., vol. 46, no. 6, pp. 4115–4128, 2024, ISSN: 0162-8828, 2160-9292, 1939-3539.
@article{zhang_motiondiffuse_2024,
title = {MotionDiffuse: Text-Driven Human Motion Generation With Diffusion Model},
author = {Mingyuan Zhang and Zhongang Cai and Liang Pan and Fangzhou Hong and Xinying Guo and Lei Yang and Ziwei Liu},
url = {https://ieeexplore.ieee.org/document/10416192/},
doi = {10.1109/TPAMI.2024.3355414},
issn = {0162-8828, 2160-9292, 1939-3539},
year = {2024},
date = {2024-06-01},
urldate = {2024-07-18},
journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
volume = {46},
number = {6},
pages = {4115–4128},
keywords = {VGL},
pubstate = {published},
tppubtype = {article}
}
Yin, Yinxuan; Nayyar, Mollik; Holman, Daniel; Lucas, Gale; Holbrook, Colin; Wagner, Alan
Validation and Evacuee Modeling of Virtual Robot-guided Emergency Evacuation Experiments Miscellaneous
2024.
Abstract | Links | BibTeX | Tags: DTIC, Virtual Humans
@misc{yin_validation_2024,
title = {Validation and Evacuee Modeling of Virtual Robot-guided Emergency Evacuation Experiments},
author = {Yinxuan Yin and Mollik Nayyar and Daniel Holman and Gale Lucas and Colin Holbrook and Alan Wagner},
url = {https://osf.io/mr78s},
doi = {10.31234/osf.io/mr78s},
year = {2024},
date = {2024-06-01},
urldate = {2024-09-17},
publisher = {Center for Open Science},
abstract = {Virtual Reality (VR) is an increasingly common tool for investigating human responses to emergency situations. Nonetheless, studies validating and comparing human subject behavior during real world emergencies to their responses in VR are notably rare, and no prior studies have validated whether human emergency responses to guidance from a robot are comparable in VR versus the real world. In the present pre-registered study, we used VR to replicate a previous robot- guided emergency evacuation study conducted in the real world and compared human subject behavior in matched physical and virtual environments. In both environments, human subjects were asked to follow a robot to a location and to then read an article. While reading, a fire alarm sounds. The robot then attempted to guide them to a distant, unfamiliar exit rather than nearby and familiar exits. We observed close correspondences between evacuee exit choice (the robot’s distant exit versus closer exits), evacuation time, and trust in the robot between the VR and physical environments. We further demonstrate that data collected in virtual reality can be used to create accurate motion models (mean error of 0.42 centimeters) predicting evacuee trajectories and locations in real life. Taken together, the results provide evidence for the ecological validity of VR approaches to studying human-robot interaction, particularly robot- guided emergency evacuation.},
keywords = {DTIC, Virtual Humans},
pubstate = {published},
tppubtype = {misc}
}
Virtual Reality (VR) is an increasingly common tool for investigating human responses to emergency situations. Nonetheless, studies validating and comparing human subject behavior during real world emergencies to their responses in VR are notably rare, and no prior studies have validated whether human emergency responses to guidance from a robot are comparable in VR versus the real world. In the present pre-registered study, we used VR to replicate a previous robot- guided emergency evacuation study conducted in the real world and compared human subject behavior in matched physical and virtual environments. In both environments, human subjects were asked to follow a robot to a location and to then read an article. While reading, a fire alarm sounds. The robot then attempted to guide them to a distant, unfamiliar exit rather than nearby and familiar exits. We observed close correspondences between evacuee exit choice (the robot’s distant exit versus closer exits), evacuation time, and trust in the robot between the VR and physical environments. We further demonstrate that data collected in virtual reality can be used to create accurate motion models (mean error of 0.42 centimeters) predicting evacuee trajectories and locations in real life. Taken together, the results provide evidence for the ecological validity of VR approaches to studying human-robot interaction, particularly robot- guided emergency evacuation.
Saxon, Leslie; Faulk, Robert T; Boberg, Jill; Barrett, Trevor; McLelland, Steve
In: J. Spec. Oper. Med., 2024, ISSN: 1553-9768.
Links | BibTeX | Tags: CBC, DTIC
@article{saxon_continuous_2024,
title = {Continuous Assessment of Active-Duty Army Special Operations and Reconnaissance Marines Using Digital Devices and Custom Software: The Digital Comprehensive Operator Readiness Assessment (DcORA) Study},
author = {Leslie Saxon and Robert T Faulk and Jill Boberg and Trevor Barrett and Steve McLelland},
url = {https://www.jsomonline.org/Citations/PXKK-I23D.php},
doi = {10.55460/PXKK-I23D},
issn = {1553-9768},
year = {2024},
date = {2024-06-01},
urldate = {2024-06-25},
journal = {J. Spec. Oper. Med.},
keywords = {CBC, DTIC},
pubstate = {published},
tppubtype = {article}
}
Greenwald, Eric; Krakowski, Ari; Hurt, Timothy; Grindstaff, Kelly; Wang, Ning
It's like I'm the AI: Youth Sensemaking About AI through Metacognitive Embodiment Proceedings Article
In: Proceedings of the 23rd Annual ACM Interaction Design and Children Conference, pp. 789–793, ACM, Delft Netherlands, 2024, ISBN: 979-8-4007-0442-0.
Links | BibTeX | Tags: AI, Machine Learning
@inproceedings{greenwald_its_2024,
title = {It's like I'm the AI: Youth Sensemaking About AI through Metacognitive Embodiment},
author = {Eric Greenwald and Ari Krakowski and Timothy Hurt and Kelly Grindstaff and Ning Wang},
url = {https://dl.acm.org/doi/10.1145/3628516.3659395},
doi = {10.1145/3628516.3659395},
isbn = {979-8-4007-0442-0},
year = {2024},
date = {2024-06-01},
urldate = {2024-06-25},
booktitle = {Proceedings of the 23rd Annual ACM Interaction Design and Children Conference},
pages = {789–793},
publisher = {ACM},
address = {Delft Netherlands},
keywords = {AI, Machine Learning},
pubstate = {published},
tppubtype = {inproceedings}
}
Chen, Meida; Lal, Devashish; Yu, Zifan; Xu, Jiuyi; Feng, Andrew; You, Suya; Nurunnabi, Abdul; Shi, Yangming
Large-Scale 3D Terrain Reconstruction Using 3D Gaussian Splatting for Visualization and Simulation Journal Article
In: Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci., vol. XLVIII-2-2024, pp. 49–54, 2024, ISSN: 2194-9034.
Abstract | Links | BibTeX | Tags: DTIC, Graphics, VGL
@article{chen_large-scale_2024,
title = {Large-Scale 3D Terrain Reconstruction Using 3D Gaussian Splatting for Visualization and Simulation},
author = {Meida Chen and Devashish Lal and Zifan Yu and Jiuyi Xu and Andrew Feng and Suya You and Abdul Nurunnabi and Yangming Shi},
url = {https://isprs-archives.copernicus.org/articles/XLVIII-2-2024/49/2024/},
doi = {10.5194/isprs-archives-XLVIII-2-2024-49-2024},
issn = {2194-9034},
year = {2024},
date = {2024-06-01},
urldate = {2024-06-20},
journal = {Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci.},
volume = {XLVIII-2-2024},
pages = {49–54},
abstract = {Abstract. The fusion of low-cost unmanned aerial systems (UAS) with advanced photogrammetric techniques has revolutionized 3D terrain reconstruction, enabling the automated creation of detailed models. Concurrently, the advent of 3D Gaussian Splatting has introduced a paradigm shift in 3D data representation, offering visually realistic renditions distinct from traditional polygon-based models. Our research builds upon this foundation, aiming to integrate Gaussian Splatting into interactive simulations for immersive virtual environments. We address challenges such as collision detection by adopting a hybrid approach, combining Gaussian Splatting with photogrammetry-derived meshes. Through comprehensive experimentation covering varying terrain sizes and Gaussian densities, we evaluate scalability, performance, and limitations. Our findings contribute to advancing the use of advanced computer graphics techniques for enhanced 3D terrain visualization and simulation.},
keywords = {DTIC, Graphics, VGL},
pubstate = {published},
tppubtype = {article}
}
Abstract. The fusion of low-cost unmanned aerial systems (UAS) with advanced photogrammetric techniques has revolutionized 3D terrain reconstruction, enabling the automated creation of detailed models. Concurrently, the advent of 3D Gaussian Splatting has introduced a paradigm shift in 3D data representation, offering visually realistic renditions distinct from traditional polygon-based models. Our research builds upon this foundation, aiming to integrate Gaussian Splatting into interactive simulations for immersive virtual environments. We address challenges such as collision detection by adopting a hybrid approach, combining Gaussian Splatting with photogrammetry-derived meshes. Through comprehensive experimentation covering varying terrain sizes and Gaussian densities, we evaluate scalability, performance, and limitations. Our findings contribute to advancing the use of advanced computer graphics techniques for enhanced 3D terrain visualization and simulation.
Nye, Benjamin D.; Core, Mark G.; Chereddy, Sai V. R.; Young, Vivian; Auerbach, Daniel
Bootstrapping Assessments for Team Simulations: Transfer Learning Between First-Person-Shooter Game Maps Book Section
In: Sottilare, Robert A.; Schwarz, Jessica (Ed.): Adaptive Instructional Systems, vol. 14727, pp. 261–271, Springer Nature Switzerland, Cham, 2024, ISBN: 978-3-031-60608-3 978-3-031-60609-0, (Series Title: Lecture Notes in Computer Science).
Links | BibTeX | Tags: DTIC, Learning Sciences, Machine Learning, UARC
@incollection{sottilare_bootstrapping_2024,
title = {Bootstrapping Assessments for Team Simulations: Transfer Learning Between First-Person-Shooter Game Maps},
author = {Benjamin D. Nye and Mark G. Core and Sai V. R. Chereddy and Vivian Young and Daniel Auerbach},
editor = {Robert A. Sottilare and Jessica Schwarz},
url = {https://link.springer.com/10.1007/978-3-031-60609-0_19},
doi = {10.1007/978-3-031-60609-0_19},
isbn = {978-3-031-60608-3 978-3-031-60609-0},
year = {2024},
date = {2024-06-01},
urldate = {2024-06-18},
booktitle = {Adaptive Instructional Systems},
volume = {14727},
pages = {261–271},
publisher = {Springer Nature Switzerland},
address = {Cham},
note = {Series Title: Lecture Notes in Computer Science},
keywords = {DTIC, Learning Sciences, Machine Learning, UARC},
pubstate = {published},
tppubtype = {incollection}
}
Core, Mark G.; Nye, Benjamin D.; Fegley, Brent D.
Trend-Aware Scenario Authoring: Adapting Training Toward Patterns from Real Operations Book Section
In: Sottilare, Robert A.; Schwarz, Jessica (Ed.): Adaptive Instructional Systems, vol. 14727, pp. 15–24, Springer Nature Switzerland, Cham, 2024, ISBN: 978-3-031-60608-3 978-3-031-60609-0, (Series Title: Lecture Notes in Computer Science).
Links | BibTeX | Tags: DTIC, Learning Sciences, UARC
@incollection{sottilare_trend-aware_2024,
title = {Trend-Aware Scenario Authoring: Adapting Training Toward Patterns from Real Operations},
author = {Mark G. Core and Benjamin D. Nye and Brent D. Fegley},
editor = {Robert A. Sottilare and Jessica Schwarz},
url = {https://link.springer.com/10.1007/978-3-031-60609-0_2},
doi = {10.1007/978-3-031-60609-0_2},
isbn = {978-3-031-60608-3 978-3-031-60609-0},
year = {2024},
date = {2024-06-01},
urldate = {2024-06-18},
booktitle = {Adaptive Instructional Systems},
volume = {14727},
pages = {15–24},
publisher = {Springer Nature Switzerland},
address = {Cham},
note = {Series Title: Lecture Notes in Computer Science},
keywords = {DTIC, Learning Sciences, UARC},
pubstate = {published},
tppubtype = {incollection}
}
Artstein, Ron; Chen, Elizabeth
Augmenting Training Data for a Virtual Character Using GPT-3.5 Proceedings Article
In: Tyhe Florida Artificial Intelligence Research Society, 2024.
Abstract | Links | BibTeX | Tags: Dialogue, DTIC, Natural Language
@inproceedings{artstein_augmenting_2024,
title = {Augmenting Training Data for a Virtual Character Using GPT-3.5},
author = {Ron Artstein and Elizabeth Chen},
url = {https://journals.flvc.org/FLAIRS/article/view/135552},
year = {2024},
date = {2024-05-01},
volume = {37},
publisher = {Tyhe Florida Artificial Intelligence Research Society},
abstract = {This paper compares different methods of using a large lan-guage model (GPT-3.5) for creating synthetic training datafor a retrieval-based conversational character. The trainingdata are in the form of linked questions and answers, whichallow a classifier to retrieve a pre-recorded answer to an un-seen question; the intuition is that a large language modelcould predict what human users might ask, thus saving theeffort of collecting real user questions as training data. Re-sults show small improvements in test performance for allsynthetic datasets. However, a classifier trained on only smallamounts of collected user data resulted in a higher F-scorethan the classifiers trained on much larger amounts of syn-thetic data generated using GPT-3.5. Based on these results,we see a potential in using large language models for gener-ating training data, but at this point it is not as valuable ascollecting actual user data for training.},
keywords = {Dialogue, DTIC, Natural Language},
pubstate = {published},
tppubtype = {inproceedings}
}
This paper compares different methods of using a large lan-guage model (GPT-3.5) for creating synthetic training datafor a retrieval-based conversational character. The trainingdata are in the form of linked questions and answers, whichallow a classifier to retrieve a pre-recorded answer to an un-seen question; the intuition is that a large language modelcould predict what human users might ask, thus saving theeffort of collecting real user questions as training data. Re-sults show small improvements in test performance for allsynthetic datasets. However, a classifier trained on only smallamounts of collected user data resulted in a higher F-scorethan the classifiers trained on much larger amounts of syn-thetic data generated using GPT-3.5. Based on these results,we see a potential in using large language models for gener-ating training data, but at this point it is not as valuable ascollecting actual user data for training.
Bohy, Hugo; Tran, Minh; Haddad, Kevin El; Dutoit, Thierry; Soleymani, Mohammad
Social-MAE: A Transformer-Based Multimodal Autoencoder for Face and Voice Proceedings Article
In: 2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG), pp. 1–5, IEEE, Istanbul, Turkiye, 2024, ISBN: 979-8-3503-9494-8.
@inproceedings{bohy_social-mae_2024,
title = {Social-MAE: A Transformer-Based Multimodal Autoencoder for Face and Voice},
author = {Hugo Bohy and Minh Tran and Kevin El Haddad and Thierry Dutoit and Mohammad Soleymani},
url = {https://ieeexplore.ieee.org/document/10581940/},
doi = {10.1109/FG59268.2024.10581940},
isbn = {979-8-3503-9494-8},
year = {2024},
date = {2024-05-01},
urldate = {2024-07-18},
booktitle = {2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG)},
pages = {1–5},
publisher = {IEEE},
address = {Istanbul, Turkiye},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
Liu, Rong; Xu, Rui; Hu, Yue; Chen, Meida; Feng, Andrew
AtomGS: Atomizing Gaussian Splatting for High-Fidelity Radiance Field Miscellaneous
2024, (Version Number: 2).
Abstract | Links | BibTeX | Tags: Graphics, VGL
@misc{liu_atomgs_2024,
title = {AtomGS: Atomizing Gaussian Splatting for High-Fidelity Radiance Field},
author = {Rong Liu and Rui Xu and Yue Hu and Meida Chen and Andrew Feng},
url = {https://arxiv.org/abs/2405.12369},
doi = {10.48550/ARXIV.2405.12369},
year = {2024},
date = {2024-05-01},
urldate = {2024-07-11},
publisher = {arXiv},
abstract = {3D Gaussian Splatting (3DGS) has recently advanced radiance field reconstruction by offering superior capabilities for novel view synthesis and real-time rendering speed. However, its strategy of blending optimization and adaptive density control might lead to sub-optimal results; it can sometimes yield noisy geometry and blurry artifacts due to prioritizing optimizing large Gaussians at the cost of adequately densifying smaller ones. To address this, we introduce AtomGS, consisting of Atomized Proliferation and Geometry-Guided Optimization. The Atomized Proliferation constrains ellipsoid Gaussians of various sizes into more uniform-sized Atom Gaussians. The strategy enhances the representation of areas with fine features by placing greater emphasis on densification in accordance with scene details. In addition, we proposed a Geometry-Guided Optimization approach that incorporates an Edge-Aware Normal Loss. This optimization method effectively smooths flat surfaces while preserving intricate details. Our evaluation shows that AtomGS outperforms existing state-of-the-art methods in rendering quality. Additionally, it achieves competitive accuracy in geometry reconstruction and offers a significant improvement in training speed over other SDF-based methods. More interactive demos can be found in our website (https://rongliu-leo.github.io/AtomGS/).},
note = {Version Number: 2},
keywords = {Graphics, VGL},
pubstate = {published},
tppubtype = {misc}
}
3D Gaussian Splatting (3DGS) has recently advanced radiance field reconstruction by offering superior capabilities for novel view synthesis and real-time rendering speed. However, its strategy of blending optimization and adaptive density control might lead to sub-optimal results; it can sometimes yield noisy geometry and blurry artifacts due to prioritizing optimizing large Gaussians at the cost of adequately densifying smaller ones. To address this, we introduce AtomGS, consisting of Atomized Proliferation and Geometry-Guided Optimization. The Atomized Proliferation constrains ellipsoid Gaussians of various sizes into more uniform-sized Atom Gaussians. The strategy enhances the representation of areas with fine features by placing greater emphasis on densification in accordance with scene details. In addition, we proposed a Geometry-Guided Optimization approach that incorporates an Edge-Aware Normal Loss. This optimization method effectively smooths flat surfaces while preserving intricate details. Our evaluation shows that AtomGS outperforms existing state-of-the-art methods in rendering quality. Additionally, it achieves competitive accuracy in geometry reconstruction and offers a significant improvement in training speed over other SDF-based methods. More interactive demos can be found in our website (https://rongliu-leo.github.io/AtomGS/).
Chang, Di; Shi, Yichun; Gao, Quankai; Fu, Jessica; Xu, Hongyi; Song, Guoxian; Yan, Qing; Zhu, Yizhe; Yang, Xiao; Soleymani, Mohammad
MagicPose: Realistic Human Poses and Facial Expressions Retargeting with Identity-aware Diffusion Miscellaneous
2024, (arXiv:2311.12052 [cs]).
Abstract | Links | BibTeX | Tags:
@misc{chang_magicpose_2024,
title = {MagicPose: Realistic Human Poses and Facial Expressions Retargeting with Identity-aware Diffusion},
author = {Di Chang and Yichun Shi and Quankai Gao and Jessica Fu and Hongyi Xu and Guoxian Song and Qing Yan and Yizhe Zhu and Xiao Yang and Mohammad Soleymani},
url = {http://arxiv.org/abs/2311.12052},
year = {2024},
date = {2024-05-01},
urldate = {2024-07-18},
publisher = {arXiv},
abstract = {In this work, we propose MagicPose, a diffusion-based model for 2D human pose and facial expression retargeting. Specifically, given a reference image, we aim to generate a person's new images by controlling the poses and facial expressions while keeping the identity unchanged. To this end, we propose a two-stage training strategy to disentangle human motions and appearance (e.g., facial expressions, skin tone and dressing), consisting of (1) the pre-training of an appearance-control block and (2) learning appearance-disentangled pose control. Our novel design enables robust appearance control over generated human images, including body, facial attributes, and even background. By leveraging the prior knowledge of image diffusion models, MagicPose generalizes well to unseen human identities and complex poses without the need for additional fine-tuning. Moreover, the proposed model is easy to use and can be considered as a plug-in module/extension to Stable Diffusion. The code is available at: https://github.com/Boese0601/MagicDance},
note = {arXiv:2311.12052 [cs]},
keywords = {},
pubstate = {published},
tppubtype = {misc}
}
In this work, we propose MagicPose, a diffusion-based model for 2D human pose and facial expression retargeting. Specifically, given a reference image, we aim to generate a person's new images by controlling the poses and facial expressions while keeping the identity unchanged. To this end, we propose a two-stage training strategy to disentangle human motions and appearance (e.g., facial expressions, skin tone and dressing), consisting of (1) the pre-training of an appearance-control block and (2) learning appearance-disentangled pose control. Our novel design enables robust appearance control over generated human images, including body, facial attributes, and even background. By leveraging the prior knowledge of image diffusion models, MagicPose generalizes well to unseen human identities and complex poses without the need for additional fine-tuning. Moreover, the proposed model is easy to use and can be considered as a plug-in module/extension to Stable Diffusion. The code is available at: https://github.com/Boese0601/MagicDance
Koresh, Caleb; Ustun, Volkan; Kumar, Rajay; Aris, Tim
Improving Reinforcement Learning Experiments in Unity through Waypoint Utilization Journal Article
In: FLAIRS, vol. 37, 2024, ISSN: 2334-0762.
Abstract | Links | BibTeX | Tags: Machine Learning
@article{koresh_improving_2024,
title = {Improving Reinforcement Learning Experiments in Unity through Waypoint Utilization},
author = {Caleb Koresh and Volkan Ustun and Rajay Kumar and Tim Aris},
url = {https://journals.flvc.org/FLAIRS/article/view/135571},
doi = {10.32473/flairs.37.1.135571},
issn = {2334-0762},
year = {2024},
date = {2024-05-01},
urldate = {2024-08-13},
journal = {FLAIRS},
volume = {37},
abstract = {Multi-agent Reinforcement Learning (MARL) models teams of agents that learn by dynamically interacting with an environment and each other, presenting opportunities to train adaptive models for team-based scenarios. However, MARL algorithms pose substantial challenges due to their immense computational requirements. This paper introduces an automatically generated waypoint-based movement system to abstract and simplify complex environments in Unity while allowing agents to learn strategic cooperation. To demonstrate the effectiveness of our approach, we utilized a simple scenario with heterogeneous roles in each team. We trained this scenario on variations of realistic terrains and compared learning between fine-grained (almost) continuous and waypoint-based movement systems. Our results indicate efficiency in learning and improved performance with waypoint-based navigation. Furthermore, our results show that waypoint-based movement systems can effectively learn differentiated behavior policies for heterogeneous roles in these experiments. These early exploratory results point out the potential of waypoint-based navigation for reducing the computational costs of developing and training MARL models in complex environments. The complete project with all scenarios and results is available on GitHub: https://github.com/HATS-ICT/ml-agents-dodgeball-env-ICT.},
keywords = {Machine Learning},
pubstate = {published},
tppubtype = {article}
}
Multi-agent Reinforcement Learning (MARL) models teams of agents that learn by dynamically interacting with an environment and each other, presenting opportunities to train adaptive models for team-based scenarios. However, MARL algorithms pose substantial challenges due to their immense computational requirements. This paper introduces an automatically generated waypoint-based movement system to abstract and simplify complex environments in Unity while allowing agents to learn strategic cooperation. To demonstrate the effectiveness of our approach, we utilized a simple scenario with heterogeneous roles in each team. We trained this scenario on variations of realistic terrains and compared learning between fine-grained (almost) continuous and waypoint-based movement systems. Our results indicate efficiency in learning and improved performance with waypoint-based navigation. Furthermore, our results show that waypoint-based movement systems can effectively learn differentiated behavior policies for heterogeneous roles in these experiments. These early exploratory results point out the potential of waypoint-based navigation for reducing the computational costs of developing and training MARL models in complex environments. The complete project with all scenarios and results is available on GitHub: https://github.com/HATS-ICT/ml-agents-dodgeball-env-ICT.
Aris, Timothy; Ustun, Volkan; Kumar, Rajay
Training Reinforcement Learning Agents to React to an Ambush for Military Simulations Journal Article
In: FLAIRS, vol. 37, 2024, ISSN: 2334-0762.
Abstract | Links | BibTeX | Tags: Simulation, VR
@article{aris_training_2024,
title = {Training Reinforcement Learning Agents to React to an Ambush for Military Simulations},
author = {Timothy Aris and Volkan Ustun and Rajay Kumar},
url = {https://journals.flvc.org/FLAIRS/article/view/135578},
doi = {10.32473/flairs.37.1.135578},
issn = {2334-0762},
year = {2024},
date = {2024-05-01},
urldate = {2024-08-13},
journal = {FLAIRS},
volume = {37},
abstract = {There is a need for realistic Opposing Forces (OPFOR)behavior in military training simulations. Current trainingsimulations generally only have simple, non-adaptivebehaviors, requiring human instructors to play the role ofOPFOR in any complicated scenario. This poster addressesthis need by focusing on a specific scenario: trainingreinforcement learning agents to react to an ambush. Itproposes a novel way to check for occlusion algorithmically.It shows vector fields showing the agent’s actions throughthe course of a training run. It shows that a single agentswitching between multiple goals is possible, at least in asimplified environment. Such an approach could reduce theneed to develop different agents for different scenarios.Finally, it shows a competent agent trained on a simplifiedReact to Ambush scenario, demonstrating the plausibility ofa scaled-up version.},
keywords = {Simulation, VR},
pubstate = {published},
tppubtype = {article}
}
There is a need for realistic Opposing Forces (OPFOR)behavior in military training simulations. Current trainingsimulations generally only have simple, non-adaptivebehaviors, requiring human instructors to play the role ofOPFOR in any complicated scenario. This poster addressesthis need by focusing on a specific scenario: trainingreinforcement learning agents to react to an ambush. Itproposes a novel way to check for occlusion algorithmically.It shows vector fields showing the agent’s actions throughthe course of a training run. It shows that a single agentswitching between multiple goals is possible, at least in asimplified environment. Such an approach could reduce theneed to develop different agents for different scenarios.Finally, it shows a competent agent trained on a simplifiedReact to Ambush scenario, demonstrating the plausibility ofa scaled-up version.
Liu, Lixing; Ustun, Volkan; Kumar, Rajay
Leveraging Organizational Hierarchy to Simplify Reward Design in Cooperative Multi-agent Reinforcement Learning Journal Article
In: FLAIRS, vol. 37, 2024, ISSN: 2334-0762.
Abstract | Links | BibTeX | Tags: Machine Learning
@article{liu_leveraging_2024,
title = {Leveraging Organizational Hierarchy to Simplify Reward Design in Cooperative Multi-agent Reinforcement Learning},
author = {Lixing Liu and Volkan Ustun and Rajay Kumar},
url = {https://journals.flvc.org/FLAIRS/article/view/135588},
doi = {10.32473/flairs.37.1.135588},
issn = {2334-0762},
year = {2024},
date = {2024-05-01},
urldate = {2024-08-13},
journal = {FLAIRS},
volume = {37},
abstract = {The effectiveness of multi-agent reinforcement learning (MARL) hinges largely on the meticulous arrangement of objectives. Yet, conventional MARL methods might not completely harness the inherent structures present in environmental states and agent relationships for goal organization. This study is conducted within the domain of military training simulations, which are typically characterized by complexity, heterogeneity, non-stationary and doctrine-driven environments with a clear organizational hierarchy and a top-down chain of command. This research investigates the approximation and integration of the organizational hierarchy into MARL for cooperative training scenarios, with the goal of streamlining the processes of reward engineering and enhancing team coordination. In the preliminary experiments, we employed two-tiered commander-subordinate feudal hierarchical (CSFH) networks to separate the prioritized team goal and individual goals. The empirical results demonstrate that the proposed framework enhances learning efficiency. It guarantees the learning of a prioritized policy for the commander agent and encourages subordinate agents to explore areas of interest more frequently, guided by appropriate soft constraints imposed by the commander.},
keywords = {Machine Learning},
pubstate = {published},
tppubtype = {article}
}
The effectiveness of multi-agent reinforcement learning (MARL) hinges largely on the meticulous arrangement of objectives. Yet, conventional MARL methods might not completely harness the inherent structures present in environmental states and agent relationships for goal organization. This study is conducted within the domain of military training simulations, which are typically characterized by complexity, heterogeneity, non-stationary and doctrine-driven environments with a clear organizational hierarchy and a top-down chain of command. This research investigates the approximation and integration of the organizational hierarchy into MARL for cooperative training scenarios, with the goal of streamlining the processes of reward engineering and enhancing team coordination. In the preliminary experiments, we employed two-tiered commander-subordinate feudal hierarchical (CSFH) networks to separate the prioritized team goal and individual goals. The empirical results demonstrate that the proposed framework enhances learning efficiency. It guarantees the learning of a prioritized policy for the commander agent and encourages subordinate agents to explore areas of interest more frequently, guided by appropriate soft constraints imposed by the commander.
Lukin, Stephanie M; Bonial, Claire; Marge, Matthew; Hudson, Taylor; Hayes, Cory J.; Pollard, Kimberly; Baker, Anthony L.; Foots, Ashley; Artstein, Ron; Gervits, Felix; Abrams, Mitchell; Cassidy, Henry; Donatelli, Lucia; Leuski, Anton; Hill, Susan G.; Traum, David; Voss, Clare
SCOUT: A Situated and Multi-Modal Human-Robot Dialogue Corpus Journal Article
In: pp. 14445 - 144458, 2024.
Abstract | Links | BibTeX | Tags:
@article{lukin-etal-2024-scout-situated,
title = {SCOUT: A Situated and Multi-Modal Human-Robot Dialogue Corpus},
author = {Stephanie M Lukin and Claire Bonial and Matthew Marge and Taylor Hudson and Cory J. Hayes and Kimberly Pollard and Anthony L. Baker and Ashley Foots and Ron Artstein and Felix Gervits and Mitchell Abrams and Henry Cassidy and Lucia Donatelli and Anton Leuski and Susan G. Hill and David Traum and Clare Voss},
url = {https://aclanthology.org/2024.lrec-main.1259},
year = {2024},
date = {2024-05-01},
pages = {14445 - 144458},
abstract = {We introduce the Situated Corpus Of Understanding Transactions (SCOUT), a multi-modal collection of human-robot dialogue in the task domain of collaborative exploration. The corpus was constructed from multiple Wizard-of-Oz experiments where human participants gave verbal instructions to a remotely-located robot to move and gather information about its surroundings. SCOUT contains 89,056 utterances and 310,095 words from 278 dialogues averaging 320 utterances per dialogue. The dialogues are aligned with the multi-modal data streams available during the experiments: 5,785 images and 30 maps. The corpus has been annotated with Abstract Meaning Representation and Dialogue-AMR to identify the speaker’s intent and meaning within an utterance, and with Transactional Units and Relations to track relationships between utterances to reveal patterns of the Dialogue Structure. We describe how the corpus and its annotations have been used to develop autonomous human-robot systems and enable research in open questions of how humans speak to robots. We release this corpus to accelerate progress in autonomous, situated, human-robot dialogue, especially in the context of navigation tasks where details about the environment need to be discovered.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
We introduce the Situated Corpus Of Understanding Transactions (SCOUT), a multi-modal collection of human-robot dialogue in the task domain of collaborative exploration. The corpus was constructed from multiple Wizard-of-Oz experiments where human participants gave verbal instructions to a remotely-located robot to move and gather information about its surroundings. SCOUT contains 89,056 utterances and 310,095 words from 278 dialogues averaging 320 utterances per dialogue. The dialogues are aligned with the multi-modal data streams available during the experiments: 5,785 images and 30 maps. The corpus has been annotated with Abstract Meaning Representation and Dialogue-AMR to identify the speaker’s intent and meaning within an utterance, and with Transactional Units and Relations to track relationships between utterances to reveal patterns of the Dialogue Structure. We describe how the corpus and its annotations have been used to develop autonomous human-robot systems and enable research in open questions of how humans speak to robots. We release this corpus to accelerate progress in autonomous, situated, human-robot dialogue, especially in the context of navigation tasks where details about the environment need to be discovered.
West, Taylor Nicole; Prinzing, Michael; Garton, Catherine; Berman, Catherine J.; Zhou, Jieni; Hale, James; Gratch, Jonathan; Fredrickson, Barbara
2024.
Abstract | Links | BibTeX | Tags: Emotions, Virtual Humans
@misc{west_improving_2024,
title = {Improving Social Connection with Weak Ties and Strangers: Effects of a New Micro-Intervention on Interaction Quality and Social Behavior},
author = {Taylor Nicole West and Michael Prinzing and Catherine Garton and Catherine J. Berman and Jieni Zhou and James Hale and Jonathan Gratch and Barbara Fredrickson},
url = {https://osf.io/ytjr6},
doi = {10.31234/osf.io/ytjr6},
year = {2024},
date = {2024-05-01},
urldate = {2024-06-25},
abstract = {We propose that the emotional quality of people’s interactions with acquaintances (i.e., weak ties) and strangers contributes to well-being. We test whether a new micro-intervention can raise the quality of these interactions. We randomized young adults (N = 335) to this connectedness micro-intervention or a control intervention. Both interventions were delivered via a psychoeducational video followed by a brief conversation with a virtual human, with whom participants developed if-then plans to carry out their assigned behavioral goal. Pre-intervention, high-quality weak-tie and stranger interactions were associated with lower loneliness and greater mental health independent of strong-tie interaction quality. Experimental data showed the connectedness intervention improved the emotional quality of participants' interactions with weak ties and strangers over two days, evident in participants’ episodic self-reports and faster in-lab conversational response time. Discussion centers on implications for developing scalable behavioral interventions to improve well-being.},
keywords = {Emotions, Virtual Humans},
pubstate = {published},
tppubtype = {misc}
}
We propose that the emotional quality of people’s interactions with acquaintances (i.e., weak ties) and strangers contributes to well-being. We test whether a new micro-intervention can raise the quality of these interactions. We randomized young adults (N = 335) to this connectedness micro-intervention or a control intervention. Both interventions were delivered via a psychoeducational video followed by a brief conversation with a virtual human, with whom participants developed if-then plans to carry out their assigned behavioral goal. Pre-intervention, high-quality weak-tie and stranger interactions were associated with lower loneliness and greater mental health independent of strong-tie interaction quality. Experimental data showed the connectedness intervention improved the emotional quality of participants' interactions with weak ties and strangers over two days, evident in participants’ episodic self-reports and faster in-lab conversational response time. Discussion centers on implications for developing scalable behavioral interventions to improve well-being.
Zhang, Hao; Chang, Di; Li, Fang; Soleymani, Mohammad; Ahuja, Narendra
MagicPose4D: Crafting Articulated Models with Appearance and Motion Control Miscellaneous
2024, (Version Number: 1).
Abstract | Links | BibTeX | Tags: VGL, Virtual Humans
@misc{zhang_magicpose4d_2024,
title = {MagicPose4D: Crafting Articulated Models with Appearance and Motion Control},
author = {Hao Zhang and Di Chang and Fang Li and Mohammad Soleymani and Narendra Ahuja},
url = {https://arxiv.org/abs/2405.14017},
doi = {10.48550/ARXIV.2405.14017},
year = {2024},
date = {2024-05-01},
urldate = {2024-06-25},
publisher = {arXiv},
abstract = {With the success of 2D and 3D visual generative models, there is growing interest in generating 4D content. Existing methods primarily rely on text prompts to produce 4D content, but they often fall short of accurately defining complex or rare motions. To address this limitation, we propose MagicPose4D, a novel framework for refined control over both appearance and motion in 4D generation. Unlike traditional methods, MagicPose4D accepts monocular videos as motion prompts, enabling precise and customizable motion generation. MagicPose4D comprises two key modules:
i) Dual-Phase 4D Reconstruction Modulevphantom which operates in two phases. The first phase focuses on capturing the model's shape using accurate 2D supervision and less accurate but geometrically informative 3D pseudo-supervision without imposing skeleton constraints. The second phase refines the model using more accurate pseudo-3D supervision, obtained in the first phase and introduces kinematic chain-based skeleton constraints to ensure physical plausibility. Additionally, we propose a Global-local Chamfer loss that aligns the overall distribution of predicted mesh vertices with the supervision while maintaining part-level alignment without extra annotations.
ii) Cross-category Motion Transfer Modulevphantom leverages the predictions from the 4D reconstruction module and uses a kinematic-chain-based skeleton to achieve cross-category motion transfer. It ensures smooth transitions between frames through dynamic rigidity, facilitating robust generalization without additional training.
Through extensive experiments, we demonstrate that MagicPose4D significantly improves the accuracy and consistency of 4D content generation, outperforming existing methods in various benchmarks.},
note = {Version Number: 1},
keywords = {VGL, Virtual Humans},
pubstate = {published},
tppubtype = {misc}
}
With the success of 2D and 3D visual generative models, there is growing interest in generating 4D content. Existing methods primarily rely on text prompts to produce 4D content, but they often fall short of accurately defining complex or rare motions. To address this limitation, we propose MagicPose4D, a novel framework for refined control over both appearance and motion in 4D generation. Unlike traditional methods, MagicPose4D accepts monocular videos as motion prompts, enabling precise and customizable motion generation. MagicPose4D comprises two key modules:
i) Dual-Phase 4D Reconstruction Modulevphantom which operates in two phases. The first phase focuses on capturing the model's shape using accurate 2D supervision and less accurate but geometrically informative 3D pseudo-supervision without imposing skeleton constraints. The second phase refines the model using more accurate pseudo-3D supervision, obtained in the first phase and introduces kinematic chain-based skeleton constraints to ensure physical plausibility. Additionally, we propose a Global-local Chamfer loss that aligns the overall distribution of predicted mesh vertices with the supervision while maintaining part-level alignment without extra annotations.
ii) Cross-category Motion Transfer Modulevphantom leverages the predictions from the 4D reconstruction module and uses a kinematic-chain-based skeleton to achieve cross-category motion transfer. It ensures smooth transitions between frames through dynamic rigidity, facilitating robust generalization without additional training.
Through extensive experiments, we demonstrate that MagicPose4D significantly improves the accuracy and consistency of 4D content generation, outperforming existing methods in various benchmarks.
i) Dual-Phase 4D Reconstruction Modulevphantom which operates in two phases. The first phase focuses on capturing the model's shape using accurate 2D supervision and less accurate but geometrically informative 3D pseudo-supervision without imposing skeleton constraints. The second phase refines the model using more accurate pseudo-3D supervision, obtained in the first phase and introduces kinematic chain-based skeleton constraints to ensure physical plausibility. Additionally, we propose a Global-local Chamfer loss that aligns the overall distribution of predicted mesh vertices with the supervision while maintaining part-level alignment without extra annotations.
ii) Cross-category Motion Transfer Modulevphantom leverages the predictions from the 4D reconstruction module and uses a kinematic-chain-based skeleton to achieve cross-category motion transfer. It ensures smooth transitions between frames through dynamic rigidity, facilitating robust generalization without additional training.
Through extensive experiments, we demonstrate that MagicPose4D significantly improves the accuracy and consistency of 4D content generation, outperforming existing methods in various benchmarks.
Jones, Brennan; Xu, Yan; Li, Qisheng; Scherer, Stefan
Designing a Proactive Context-Aware AI Chatbot for People's Long-Term Goals Proceedings Article
In: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pp. 1–7, ACM, Honolulu HI USA, 2024, ISBN: 979-8-4007-0331-7.
Links | BibTeX | Tags: AI, Simulation
@inproceedings{jones_designing_2024,
title = {Designing a Proactive Context-Aware AI Chatbot for People's Long-Term Goals},
author = {Brennan Jones and Yan Xu and Qisheng Li and Stefan Scherer},
url = {https://dl.acm.org/doi/10.1145/3613905.3650912},
doi = {10.1145/3613905.3650912},
isbn = {979-8-4007-0331-7},
year = {2024},
date = {2024-05-01},
urldate = {2024-06-25},
booktitle = {Extended Abstracts of the CHI Conference on Human Factors in Computing Systems},
pages = {1–7},
publisher = {ACM},
address = {Honolulu HI USA},
keywords = {AI, Simulation},
pubstate = {published},
tppubtype = {inproceedings}
}
Chemburkar, Ankur; Gordon, Andrew; Feng, Andrew
Evaluating Vision-Language Models on the TriangleCOPA Benchmark Journal Article
In: FLAIRS-37, vol. 37, 2024.
Abstract | BibTeX | Tags: DTIC, Narrative
@article{chemburkar_evaluating_2024,
title = {Evaluating Vision-Language Models on the TriangleCOPA Benchmark},
author = {Ankur Chemburkar and Andrew Gordon and Andrew Feng},
year = {2024},
date = {2024-05-01},
journal = {FLAIRS-37},
volume = {37},
abstract = {The TriangleCOPA benchmark consists of 100 textual questions with videos depicting the movements of simple shapes in the style of the classic social-psychology film created by Fritz Heider and Marianne Simmel in 1944. In our experiments, we investigate the performance of current vision-language models on this challenging benchmark, assessing the capability of these models for visual anthropomorphism and abstract interpretation.},
keywords = {DTIC, Narrative},
pubstate = {published},
tppubtype = {article}
}
The TriangleCOPA benchmark consists of 100 textual questions with videos depicting the movements of simple shapes in the style of the classic social-psychology film created by Fritz Heider and Marianne Simmel in 1944. In our experiments, we investigate the performance of current vision-language models on this challenging benchmark, assessing the capability of these models for visual anthropomorphism and abstract interpretation.
Mozgai, Sharon A; Kaurloto, Cari; Winn, Jade G; Leeds, Andrew; Beland, Sarah; Sookiassian, Arman; Hartholt, Arno
Accelerating Scoping Reviews: A Case Study in the User-Centered Design of an AI-Enabled Interdisciplinary Research Tool Proceedings Article
In: Extended Abstracts of the CHI Conference on Human Factors in Computing Systems, pp. 1–8, ACM, Honolulu HI USA, 2024, ISBN: 979-8-4007-0331-7.
Links | BibTeX | Tags: AI, DTIC, UARC, Virtual Humans
@inproceedings{mozgai_accelerating_2024,
title = {Accelerating Scoping Reviews: A Case Study in the User-Centered Design of an AI-Enabled Interdisciplinary Research Tool},
author = {Sharon A Mozgai and Cari Kaurloto and Jade G Winn and Andrew Leeds and Sarah Beland and Arman Sookiassian and Arno Hartholt},
url = {https://dl.acm.org/doi/10.1145/3613905.3637110},
doi = {10.1145/3613905.3637110},
isbn = {979-8-4007-0331-7},
year = {2024},
date = {2024-05-01},
urldate = {2024-06-18},
booktitle = {Extended Abstracts of the CHI Conference on Human Factors in Computing Systems},
pages = {1–8},
publisher = {ACM},
address = {Honolulu HI USA},
keywords = {AI, DTIC, UARC, Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
Murawski, Alaine; Ramirez‐Zohfeld, Vanessa; Mell, Johnathan; Tschoe, Marianne; Schierer, Allison; Olvera, Charles; Brett, Jeanne; Gratch, Jonathan; Lindquist, Lee A.
In: J American Geriatrics Society, vol. 72, no. 4, pp. 1112–1121, 2024, ISSN: 0002-8614, 1532-5415.
Abstract | Links | BibTeX | Tags:
@article{murawski_span_2024,
title = {<span style="font-variant:small-caps;">NegotiAge</span> : Development and pilot testing of an artificial intelligence‐based family caregiver negotiation program},
author = {Alaine Murawski and Vanessa Ramirez‐Zohfeld and Johnathan Mell and Marianne Tschoe and Allison Schierer and Charles Olvera and Jeanne Brett and Jonathan Gratch and Lee A. Lindquist},
url = {https://agsjournals.onlinelibrary.wiley.com/doi/10.1111/jgs.18775},
doi = {10.1111/jgs.18775},
issn = {0002-8614, 1532-5415},
year = {2024},
date = {2024-04-01},
urldate = {2024-12-05},
journal = {J American Geriatrics Society},
volume = {72},
number = {4},
pages = {1112–1121},
abstract = {Abstract
Background
Family caregivers of people with Alzheimer's disease experience conflicts as they navigate health care but lack training to resolve these disputes. We sought to develop and pilot test an artificial‐intelligence negotiation training program, NegotiAge, for family caregivers.
Methods
We convened negotiation experts, a geriatrician, a social worker, and community‐based family caregivers. Content matter experts created short videos to teach negotiation skills. Caregivers generated dialogue surrounding conflicts. Computer scientists utilized the dialogue with the Interactive Arbitration Guide Online (IAGO) platform to develop avatar‐based agents (e.g., sibling, older adult, physician) for caregivers to practice negotiating. Pilot testing was conducted with family caregivers to assess usability (USE) and satisfaction (open‐ended questions with thematic analysis).
Results
Development: With NegotiAge, caregivers progress through didactic material, then receive scenarios to negotiate (e.g., physician recommends gastric tube, sibling disagrees with home support, older adult refusing support). Caregivers negotiate in real‐time with avatars who are designed to act like humans, including emotional tactics and irrational behaviors. Caregivers send/receive offers, using tactics until either mutual agreement or time expires. Immediate feedback is generated for the user to improve skills training. Pilot testing: Family caregivers (
n = 12) completed the program and survey. USE questionnaire (Likert scale 1–7) subset scores revealed: (1) Useful—Mean 5.69 (SD 0.76); (2) Ease—Mean 5.24 (SD 0.96); (3) Learn—Mean 5.69 (SD 0.74); (4) Satisfy—Mean 5.62 (SD 1.10). Items that received over 80% agreements were: It helps me be more effective; It helps me be more productive; It is useful; It gives me more control over the activities in my life; It makes the things I want to accomplish easier to get done. Participants were highly satisfied and found NegotiAge fun to use (91.7%), with 100% who would recommend it to a friend.
Conclusion
NegotiAge is an Artificial‐Intelligent Caregiver Negotiation Program, that is usable and feasible for family caregivers to become familiar with negotiating conflicts commonly seen in health care.},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Abstract
Background
Family caregivers of people with Alzheimer's disease experience conflicts as they navigate health care but lack training to resolve these disputes. We sought to develop and pilot test an artificial‐intelligence negotiation training program, NegotiAge, for family caregivers.
Methods
We convened negotiation experts, a geriatrician, a social worker, and community‐based family caregivers. Content matter experts created short videos to teach negotiation skills. Caregivers generated dialogue surrounding conflicts. Computer scientists utilized the dialogue with the Interactive Arbitration Guide Online (IAGO) platform to develop avatar‐based agents (e.g., sibling, older adult, physician) for caregivers to practice negotiating. Pilot testing was conducted with family caregivers to assess usability (USE) and satisfaction (open‐ended questions with thematic analysis).
Results
Development: With NegotiAge, caregivers progress through didactic material, then receive scenarios to negotiate (e.g., physician recommends gastric tube, sibling disagrees with home support, older adult refusing support). Caregivers negotiate in real‐time with avatars who are designed to act like humans, including emotional tactics and irrational behaviors. Caregivers send/receive offers, using tactics until either mutual agreement or time expires. Immediate feedback is generated for the user to improve skills training. Pilot testing: Family caregivers (
n = 12) completed the program and survey. USE questionnaire (Likert scale 1–7) subset scores revealed: (1) Useful—Mean 5.69 (SD 0.76); (2) Ease—Mean 5.24 (SD 0.96); (3) Learn—Mean 5.69 (SD 0.74); (4) Satisfy—Mean 5.62 (SD 1.10). Items that received over 80% agreements were: It helps me be more effective; It helps me be more productive; It is useful; It gives me more control over the activities in my life; It makes the things I want to accomplish easier to get done. Participants were highly satisfied and found NegotiAge fun to use (91.7%), with 100% who would recommend it to a friend.
Conclusion
NegotiAge is an Artificial‐Intelligent Caregiver Negotiation Program, that is usable and feasible for family caregivers to become familiar with negotiating conflicts commonly seen in health care.
Background
Family caregivers of people with Alzheimer's disease experience conflicts as they navigate health care but lack training to resolve these disputes. We sought to develop and pilot test an artificial‐intelligence negotiation training program, NegotiAge, for family caregivers.
Methods
We convened negotiation experts, a geriatrician, a social worker, and community‐based family caregivers. Content matter experts created short videos to teach negotiation skills. Caregivers generated dialogue surrounding conflicts. Computer scientists utilized the dialogue with the Interactive Arbitration Guide Online (IAGO) platform to develop avatar‐based agents (e.g., sibling, older adult, physician) for caregivers to practice negotiating. Pilot testing was conducted with family caregivers to assess usability (USE) and satisfaction (open‐ended questions with thematic analysis).
Results
Development: With NegotiAge, caregivers progress through didactic material, then receive scenarios to negotiate (e.g., physician recommends gastric tube, sibling disagrees with home support, older adult refusing support). Caregivers negotiate in real‐time with avatars who are designed to act like humans, including emotional tactics and irrational behaviors. Caregivers send/receive offers, using tactics until either mutual agreement or time expires. Immediate feedback is generated for the user to improve skills training. Pilot testing: Family caregivers (
n = 12) completed the program and survey. USE questionnaire (Likert scale 1–7) subset scores revealed: (1) Useful—Mean 5.69 (SD 0.76); (2) Ease—Mean 5.24 (SD 0.96); (3) Learn—Mean 5.69 (SD 0.74); (4) Satisfy—Mean 5.62 (SD 1.10). Items that received over 80% agreements were: It helps me be more effective; It helps me be more productive; It is useful; It gives me more control over the activities in my life; It makes the things I want to accomplish easier to get done. Participants were highly satisfied and found NegotiAge fun to use (91.7%), with 100% who would recommend it to a friend.
Conclusion
NegotiAge is an Artificial‐Intelligent Caregiver Negotiation Program, that is usable and feasible for family caregivers to become familiar with negotiating conflicts commonly seen in health care.
Ehsanpour, Mahsa; Reid, Ian; Rezatofighi, Hamid
Social-MAE: Social Masked Autoencoder for Multi-person Motion Representation Learning Miscellaneous
2024, (Version Number: 1).
Abstract | Links | BibTeX | Tags: Social Simulation
@misc{ehsanpour_social-mae_2024,
title = {Social-MAE: Social Masked Autoencoder for Multi-person Motion Representation Learning},
author = {Mahsa Ehsanpour and Ian Reid and Hamid Rezatofighi},
url = {https://arxiv.org/abs/2404.05578},
doi = {10.48550/ARXIV.2404.05578},
year = {2024},
date = {2024-04-01},
urldate = {2024-07-12},
publisher = {arXiv},
abstract = {For a complete comprehension of multi-person scenes, it is essential to go beyond basic tasks like detection and tracking. Higher-level tasks, such as understanding the interactions and social activities among individuals, are also crucial. Progress towards models that can fully understand scenes involving multiple people is hindered by a lack of sufficient annotated data for such high-level tasks. To address this challenge, we introduce Social-MAE, a simple yet effective transformer-based masked autoencoder framework for multi-person human motion data. The framework uses masked modeling to pre-train the encoder to reconstruct masked human joint trajectories, enabling it to learn generalizable and data efficient representations of motion in human crowded scenes. Social-MAE comprises a transformer as the MAE encoder and a lighter-weight transformer as the MAE decoder which operates on multi-person joints' trajectory in the frequency domain. After the reconstruction task, the MAE decoder is replaced with a task-specific decoder and the model is fine-tuned end-to-end for a variety of high-level social tasks. Our proposed model combined with our pre-training approach achieves the state-of-the-art results on various high-level social tasks, including multi-person pose forecasting, social grouping, and social action understanding. These improvements are demonstrated across four popular multi-person datasets encompassing both human 2D and 3D body pose.},
note = {Version Number: 1},
keywords = {Social Simulation},
pubstate = {published},
tppubtype = {misc}
}
For a complete comprehension of multi-person scenes, it is essential to go beyond basic tasks like detection and tracking. Higher-level tasks, such as understanding the interactions and social activities among individuals, are also crucial. Progress towards models that can fully understand scenes involving multiple people is hindered by a lack of sufficient annotated data for such high-level tasks. To address this challenge, we introduce Social-MAE, a simple yet effective transformer-based masked autoencoder framework for multi-person human motion data. The framework uses masked modeling to pre-train the encoder to reconstruct masked human joint trajectories, enabling it to learn generalizable and data efficient representations of motion in human crowded scenes. Social-MAE comprises a transformer as the MAE encoder and a lighter-weight transformer as the MAE decoder which operates on multi-person joints' trajectory in the frequency domain. After the reconstruction task, the MAE decoder is replaced with a task-specific decoder and the model is fine-tuned end-to-end for a variety of high-level social tasks. Our proposed model combined with our pre-training approach achieves the state-of-the-art results on various high-level social tasks, including multi-person pose forecasting, social grouping, and social action understanding. These improvements are demonstrated across four popular multi-person datasets encompassing both human 2D and 3D body pose.
Soleymani, Mohammad; Kumano, Shiro; Provost, Emily Mower; Bianchi-Berthouze, Nadia; Sano, Akane; Suzuki, Kenji
Guest Editorial Best of ACII 2021 Journal Article
In: IEEE Trans. Affective Comput., vol. 15, no. 2, pp. 376–379, 2024, ISSN: 1949-3045, 2371-9850.
@article{soleymani_guest_2024,
title = {Guest Editorial Best of ACII 2021},
author = {Mohammad Soleymani and Shiro Kumano and Emily Mower Provost and Nadia Bianchi-Berthouze and Akane Sano and Kenji Suzuki},
url = {https://ieeexplore.ieee.org/document/10542496/},
doi = {10.1109/TAFFC.2024.3389249},
issn = {1949-3045, 2371-9850},
year = {2024},
date = {2024-04-01},
urldate = {2024-06-25},
journal = {IEEE Trans. Affective Comput.},
volume = {15},
number = {2},
pages = {376–379},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Zhang, Hui; Kuang, Bingran; Zhao, Yajie
Camera Calibration using a Single View of a Symmetric Object Proceedings Article
In: ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2705–2709, IEEE, Seoul, Korea, Republic of, 2024, ISBN: 979-8-3503-4485-1.
Links | BibTeX | Tags: Graphics, VGL
@inproceedings{zhang_camera_2024,
title = {Camera Calibration using a Single View of a Symmetric Object},
author = {Hui Zhang and Bingran Kuang and Yajie Zhao},
url = {https://ieeexplore.ieee.org/document/10446005/},
doi = {10.1109/ICASSP48485.2024.10446005},
isbn = {979-8-3503-4485-1},
year = {2024},
date = {2024-04-01},
urldate = {2024-06-25},
booktitle = {ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)},
pages = {2705–2709},
publisher = {IEEE},
address = {Seoul, Korea, Republic of},
keywords = {Graphics, VGL},
pubstate = {published},
tppubtype = {inproceedings}
}
Rizzo, Albert Skip; Hartholt, Arno; Mozgai, Sharon
Settling the Score: Virtual Reality as a Tool to Enhance Trauma-Focused Therapy for PTSD Book Section
In: Rich, Grant J.; Kumar, V. K.; Farley, Frank H. (Ed.): Handbook of Media Psychology, pp. 187–213, Springer Nature Switzerland, Cham, 2024, ISBN: 978-3-031-56536-6 978-3-031-56537-3.
Links | BibTeX | Tags: DTIC, MedVR, Simulation, VR
@incollection{rich_settling_2024,
title = {Settling the Score: Virtual Reality as a Tool to Enhance Trauma-Focused Therapy for PTSD},
author = {Albert Skip Rizzo and Arno Hartholt and Sharon Mozgai},
editor = {Grant J. Rich and V. K. Kumar and Frank H. Farley},
url = {https://link.springer.com/10.1007/978-3-031-56537-3_14},
doi = {10.1007/978-3-031-56537-3_14},
isbn = {978-3-031-56536-6 978-3-031-56537-3},
year = {2024},
date = {2024-04-01},
urldate = {2024-06-18},
booktitle = {Handbook of Media Psychology},
pages = {187–213},
publisher = {Springer Nature Switzerland},
address = {Cham},
keywords = {DTIC, MedVR, Simulation, VR},
pubstate = {published},
tppubtype = {incollection}
}
Goh, Crystal; Ma, Yu; Rizzo, Albert
Normative performance data on visual attention in neurotypical children: virtual reality assessment of cognitive and psychomotor development Journal Article
In: Front. Virtual Real., vol. 5, pp. 1309176, 2024, ISSN: 2673-4192.
Abstract | Links | BibTeX | Tags: DTIC, MedVR
@article{goh_normative_2024,
title = {Normative performance data on visual attention in neurotypical children: virtual reality assessment of cognitive and psychomotor development},
author = {Crystal Goh and Yu Ma and Albert Rizzo},
url = {https://www.frontiersin.org/articles/10.3389/frvir.2024.1309176/full},
doi = {10.3389/frvir.2024.1309176},
issn = {2673-4192},
year = {2024},
date = {2024-04-01},
urldate = {2024-04-16},
journal = {Front. Virtual Real.},
volume = {5},
pages = {1309176},
abstract = {Introduction:
Virtual Reality (VR) is revolutionizing healthcare research and practice by offering innovative methodologies across various clinical conditions. Advances in VR technology enable the creation of controllable, multisensory 3D environments, making it an appealing tool for capturing and quantifying behavior in realistic scenarios. This paper details the application of VR as a tool for neurocognitive evaluation, specifically in attention process assessment, an area of relevance for informing the diagnosis of childhood health conditions such as Attention Deficit Hyperactivity Disorder (ADHD).
Methods:
The data presented focuses on attention performance results from a large sample (
n = 837) of neurotypical male and female children (ages 6–13) tested on a visual continuous performance task, administered within an immersive VR classroom environment. This data was collected to create a normative baseline database for use to inform comparisons with the performances of children with ADHD to support diagnostic decision-making in this area.
Results:
Results indicate systematic improvements on most metrics across the age span, and sex differences are noted on key variables thought to reflect differential measures of hyperactivity and inattention in children with ADHD. Results support VR technology as a safe and viable option for testing attention processes in children, under stimulus conditions that closely mimic ecologically relevant challenges found in everyday life.
Discussion:
In response to these stimulus conditions, VR can support advanced methods for capturing and quantifying users’ behavioral responses. VR offers a more systematic and objective approach for clinical assessment and intervention and provides conceptual support for its use in a wide variety of healthcare contexts.},
keywords = {DTIC, MedVR},
pubstate = {published},
tppubtype = {article}
}
Introduction:
Virtual Reality (VR) is revolutionizing healthcare research and practice by offering innovative methodologies across various clinical conditions. Advances in VR technology enable the creation of controllable, multisensory 3D environments, making it an appealing tool for capturing and quantifying behavior in realistic scenarios. This paper details the application of VR as a tool for neurocognitive evaluation, specifically in attention process assessment, an area of relevance for informing the diagnosis of childhood health conditions such as Attention Deficit Hyperactivity Disorder (ADHD).
Methods:
The data presented focuses on attention performance results from a large sample (
n = 837) of neurotypical male and female children (ages 6–13) tested on a visual continuous performance task, administered within an immersive VR classroom environment. This data was collected to create a normative baseline database for use to inform comparisons with the performances of children with ADHD to support diagnostic decision-making in this area.
Results:
Results indicate systematic improvements on most metrics across the age span, and sex differences are noted on key variables thought to reflect differential measures of hyperactivity and inattention in children with ADHD. Results support VR technology as a safe and viable option for testing attention processes in children, under stimulus conditions that closely mimic ecologically relevant challenges found in everyday life.
Discussion:
In response to these stimulus conditions, VR can support advanced methods for capturing and quantifying users’ behavioral responses. VR offers a more systematic and objective approach for clinical assessment and intervention and provides conceptual support for its use in a wide variety of healthcare contexts.
Virtual Reality (VR) is revolutionizing healthcare research and practice by offering innovative methodologies across various clinical conditions. Advances in VR technology enable the creation of controllable, multisensory 3D environments, making it an appealing tool for capturing and quantifying behavior in realistic scenarios. This paper details the application of VR as a tool for neurocognitive evaluation, specifically in attention process assessment, an area of relevance for informing the diagnosis of childhood health conditions such as Attention Deficit Hyperactivity Disorder (ADHD).
Methods:
The data presented focuses on attention performance results from a large sample (
n = 837) of neurotypical male and female children (ages 6–13) tested on a visual continuous performance task, administered within an immersive VR classroom environment. This data was collected to create a normative baseline database for use to inform comparisons with the performances of children with ADHD to support diagnostic decision-making in this area.
Results:
Results indicate systematic improvements on most metrics across the age span, and sex differences are noted on key variables thought to reflect differential measures of hyperactivity and inattention in children with ADHD. Results support VR technology as a safe and viable option for testing attention processes in children, under stimulus conditions that closely mimic ecologically relevant challenges found in everyday life.
Discussion:
In response to these stimulus conditions, VR can support advanced methods for capturing and quantifying users’ behavioral responses. VR offers a more systematic and objective approach for clinical assessment and intervention and provides conceptual support for its use in a wide variety of healthcare contexts.
Soleymani, Mohammad; Rahmani, Mehdi; Bigdeli, Nooshin
Robust Tube-Based Reference Tracking Nonlinear Model Predictive Control for Wind Turbines Journal Article
In: IEEE Trans. Automat. Sci. Eng., pp. 1–13, 2024, ISSN: 1545-5955, 1558-3783.
@article{soleymani_robust_2024,
title = {Robust Tube-Based Reference Tracking Nonlinear Model Predictive Control for Wind Turbines},
author = {Mohammad Soleymani and Mehdi Rahmani and Nooshin Bigdeli},
url = {https://ieeexplore.ieee.org/document/10495787/},
doi = {10.1109/TASE.2024.3385714},
issn = {1545-5955, 1558-3783},
year = {2024},
date = {2024-04-01},
urldate = {2024-04-16},
journal = {IEEE Trans. Automat. Sci. Eng.},
pages = {1–13},
keywords = {DTIC},
pubstate = {published},
tppubtype = {article}
}
Gurney, Nikolos; Loewenstein, George; Chater, Nick
Conversational technology and reactions to withheld information Journal Article
In: PLoS ONE, vol. 19, no. 4, pp. e0301382, 2024, ISSN: 1932-6203.
Abstract | Links | BibTeX | Tags: DTIC, Social Simulation, UARC
@article{gurney_conversational_2024,
title = {Conversational technology and reactions to withheld information},
author = {Nikolos Gurney and George Loewenstein and Nick Chater},
editor = {Petre Caraiani},
url = {https://dx.plos.org/10.1371/journal.pone.0301382},
doi = {10.1371/journal.pone.0301382},
issn = {1932-6203},
year = {2024},
date = {2024-04-01},
urldate = {2024-04-16},
journal = {PLoS ONE},
volume = {19},
number = {4},
pages = {e0301382},
abstract = {People frequently face decisions that require making inferences about withheld information. The advent of large language models coupled with conversational technology, e.g., Alexa, Siri, Cortana, and the Google Assistant, is changing the mode in which people make these inferences. We demonstrate that conversational modes of information provision, relative to traditional digital media, result in more critical responses to withheld information, including: (1) a reduction in evaluations of a product or service for which information is withheld and (2) an increased likelihood of recalling that information was withheld. These effects are robust across multiple conversational modes: a recorded phone conversation, an unfolding chat conversation, and a conversation script. We provide further evidence that these effects hold for conversations with the Google Assistant, a prominent conversational technology. The experimental results point to participants’ intuitions about why the information was withheld as the driver of the effect.},
keywords = {DTIC, Social Simulation, UARC},
pubstate = {published},
tppubtype = {article}
}
People frequently face decisions that require making inferences about withheld information. The advent of large language models coupled with conversational technology, e.g., Alexa, Siri, Cortana, and the Google Assistant, is changing the mode in which people make these inferences. We demonstrate that conversational modes of information provision, relative to traditional digital media, result in more critical responses to withheld information, including: (1) a reduction in evaluations of a product or service for which information is withheld and (2) an increased likelihood of recalling that information was withheld. These effects are robust across multiple conversational modes: a recorded phone conversation, an unfolding chat conversation, and a conversation script. We provide further evidence that these effects hold for conversations with the Google Assistant, a prominent conversational technology. The experimental results point to participants’ intuitions about why the information was withheld as the driver of the effect.
Hartholt, Arno; Leeds, Andrew; Fast, Ed; Sookiassian, Edwin; Kim, Kevin; Beland, Sarah; Kulkarni, Pranav; Mozgai, Sharon
Multidisciplinary Research & Development of Multi-Agents and Virtual Humans Leveraging Integrated Middleware Platforms Proceedings Article
In: 2024.
Abstract | Links | BibTeX | Tags: DTIC, UARC, Virtual Humans
@inproceedings{hartholt_multidisciplinary_2024,
title = {Multidisciplinary Research & Development of Multi-Agents and Virtual Humans Leveraging Integrated Middleware Platforms},
author = {Arno Hartholt and Andrew Leeds and Ed Fast and Edwin Sookiassian and Kevin Kim and Sarah Beland and Pranav Kulkarni and Sharon Mozgai},
url = {https://openaccess.cms-conferences.org/publications/book/978-1-958651-95-7/article/978-1-958651-95-7_33},
doi = {10.54941/ahfe1004497},
year = {2024},
date = {2024-04-01},
urldate = {2024-04-16},
abstract = {The current pace of technological advancements has led to an ever-increasing availability of technologies to investigate and help address the challenges that contemporary society faces today. However, while this trend increases the potential for creating more relevant, effective, and efficient solutions, it also inherently increases the complexity of realizing that potential. Our work aims to manage this complexity through the creation and dissemination of integrated middleware platforms that enable researchers and developers to rapidly prototype novel solutions within the areas of modelling & simulation, virtual humans, and virtual worlds. In this paper, we discuss two related platforms: the Rapid Integration & Development Environment (RIDE) and the Virtual Human Toolkit (VHToolkit). Specifically, we explore two use cases: 1) the development of an authoring tool aimed at domain experts to rapidly create low-echelon military training scenarios, and 2) the development of a virtual human led mHealth wellness and suicide prevention app for veterans.},
keywords = {DTIC, UARC, Virtual Humans},
pubstate = {published},
tppubtype = {inproceedings}
}
The current pace of technological advancements has led to an ever-increasing availability of technologies to investigate and help address the challenges that contemporary society faces today. However, while this trend increases the potential for creating more relevant, effective, and efficient solutions, it also inherently increases the complexity of realizing that potential. Our work aims to manage this complexity through the creation and dissemination of integrated middleware platforms that enable researchers and developers to rapidly prototype novel solutions within the areas of modelling & simulation, virtual humans, and virtual worlds. In this paper, we discuss two related platforms: the Rapid Integration & Development Environment (RIDE) and the Virtual Human Toolkit (VHToolkit). Specifically, we explore two use cases: 1) the development of an authoring tool aimed at domain experts to rapidly create low-echelon military training scenarios, and 2) the development of a virtual human led mHealth wellness and suicide prevention app for veterans.
Georgila, Kallirroi; Traum, David
Evaluation of Off-the-shelf Whisper Models for Speech Recognition Across Diverse Dialogue Domains Proceedings Article
In: Proceedings of the 14th International Workshop on Spoken Dialogue Systems Technology, Sapporo, Japan, 2024.
@inproceedings{georgila_evaluation_2024,
title = {Evaluation of Off-the-shelf Whisper Models for Speech Recognition Across Diverse Dialogue Domains},
author = {Kallirroi Georgila and David Traum},
url = {chrome-extension://efaidnbmnnnibpcajpcglclefindmkaj/https://kgeorgila.github.io/publications/georgila_iwsds24.pdf},
year = {2024},
date = {2024-03-01},
booktitle = {Proceedings of the 14th International Workshop on Spoken Dialogue Systems Technology},
address = {Sapporo, Japan},
keywords = {},
pubstate = {published},
tppubtype = {inproceedings}
}
DeTore, Nicole R.; Balogun-Mwangi, Oyenike; Eberlin, Elizabeth S.; Dokholyan, Katherine N.; Rizzo, Albert; Holt, Daphne J.
An Artificial Intelligence-Based Virtual Human Avatar Application to Assess the Mental Health of Health Care Professionals: A Validation Study Journal Article
In: Journal of Medical Extended Reality, vol. 1, no. 1, pp. 215–226, 2024, ISSN: 2994-1520.
@article{detore_artificial_2024,
title = {An Artificial Intelligence-Based Virtual Human Avatar Application to Assess the Mental Health of Health Care Professionals: A Validation Study},
author = {Nicole R. DeTore and Oyenike Balogun-Mwangi and Elizabeth S. Eberlin and Katherine N. Dokholyan and Albert Rizzo and Daphne J. Holt},
url = {https://www.liebertpub.com/doi/10.1089/jmxr.2024.0016},
doi = {10.1089/jmxr.2024.0016},
issn = {2994-1520},
year = {2024},
date = {2024-03-01},
urldate = {2024-11-01},
journal = {Journal of Medical Extended Reality},
volume = {1},
number = {1},
pages = {215–226},
keywords = {},
pubstate = {published},
tppubtype = {article}
}
Harris, Vera; Braggs, Robert; Traum, David
I’m not sure I heard you right, but I think I know what you mean – investigations into the impact of speech recognition errors on response selection for a virtual human. Proceedings Article
In: Sapporo Japan, 2024.
Links | BibTeX | Tags: Machine Learning
@inproceedings{harris_im_2024,
title = {I’m not sure I heard you right, but I think I know what you mean – investigations into the impact of speech recognition errors on response selection for a virtual human.},
author = {Vera Harris and Robert Braggs and David Traum},
url = {chrome-extension://efaidnbmnnnibpcajpcglclefindmkaj/https://people.ict.usc.edu/~traum/Papers/23-harris-iwsds2024.pdf},
year = {2024},
date = {2024-03-01},
address = {Sapporo Japan},
keywords = {Machine Learning},
pubstate = {published},
tppubtype = {inproceedings}
}
Brixey, Jacqueline; Traum, David
Why should a dialogue system speak more than one language? Proceedings Article
In: Sapporo Japan, 2024.
Links | BibTeX | Tags: Natural Language
@inproceedings{brixey_why_2024,
title = {Why should a dialogue system speak more than one language?},
author = {Jacqueline Brixey and David Traum},
url = {chrome-extension://efaidnbmnnnibpcajpcglclefindmkaj/https://people.ict.usc.edu/~traum/Papers/24-Why%20should%20a%20dialogue%20system%20speak%20more%20than%20one%20language.pdf},
year = {2024},
date = {2024-03-01},
address = {Sapporo Japan},
keywords = {Natural Language},
pubstate = {published},
tppubtype = {inproceedings}
}
Chen, Haiwei; Zhao, Yajie
Don't Look into the Dark: Latent Codes for Pluralistic Image Inpainting Miscellaneous
2024, (arXiv:2403.18186 [cs]).
Abstract | Links | BibTeX | Tags: VGL
@misc{chen_dont_2024,
title = {Don't Look into the Dark: Latent Codes for Pluralistic Image Inpainting},
author = {Haiwei Chen and Yajie Zhao},
url = {http://arxiv.org/abs/2403.18186},
year = {2024},
date = {2024-03-01},
urldate = {2024-08-15},
publisher = {arXiv},
abstract = {We present a method for large-mask pluralistic image inpainting based on the generative framework of discrete latent codes. Our method learns latent priors, discretized as tokens, by only performing computations at the visible locations of the image. This is realized by a restrictive partial encoder that predicts the token label for each visible block, a bidirectional transformer that infers the missing labels by only looking at these tokens, and a dedicated synthesis network that couples the tokens with the partial image priors to generate coherent and pluralistic complete image even under extreme mask settings. Experiments on public benchmarks validate our design choices as the proposed method outperforms strong baselines in both visual quality and diversity metrics.},
note = {arXiv:2403.18186 [cs]},
keywords = {VGL},
pubstate = {published},
tppubtype = {misc}
}
We present a method for large-mask pluralistic image inpainting based on the generative framework of discrete latent codes. Our method learns latent priors, discretized as tokens, by only performing computations at the visible locations of the image. This is realized by a restrictive partial encoder that predicts the token label for each visible block, a bidirectional transformer that infers the missing labels by only looking at these tokens, and a dedicated synthesis network that couples the tokens with the partial image priors to generate coherent and pluralistic complete image even under extreme mask settings. Experiments on public benchmarks validate our design choices as the proposed method outperforms strong baselines in both visual quality and diversity metrics.
Singh, Ishika; Traum, David; Thomason, Jesse
TwoStep: Multi-agent Task Planning using Classical Planners and Large Language Models Miscellaneous
2024, (arXiv:2403.17246 [cs]).
Abstract | Links | BibTeX | Tags:
@misc{singh_twostep_2024,
title = {TwoStep: Multi-agent Task Planning using Classical Planners and Large Language Models},
author = {Ishika Singh and David Traum and Jesse Thomason},
url = {http://arxiv.org/abs/2403.17246},
year = {2024},
date = {2024-03-01},
urldate = {2024-08-15},
publisher = {arXiv},
abstract = {Classical planning formulations like the Planning Domain Definition Language (PDDL) admit action sequences guaranteed to achieve a goal state given an initial state if any are possible. However, reasoning problems defined in PDDL do not capture temporal aspects of action taking, for example that two agents in the domain can execute an action simultaneously if postconditions of each do not interfere with preconditions of the other. A human expert can decompose a goal into largely independent constituent parts and assign each agent to one of these subgoals to take advantage of simultaneous actions for faster execution of plan steps, each using only single agent planning. By contrast, large language models (LLMs) used for directly inferring plan steps do not guarantee execution success, but do leverage commonsense reasoning to assemble action sequences. We combine the strengths of classical planning and LLMs by approximating human intuitions for two-agent planning goal decomposition. We demonstrate that LLM-based goal decomposition leads to faster planning times than solving multi-agent PDDL problems directly while simultaneously achieving fewer plan execution steps than a single agent plan alone and preserving execution success. Additionally, we find that LLM-based approximations of subgoals can achieve similar multi-agent execution steps than those specified by human experts. Website and resources at https://glamor-usc.github.io/twostep},
note = {arXiv:2403.17246 [cs]},
keywords = {},
pubstate = {published},
tppubtype = {misc}
}
Classical planning formulations like the Planning Domain Definition Language (PDDL) admit action sequences guaranteed to achieve a goal state given an initial state if any are possible. However, reasoning problems defined in PDDL do not capture temporal aspects of action taking, for example that two agents in the domain can execute an action simultaneously if postconditions of each do not interfere with preconditions of the other. A human expert can decompose a goal into largely independent constituent parts and assign each agent to one of these subgoals to take advantage of simultaneous actions for faster execution of plan steps, each using only single agent planning. By contrast, large language models (LLMs) used for directly inferring plan steps do not guarantee execution success, but do leverage commonsense reasoning to assemble action sequences. We combine the strengths of classical planning and LLMs by approximating human intuitions for two-agent planning goal decomposition. We demonstrate that LLM-based goal decomposition leads to faster planning times than solving multi-agent PDDL problems directly while simultaneously achieving fewer plan execution steps than a single agent plan alone and preserving execution success. Additionally, we find that LLM-based approximations of subgoals can achieve similar multi-agent execution steps than those specified by human experts. Website and resources at https://glamor-usc.github.io/twostep
Gordon, Andrew S.; Feng, Andrew
Combining the Predictions of Out-of-Domain Classifiers Using Etcetera Abduction Proceedings Article
In: 2024 58th Annual Conference on Information Sciences and Systems (CISS), pp. 1–6, IEEE, Princeton, NJ, USA, 2024, ISBN: 979-8-3503-6929-8.
Links | BibTeX | Tags: DTIC, Narrative, The Narrative Group, UARC
@inproceedings{gordon_combining_2024,
title = {Combining the Predictions of Out-of-Domain Classifiers Using Etcetera Abduction},
author = {Andrew S. Gordon and Andrew Feng},
url = {https://ieeexplore.ieee.org/document/10480194/},
doi = {10.1109/CISS59072.2024.10480194},
isbn = {979-8-3503-6929-8},
year = {2024},
date = {2024-03-01},
urldate = {2024-04-16},
booktitle = {2024 58th Annual Conference on Information Sciences and Systems (CISS)},
pages = {1–6},
publisher = {IEEE},
address = {Princeton, NJ, USA},
keywords = {DTIC, Narrative, The Narrative Group, UARC},
pubstate = {published},
tppubtype = {inproceedings}
}
Frummet, Alexander; Speggiorin, Alessandro; Elsweiler, David; Leuski, Anton; Dalton, Jeff
Cooking with Conversation: Enhancing User Engagement and Learning with a Knowledge-Enhancing Assistant Journal Article
In: ACM Trans. Inf. Syst., pp. 3649500, 2024, ISSN: 1046-8188, 1558-2868.
Abstract | Links | BibTeX | Tags: DTIC, Natural Language, UARC
@article{frummet_cooking_2024,
title = {Cooking with Conversation: Enhancing User Engagement and Learning with a Knowledge-Enhancing Assistant},
author = {Alexander Frummet and Alessandro Speggiorin and David Elsweiler and Anton Leuski and Jeff Dalton},
url = {https://dl.acm.org/doi/10.1145/3649500},
doi = {10.1145/3649500},
issn = {1046-8188, 1558-2868},
year = {2024},
date = {2024-03-01},
urldate = {2024-04-16},
journal = {ACM Trans. Inf. Syst.},
pages = {3649500},
abstract = {We present two empirical studies to investigate users’ expectations and behaviours when using digital assistants, such as Alexa and Google Home, in a kitchen context: First, a survey (N=200) queries participants on their expectations for the kinds of information that such systems should be able to provide. While consensus exists on expecting information about cooking steps and processes, younger participants who enjoy cooking express a higher likelihood of expecting details on food history or the science of cooking. In a follow-up Wizard-of-Oz study (N = 48), users were guided through the steps of a recipe either by an
active
wizard that alerted participants to information it could provide or a
passive
wizard who only answered questions that were provided by the user. The
active
policy led to almost double the number of conversational utterances and 1.5 times more knowledge-related user questions compared to the
passive
policy. Also, it resulted in 1.7 times more knowledge communicated than the
passive
policy. We discuss the findings in the context of related work and reveal implications for the design and use of such assistants for cooking and other purposes such as DIY and craft tasks, as well as the lessons we learned for evaluating such systems.},
keywords = {DTIC, Natural Language, UARC},
pubstate = {published},
tppubtype = {article}
}
We present two empirical studies to investigate users’ expectations and behaviours when using digital assistants, such as Alexa and Google Home, in a kitchen context: First, a survey (N=200) queries participants on their expectations for the kinds of information that such systems should be able to provide. While consensus exists on expecting information about cooking steps and processes, younger participants who enjoy cooking express a higher likelihood of expecting details on food history or the science of cooking. In a follow-up Wizard-of-Oz study (N = 48), users were guided through the steps of a recipe either by an
active
wizard that alerted participants to information it could provide or a
passive
wizard who only answered questions that were provided by the user. The
active
policy led to almost double the number of conversational utterances and 1.5 times more knowledge-related user questions compared to the
passive
policy. Also, it resulted in 1.7 times more knowledge communicated than the
passive
policy. We discuss the findings in the context of related work and reveal implications for the design and use of such assistants for cooking and other purposes such as DIY and craft tasks, as well as the lessons we learned for evaluating such systems.
active
wizard that alerted participants to information it could provide or a
passive
wizard who only answered questions that were provided by the user. The
active
policy led to almost double the number of conversational utterances and 1.5 times more knowledge-related user questions compared to the
passive
policy. Also, it resulted in 1.7 times more knowledge communicated than the
passive
policy. We discuss the findings in the context of related work and reveal implications for the design and use of such assistants for cooking and other purposes such as DIY and craft tasks, as well as the lessons we learned for evaluating such systems.