PHOTO CREDIT: Paolo Villanueva
by Spencer Lin, ICT’s XR Researcher and Developer, and Masters, Computer Science student at USC Viterbi School of Engineering
Outside of my work at ICT (but highly connected in terms of skillset), I’m often participate in hacking events with other students, from USC and other institutions. Recently, a bunch of us connected at MIT Reality Hack in Cambridge, MA, to form Team Capybara – and returned with the Best Use of Apple Vision Pro award for our colocation multiplayer social AR app: Paw Pals Connect!
TECH DETAILS
Paw Pals Connect is a colocation multiplayer social AR app that features off-cloud LLM-driven virtual dogs designed to connect you with fellow virtual pet owners. It was built using the Apple Vision Pro and Unity, leveraging an open source MIT-licensed multimodal agent framework, Estuary, to drive the agent behavior.
The vision is for your Paw Pal to learn more about you over time to create happy coincidences in real life so that you can organically form new friendships with real humans.
Pets understand users using an off-cloud Large Language Model to drive pet behaviors such as intelligent pathfinding based on semantic understanding of the environment (ie. a pet will understand what a chair is automatically and perform contextually appropriate actions such as jumping onto it).