Programmer/Research Assistant Intern 0929
January 08, 2009
Project Name
Multi-modal Sketch & Speech interface for vhumans and virtual worlds
Project Description
The ICT Virtual Human project is pushing the state of the art in technologies for modeling natural language dialogue, emotion reasoning, and non-verbal communication for animated characters. We have also recently been working on a project (Deep Green) for conveying military courses of action by sketching symbols and speaking. This project would explore the connection of these two streams of work – can we re-use the integrated sketching and speech interface for communicating with virtual characters in a virtual world? This is particularly important given the increasing popularity of sketch and touch interfaces such as tablet computers and mobile computing devices such as the iPhone. We foresee the following types of interaction where sketching may help:
- User can ask questions about the environment by using sketches to circumscribe objects or groups of objects.
- User can show a virtual human where to go, by sketching the path
- User could dynamically change the environment by sketching an object and location to put it in
Job Description
The intern will work with research staff and graduate students to design a specific set of objectives and look at mapping from the sketch space to the virtual world coordinates. A specific set of capabilities will be agreed on, and the intern will implement the translation procedures so that information from sketching and speech can be conveyed to the virtual humans in terms of the virtual world knowledge. The extensions will be evaluated by having users try to communicate with the virtual human on simple tasks involving these capabilities compared to trying to communicate using only speech or simple pointing gestures.
Skill set:
- Strong Java programming skills
- Some familiarity with sketch recognition, game engines or multi-modal fusion
- Natural Language processing (especially semantics and dialogue) background is a plus.