280 – Programmer, Head-Mounted Facial Capture and Rendering for Augmented Reality

Project Name
Head-Mounted Facial Capture and Rendering for Augmented Reality

Project Description
The lab is developing techniques for enabling natural and expressive face-to-face communication between subjects in an augmented reality (AR) environment by removing the barriers introduced by immersive headmounted displays (HMDs). The degree to which users in AR environments can expressively interact with each other is hindered by HMDs, which occlude a large portion of the face. We propose a method to overlay a virtual face that replicates the subject’s appearance and expressions using facial performance capture. While state-of-the-art real-time face tracking technologies fail in the presence of occlusions, recent efforts by USC CS and USC ICT Graphics Lab and have resulted in algorithms and systems that allow for wearers of virtual reality HMDs to transfer their expressions to an avatar using sensors mounted to HMDs.

Job Description
Intern will work with researchers to develop a prototype AR HMD device based on Microsoft’s HoloLens and develop new algorithms for light-weight facial performance capture, as well as new techniques for appearance synthesis. Intern will support research efforts to fill in occluded facial regions using a digital face. This will also require capturing and rendering the dynamic lighting conditions on the face.

Preferred Skills

  • C++, OpenGL, GPU programming
  • Experience with computer vision techniques: multi-camera stereo, optical flow, facial feature
  • Operating System: Windows
  • Experience with detection, bilinear morphable models, texture synthesis, markov random fields

Apply now.

Go back to the summer research program list.