Digital Ira and Beyond: Creating Photoreal Real-Time Digital Characters

August 10, 2014 | Vancouver, Canada

Speaker: Paul Debevec, Graham Fyffe, Hao Li, Javier von der Pahlen (Activision), Jorge Jimenez (Activision), Etienne Danvoye (Activision),
Host: SIGGRAPH 2014

This course summarizes the process of creating Digital Ira, presented in SIGGRAPH 2013 Real-Time Live! It covers the complete set of technologies from high-resolution facial scanning, blendshape rigging, video-based performance capture, animation compression, and realtime skin and eye shading to hair rendering. The course also presents and explains late-breaking results and refinements, and points the way to future directions that may increase the quality and efficiency of this kind of digital-character pipeline.

For this project, an actor was scanned in 30 high-resolution expressions, from which eight were chosen for real-time performance rendering. Performance clips were captured using multi-view video. Expression UVs were interactively correlated with the neutral expression, then retopologized to an artist mesh. An animation solver created a performance graph representing dense GPU optical flow between video frames and the eight expressions. Dense optical flow and 3D triangulation were computed, yielding per-frame spatially varying blendshape weights approximating the performance.

The performance was converted to standard bone animation on a 4k mesh using a bone-weight and transform solver. Surface stress values were used to blend albedo, specular, normal, and displacement maps from the high-resolution scans per-vertex at run time. DX11 rendering includes SSS, translucency, eye refraction and caustics, physically based two-lobe specular reflection with microstructure, DOF, antialiasing, and grain.

The course explains each of these processes, why each design choice was made, and alternative components that could replace any of the steps. It also covers emerging technologies in performance capture and facial rendering. Attendees receive a solid understanding of the techniques used to create photoreal digital characters in video games and other applications, and the confidence to incorporate some of the techniques into their own pipelines.