Workflows for next-generation Hollywood film production
Background
Hollywood filmmaking is in the midst of radical change in how movies are produced. Fueled by advances in generative AI, photoreal virtual characters, and the close integration of game technologies, the full breadth of storytelling possibilities is widening for small teams of independent filmmakers with modest budgets. At ICT, we are looking for new ways that AI can be integrated into production workflows to further reduce the size of production crews: Small Teams / Big Pictures.
Objectives
Acting and Interacting in Generated Worlds is an ICT project that specifically looks at streamlining workflows for capturing the real-world performances of professional actors for playback in fully-synthetic environments. This research aims to move beyond the current trend in virtual production of filming actors on controlled lightstages and LED volumes. Instead, we seek to engineer workflows that allow the performances of actors to be captured in uncontrolled lighting environments, then retargeted onto photoreal digital doubles in photoreal digital worlds to render Hollywood-quality visuals.
We focus on these three objectives:
- Performance capture in uncontrolled lighting environments. How can a handful of filmmakers quickly capture the performance of actors with the fidelity and resolution required for downstream digital retargeting?
- Retargeting performance capture onto photoreal virtual humans. Can we fully automated the process of retargeting performance captures onto digital doubles for playback in photoreal virtual sets, e.g., the Unreal game engine?
- Interacting with objects and characters that only exist on the virtual stage. Can we provide hardware or stagecraft in real environments that can be automatically mapped to objects, vehicles, creatures, and other people, to allow actors to convincingly interact with digital things in the set of a virtual production?
If successful, this research will create new opportunities for small teams of filmmakers to utilize advances in virtual production without the need for expensive lightstages and LED volumes. By offloading to the digital realm the difficult craft of lighting actors during their performances, filmmakers can focus more directly on the craft of acting.
Next Steps
Acting and Interacting in Generated Worlds is an ICT project in its early stages. We are currently focused on this following Next Steps:
- Collaboration with student filmmakers: Critical to the success of our effort is that we develop our workflows alongside the storytellers that we aim to support. We are seeking partnerships with students at USC and beyond who are interested in applying our prototype workflows in their productions, guided by our researchers at ICT’s facility in the westside neighborhood of Playa Vista.
- Internships: We will offer one or more summer internships to undergraduate or graduate students in 2024. Information will be provided at the following webpage:
- https://ict.usc.edu/careers/internships/
- Course development: We are in the early stages of developing USC Viterbi School of Engineering courses related to this project, including an undergraduate course focused on the use of generative AI in support of storytelling, and a graduate course on Creative Technologies.
- Funding: We are actively seeking research funding to support this project.