Computer Sciences Colloquium - Enriching digital facial content through physical avatars and projection-based illumination

Abstract

Realistic facial synthesis is one of the most fundamental problems in computer graphics. While tremendous advancements have been made in capturing, animating, and rendering realistic human digital content, displaying it on traditional screens conveys only a limited sense of physical presence. In this talk, I will demonstrate how bringing this content into the physical world through projection greatly enriches said sense of presence and immersion. Beside demonstrating this concept in the setting of a social event, I will present what was at the time the first process for augmenting deforming physical avatars using projector-based illumination. The result of the system is a highly expressive physical avatar that features facial details and motion otherwise unattainable due to physical constraints of animatronics. In addition, I will present our latest advancement, which is the first system for live dynamic augmentation of novel human facial performances. We believe that the described system and its methods are the fundamental building blocks that have the potential to give rise to a large variety of new creative application scenarios in the near future.