ShowMak3r: Compositional TV Show Reconstruction

Reconstructing dynamic radiance fields from video clips is challenging, especially when entertainment videos like TV shows are given. Many challenges make the reconstruction difficult due to (1) actors occluding with each other and having diverse facial expressions, (2) cluttered stages, and (3) small baseline views or sudden shot changes. To address these issues, we present ShowMak3r, a comprehensive reconstruction pipeline that allows the editing of scenes like how video clips are made in a production control room. In ShowMak3r, a 3DLocator module locates recovered actors on the stage using depth prior, and estimates unseen human poses via interpolation. The proposed ShotMatcher module then tracks the actors under shot changes. Furthermore, ShowMak3r introduces a face-fitting network that dynamically recovers the actors’ expressions. Experiments on various datasets show that our pipeline can reassemble TV show scenes with new cameras at different timestamps. We also demonstrate that ShowMak3r enables interesting applications such as synthetic shot-making, actor relocation, insertion, deletion, and pose manipulation.

Overview of our ShowMak3r pipeline. Given a TV show video clip, we perform dense reconstruction of the stage, locate SMPL models to the stage with 3DLocator, associate SMPL models across shots to track actors with ShotMatcher, and recover the detailed appearance of the actors with Face-Fitting network. 3D Gaussians of the stage and the actors are rendered to produce novel frames.