Virtual Production Making-Of
For the narrative music video series NEOSHIN, a wide variety of techniques were used. For decades, film and TV productions have used green and blue screens to place actors into new environments. Now, many of these shots can be achieved directly in-camera by using large LED walls in combination with tracked camera data and game engines in order to project 3d environments in realtime behind actors. These methods were put to the test on NEOSHIN's ``EPISODE 01: COLDBLOOD``, in which the real world band has been transferred into a virtual nightclub in the year 2073X running in Unreal Engine 4. This process of combining traditional cinematography techniques with advanced world-building technology effectively eliminates the need for a green screen while providing more accurate light interaction and reflections.
Motion Capture Making-Of
To bridge the gap between today and the year 2073X the full performance of NEOSHIN has been captured using state of the art motion capture technology. Making use of 24 Optitrack cameras, every little detail of the full music performance is captured and transfered onto the digital avatars. In addition to the band performance, the acting for the protagonist Ayuko is driven by Motion Capture as well. Using a head-mounted GoPro camera along with the mocap system allows us to fully capture both gestures and facial expressions to translate onto the digital double. As a final step, the outfits of NEOSHIN have been digitized using an advanced photogrammetry scanning rig. Combining the scanned outfits with the captured performance allows us to create a photorealistic virtual version of the band.
Photogrammetry is the science and technology of obtaining reliable information about physical objects and the environment through the process of recording, measuring and interpreting photographic images and patterns of electromagnetic radiant imagery and other phenomena. There are many variants of photogrammetry. One example is the extraction of three-dimensional measurements from two-dimensional data (i.e. images); for example, the distance between two points that lie on a plane parallel to the photographic image plane can be determined by measuring their distance on the image, if the scale of the image is known. Another is the extraction of accurate color ranges and values representing such quantities as albedo, specular reflection, metallicity, or ambient occlusion from photographs of materials for the purposes of physically based rendering.