Neverland : On Set and Solving Problems

For this short film, I was tasked with supervising a series of complex VFX shots, each requiring careful planning, problem solving, and meticulous data collection. My role began before the cameras rolled, ensuring the groundwork was laid for a smooth post-production process.

We reviewed the shots that were going to need visual effects and what we would need to do on set to make sure they could be completed as efficiently but realistically as possible. I made a list of the data that would be needed on set as well as ways to tackle specific shots.

Lens Distortion Mapping

The first step was mapping the distortion of each lens used on set. Working with the camera department, I aligned the camera sensor with a distortion grid displayed on a large TV. We captured footage of the grid at various distances and with each lens. These recordings enabled me to undistort the lens characteristics during post-production, ensuring more accurate 3D tracking and seamless compositing of elements into the final shots.

Color Matching with Macbeth Charts

For shots requiring CG elements or when time allowed, I captured footage of a Macbeth color chart placed at the center of the scene. This provided a reliable reference for matching colors during compositing. By rendering a virtual version of the color chart within the 3D scene, I could ensure precise color consistency between real and digital elements.

360 HDRi Data

To recreate realistic lighting and reflections, I used a 360-degree camera to capture exposure-bracketed images for HDRi creation. These HDRi maps provided detailed lighting information for CG renders.

Photogrammetry and Reference Materials

Whenever possible, I performed photogrammetry scans of actors and key props, capturing detailed reference imagery to aid in creating accurate digital doubles or CG replacements. These scans were instrumental in achieving high-quality composites. Additionally, I used a LiDAR scanner to create accurate 3D scans of rooms and tracking markers, further enhancing the fidelity of reflections and lighting in the composited shots while improving tracking data.

Witness Camera Integration

In one particularly challenging scenario, traditional 3D tracking was difficult due to subtle movement and lack of points to track due to a very narrow focus depth. To overcome this, I attached a 360-degree camera to the principal camera as a witness cam. By measuring the offset between the 360 camera and the principal camera’s sensor, I ensured accurate 3D tracking and integration of CG elements.

Creative On-Set Problem Solving - Set Adjustment

Last minute complications would arise such as set design issues that last minute were requested as possible VFX fixes which I needed to find solutions for on the spot. In one instance, the camera needed to move to move around the actor to reveal the other side of his face, unfortunately this would reveal a section of the set that was undecorated and filled with equipment, with little time to get the shot, I ask asked if I could fill this wall in to hide the area. I quickly threw up some tracking markers and my color chart was already mounted in that location and was able to be recorded in the shot.

Creative On-Set Problem Solving - When To Use A Gaussian Splat

In another instance they wanted to capture a complicated camera dolly that would be nearly impossible due to the angle and location of the dolly as it would not only appear in frame , but obscure critical focal points the camera would need to see to convey the story. At this point I was asked what I could do. I quickly weighed my options and considered the context. Typically this would be tackled by 3d tracking the scene and rebuilding the obstructed elements, this obviously takes time to make it look perfect. I think considered my second option, the scene had 0 movement from the actor who was facing away from camera as it moved past him, the only thing in the scene that would be moving is the static that was being displayed on a TV, something that is very easy to comp in later. I decided this is the perfect use case for a gaussian splat. This technique, to make a long story short, would require several passes with the hero camera passing through the general path of the desired dolly move, each pass covering a slightly different angle, while everything in frame stays as still as possible. This was easy for the actor as he was sitting, but in a similar situation where the actor is facing away, is not the focal point, and is also out of focus, it might be worth comping them back in later. Once a few passes are recorded, I can train this data on my computer specifically for AI training while I can take that time on my compositing computer on other shots that way no time is wasted. After sometime of training, I now have a perfect, indistinguishable 3D recreation of the scene, reproducing reflections and lighting as well as the cameras color and lens profile perfectly. Now I can move my camera around in 3D space (within a relatively similar POV) and recreate a perfect dolly shot that is both infinitely better than a traditional composite as well as with 0 lost time that would need to have been spent on that composite.

This project demonstrated the critical role of on-set VFX supervision in bridging production and post-production. By proactively collecting data, solving technical challenges, and ensuring precision, I laid the foundation for a smooth and successful VFX pipeline. The result was a visually cohesive and polished final product that met the film’s ambitious creative vision.