By popular demand, this blog is a summary from Session 1 of my Masterclass in Virtual Production held at the Cannes Film Festival, as part of the Cannes NEXT program, May 19. 2022
Created & written by Cinematographer Jannicke Mikkelsen, FNF
Instagram @jm_fnf ✉️ jm@jmfnf.com
Virtual Production is an extremely wide umbrella term. In its simplicity, anything in the film industry that has been converted to digital can be considered Virtual Production. Referred to as VP, 'Virtual Production' is a recently coined phrase and buzzword that poorly describes with any accuracy what it actually is. Virtual Production can be assumed to be Metaverse, gaming, XR, VR, AR or NFTs. VP can also be video projection behind actors. One aspect of VP is basically just an adaption from analogue to digital workflow in the film industry, whilst another aspect of VP is the introduction of an entirely new work force entering the film industry; New people popping up on film productions with whom the rest of the film crew have no clue what do. Let me explain:
Digital World = Virtual World
In my 15 years of working in the film industry, I feel lucky to be part of the rapid change in technology that make film productions possible. Back when I was a film student I felt I had to lie and pretend I knew how to shoot analogue 16mm/35mm film. I didn't. At that time it was just assumed that if you were in the film industry you had worked on industry standard film cameras shooting film. This in turn meant you knew how to correctly expose the silver emulsion on celluloid film, but worse of all, it was assumed you knew how to load film, and every model of film magazine required its own sequence of threading and care. Trust me, when you are an inexperienced film student there is no bigger fear in life than working blind while loading or unloading a reel of exposed film from magazine to film lab can. Later, I’ve understood that the phenomena of having only experienced digital technology has a name: - Millennial.
Enter two decades later, and now we are all talking about virtual film productions, and the process is equally as mystifying to film creators today as celluloid film was to me back when I started.
First I'd like to thank Marché Du Film - Festival De Cannes & Cannes NEXT @mdf_cannes and EPIC & Unreal Engine for inviting me as a Virtual Production expert to host a 3+ hour Masterclass and attend the Virtual Production panel discussion.
I'd also like to thank Film Soho who hosted the Cannes NEXT Virtual Production Masterclass at their fully kitted out pop-up virtual studio located in the V-Studios tent at Le Grand Hotel Cannes gardens next to the Tom Cruise Top Gun Maverick Exhibit. Film Soho is a group of innovative brands that encompass the entire film and TV ecosystem. Under the Film Soho umbrella is V-Studios who will allow filmmakers to place actors in virtual environments without ever having to leave Central London. Disguise, n-cam, CVP Group & ZOAN who are all leading innovative companies in virtual production and metaverse technology.
What is Virtual Production?
At the Cannes 2022 Masterclass on Virtual Production we focused on Virtual Studio productions in the Unreal Engine. We were situated in the fantastic V-Studios tent with a state-of-the-art Virtual Production stage featuring an LED backdrop and a dessert rock film set to match the digital backdrop. This allowed me to create an interactive hands-on workshop within the masterclass itself.
A shift happened in the film industry round about 2007 - 2008 when digital cameras and digital rushes became a serious contender in film production. Take note that this also corresponds to the exact same time of the global economic crisis at that time and the film industry had to adapt rapidly to survive. At this time I was a student and digital was simply there, available to explore and didn't cost me a penny to shoot. Jump ahead to today's situation and at the tale end of a global pandemic 2020 - 2022. Virtual Production allows filmmakers to explore their scripts, and combined with digital cameras on cellphones, everyone now has a virtual film studio in their pocket.
Virtual Production (VP) is a loosely defined term for any digital workflow or digital creation in the production of a movie or series, and VP is neither restricted to the creation of fiction nor documentary. Digital workflow and ‘cloud’ can also be defined as virtual production, and most film productions already use digital assets (VFX) which have now been re-branded under the umbrella of VP.
The reason for the rebranding of digital technology is in part due to the rise of highly effective game engines driving the future of image technology.
Gaming technology from the gaming industry is infiltrating film production, simply because: It Makes Sense.
The most obvious challenge we face in Virtual Production is the merging of two worlds: the real world & the virtual world. The responsibility asked of me as the Cinematographer is to coordinate both worlds and know these worlds cinematic possibilities and limitations in order for the production to be able to seamlessly merge both worlds. Beyond the role of a VP Supervisor, the Cinematographer is in charge of the artistic interpretation and cinematic sensitivity of which the script demands, and most importantly of all knowing how to steer the film crew in a unified direction to achieve the artistic vision on time and on budget.
Session 1 of this VP masterclass can be defined as 'Virtual Production Lite'.
All you need to know about Virtual Production in 45minutes
This is the skinny on Virtual Production. What is it and who needs it? What kind of virtual production should you go for and how to decipher what virtual production elements are right for your production?
One contributing factor as to why the definition of virtual production and digital technology is as confusing as it is, is due to what we in the tech industry comically refer to as 'Moores Law'. Gone are the days when you could learn your craft as an apprentice working with celluloid film to hone your skill until you became a Master. Digital is forever evolving and the rules of the game is constantly changing with the improvement of technology. If you’re not on top of your game continually learning and updating your knowledge, your apprentice will swiftly surpass you on the tech-game. With this in mind, here begs the next question: What drives a film production when introducing virtual production? Technology or art? In virtual production it is easy to allow the virtual tail to wag the artistic dog.
When you enter the world of virtual production there are two main capture method categories of virtual film production. These two main methods are:
Virtual Shoot & Virtual Studio
Out of utter industry confusion over what virtual production(VP) actually is; I’ve chosen to define and divide VP into two main categories to reflect the location of where the virtual production takes place:
Virtual Shoot is when your film location takes place inside of the game engine and you use virtual cameras to shoot your production.
Virtual Studio is when you are on located in a studio with an LED screen as your backdrop and shooting your production with a regular camera with optical lenses.
I’ve experienced the role of a cinematographer to become increasingly demanding with the rise of virtual production. I see the increasing benefit of having two cinematographers collaborate on the same film production, where the one is the traditional DoP and the second role is a supporting cinematographer who is head of the VP department: AKA the Virtual Cinematographer. This is only one of many new roles entering film productions in the near future as they adapt and convert to virtual production.
Virtual Studio - it's the one where you use a virtual backdrop.
Short recap of the use of movie backdrops. All films shot in a studio use backdrops, whether it is a photographic backdrop, a painted diorama canvas, or a chroma key blue/green screen. Rear projection was one way of creating video action behind an actor, which later got replaced by the blue screen and then the green screen. Now we have LED screens and photorealistic volumetric universes built in game engines that get projected on to LED backdrops, simulating the same technique from the days of rear projection. The main difference is we can now move the camera that we are shooting the movie scene with - and the background perspective projected on to the LED backdrop moves with the point of view of the camera.
Our virtual backdrops locations are built in game engines to create a 3D model of the movies environment where we can move anywhere within that location. These are volumetric locations and can also be referred to as metaverse locations. It is also interesting to note here that these metaverse locations can be made into NFTs and sold, traded, and most noteworthy, rented out to other film production as a virtual landscape.
You can now own a virtual film location as an NFT and charge rent for future film productions who wish to shoot at your virtual location.
The stage in front of the LED screen is like a regular traditional film set smartly staged to blend seamlessly in with the backdrop. The scene is shot with a regular movie camera kitted out with tracking technology which links the physical camera with the virtual world projected on the LED backdrop. What is actually happening is that your real world camera is now a digital twin with the virtual camera which exists inside of the game engine. That virtual camera image is the one you see on the LED backdrop and your real world camera is the one that is capturing the action filming the image on the backdrop behind the actors. -Remember I said this first session, session 1, is VP Lite. We can get in to all of the details in the upcoming sessions.
Roughly explained, when shooting your movie in a Virtual Studio which uses an LED backdrop, you will tether the virtual camera to your real camera. Essentially in a Virtual Studio setup you can view the virtual world as the digital twin to the real filmset you are shooting on. This is why you need tracking devices on the real camera, tracking devices like n-cam who we used on this stage demo at Cannes Film Festival, Cannes NEXT 2022.
Virtual Shoot - it's the one where nothing exists in reality.
Your entire movie scene exists inside of a game engine. Actors, action, extras, location, props, camera, lights, everything exists inside of the game engine. The only way you can access this work is through a computer screen, monitor screen, or virtual camera monitor.
Your film location is created by 3D artists who create a photo realistic environment where you can have virtual actors or real actors translated into virtual actors through motion capture (mocap). Again, remember this is the VP Lite session of the masterclass and we can dive in to this technology in the VP Super Geek session 3. All that is important to note here is that the entire movie scene takes place inside of a game engine and that is where you can access your movie universe.
The Virtual Shoot category of Virtual Production has a plethora of exciting new tools for filmmakers. I am a cinematographer and within Virtual Shoot I have discovered Virtual Cinematography and this is one of the newest roles entering film productions.
Virtual Cinematography shouldn't be confused with the work of a regular Cinematographer. The skillset of a Virtual Cinematographer is far more computer-technical and the main priority of the Virtual Cinematographer is to translate digital technology in to real life execution for the traditional Cinematographer. This is when it may be useful to introduce the definition IRL, In Real Life, to distinguish between the two worlds we are constantly jumping in and out of on a virtual production. On a virtual production we now have the Virtual Cinematographer and the IRL Cinematographer.
IRL = In Real Life
Virtual = Not Real Life
Virtual Shoot has a few advantages we can't easily access In Real Life (IRL). One of these incredible added advantages is for me as a cinematographer to be able to shape shift size to capture the action happening in the scene. In the virtual world reality is relative. This means that every movement I make inside of the game engine is assigned a numeric value. In turn this means my position, movement, weight, and size is assigned a numeric value. This means I can be any size I want, and by this I'm able to run around my scene and explore every angle of my virtual film set. Because size is just a numeric value, I can now decide to make my real step to the left IRL a 1:1 scale step to my left, or a 500mile step to my left. Size is relative and so is reality inside of the game engine.
Because everything is relative, figuring out what numeric values simulate reality becomes its own art form. Because reality is relative, you as the now camera operator must find the numeric value that corresponds to what essentially feels 1:1 with your movement.
Rule #1 of Virtual Worlds: Reality is Relative.
Calibration of numeric values that simulate reality is important to figure out as a base line for shooting a virtual production on a Virtual Shoot. These numeric values are never the same and will differ from person to person and from virtual location to the next. The art is in your interpretation of reality.
Virtual camera's are genius in that you can shoot your movie anywhere at any time. All you need is your laptop and a smart phone or tablet. Within the V-Cam app you will discover all the various parameters of numeric values you can adjust and experiment with as to what values suit your movie script and cinematography.
A Virtual Camera is just an app on your phone or tablet 📲
Since size is now relative , you can also turn yourself into a miniature cinematographer and film extreme closeups and macro details. What the v-cam enables you to do is to walk around with a view finder on the virtual set and explore your virtual location, re-light your location thanks to raytracing, and through a library of virtual lenses pick any lens you've ever wanted to test. They may not look exactly like the optical lenses, but their characteristics have been pretty closely matched in the virtual camera tool set.
Rule #2 of Virtual Worlds: Know your orientation & attitude.
Rule #2 becomes applicable when virtual tools are paired with the virtual location, such as shooting with a virtual camera. On some occasions you may find that you are challenging the laws of physics when operating a virtual camera within the virtual shoot. The many challenges you run into will depend on in which coordinate quadrant you virtually exist.
The entire universe of your game engine is a sphere with the coordinate axis X, Y, Z and a point of Origin (0,0,0). When you create, build, and 3D model your virtual location, the direction UP ⬆️, the Zenith, is generally assigned the +Y axis. Direction DOWN ⬇️, the Nadir, is the -Y axis. +X & -X will be your side to side movement, and +Z & -Z will be your forwards and back movement. There is also a virtual camera inside of the game engine which represents your IRL virtual camera's digital twin. Your IRL virtual camera can be your phone or tablet running the virtual camera app.
Example:
You stand in the physical IRL world whilst you operate the virtual camera inside of the game engine. In example you use your phone as your virtual camera, which now controls the virtual camera inside of the game engine. Every movement you make with your phone is replicated by the digital twin virtual camera inside of the game engine.
It helps to imagine yourself standing inside of the coordinates and make a mental note of where you are in relation to the point of Origin. In aviation we use the term 'attitude' to also define our orientation in relation to a reference point. I use the first quadrant (+X, +Y) as my reference point and map this out on the studio floor in front of me. When I walk on the studio floor I can define the numeric values to represent the distance I cover in any of the directions of movement (X,Y,Z). Your virtual camera will also PITCH, ROLL and YAW. Therefore I find it easier to think in aviation terms when controlling my attitude and orientation to establish my position in the game engine. In camera language the PITCH = TILT, ROLL = Dutch tilt/canted camera, YAW = Pan.
However, the virtual axis of orientation can become greatly confusing and disorientating if you happen to venture in to the negative -Y, -X, -Z zones of your virtual location. My experience of camera operating whilst accidentally stumbling in to the negative zones of the sphere can best be described as walking in to some sort of a multiverse where everything is similar but your camera controls and movement of your own body operating the camera is mirrored, or even flip-flopped and mirrored together. Typically outer space is one of the virtual locations that can cause tremendous headaches over where in space you are orientated and what power of gravity your virtual actors and virtual camera is interacting with. Consider yourself warned.
Exploration of a Virtual Location is one of the most fun things you can do as a creator. However, as a cinematographer, Virtual Production is like the ultimate candy store! In real life a great deal of our work and tests are restricted by budget and logistics. In VP previz however, budget and logistics issues are close to eliminated and our creative options near unlimited. As long as your are equipped with a laptop with a fairly new CPU/graphics card and a smart phone you can start previs-ing your movie. As a cinematographer I can explore my virtual location with a phone that gives me the simulated image capture of a 70mm IMAX camera. I can just as easily with the swipe of a finger switch to a Large Format Digital camera, just to see what this camera setup and lenses may give me Vs. the IMAX camera. I can access a near unlimited selection of lenses and test out the characteristics typical for each lens set. Want to shoot anamorphic, but production says you can't afford it? Explore your scene and see if there is a surplus equipment you can do without to tighten up the budget to afford you those lenses you've always wanted to shoot on.
What Virtual Production will you use?
Virtual Shoot or Virtual Studio? The reality is all film productions will most probably see the advantage in using a combination of both categories of VP throughout the production process.
I've listed 5 VP processes that I find the most beneficial: Location Scouting: When I go out location scouting I bring my phone with me and use an app called 'Polycam' to scan my location in 3D. I also use a 360 camera or sequence of stills to stitch together a 360 image of the location. The location can now be imported in to a game engine and serve as a blueprint for the virtual recreation of the location. Tech Recce: Traditionally all Heads of Department's will go out together on the Tech Recce and assess the location for shoot suitability and logistical requirements. Tech Recce's also happen in VP once the virtual location has been created. In VP the HOD's of the film crew are assessing the location for assets they will recreate as practical set builds in the Virtual Studio.
Scene Previs: A previs can be created for the entire movie or single scenes.
Digital Twin: Creating a digital twin of your film set, or creating a film set to mirror your virtual location.
Postvis: Creating a virtual simulation of the action that was recorded on set to assess the needs of VFX. The postvis can and should be the same as the previs with improved assets.
Virtual Production must adapt a holistic workflow to succeed
Unreal Engine have a useful flowchart outlining how virtual production fits in to a modern day film production pipeline. All digital assets are recycled and upscaled, not discarded. Virtual Production is not a 3D story board or previs that is discarded as soon as the production shoot start. The previs is integrated as part of the production backbone and streamlines production from start to finish.
As part of the holistic VP workflow I've mapped out the crucial start of film production, in the gap between idea/ written treatment - and the cluster of art dept/previs dept/virtual art dept/vfx dept.
It id detrimental to any film production that the traditional HOD's are involved at this start phase at the same time as the crew from VFX studio. Virtual Production is not in place to push out the traditional creators, but aid as a tool to upgrade and streamline film production. This in turn begs the question asked by all producers: How much will VP cost?!?
Virtual Production requires an updated financing model for film production. The costs are up front, however the total budget = the same as previously.
Budgeting for virtual production carry a lot of the costs up-front as all HOD's are involved in the creation of the previs and early creation of the virtual location. A virtual production may start with a small core consisting of the above the line creatives, but soon blooms out to include all HOD's to weigh in on the best solutions based on their expertise. Funding for virtual production requires costs to be covered upfront and a financing model may lean towards production financial loans vs. traditional investment models.
Can and Cannot's in Virtual Production. There are pros and cons to all VP categories and methods. Looking at specifically a Virtual Studio production there are a few capabilities that differ from a traditional VFX shoot.
Where did the green screen go?
Virtual studio shoots no longer require green screens to superimpose a digital video backdrop behind the actor. However you may still wish to combine VP with green screen technology.
A few frequently asked questions are: Can I change the background in post production?
-No. The location that is projected on the background is your location and to change location would require to re-shoot at a different location, which in VP means to create an entirely different 3D virtual location.
Can I shoot green screen?
-Yes. If required you can shoot both virtual and green screen by alternating the projection on the LED screen. At this stage in technology development I would say it is too early to do this successfully but the possibility exists and is under development. You can also shoot a clean green screen which has the advantage of no green spill in the environment around when projected from the LED screen rather than a green chroma key drape.
Can I change assets?
-Yes. However you can only do this easily at the very start of a VP production before the customisation begins. The drawback with changing, replacing, or adding assets is that the creation takes time. An asset is anything that is created in 3D and exist in your virtual location. The virtual location itself is an asset, so is the actor, their clothes and the prop they are holding. An asset is created by a skilled 3D artist who needs time to create their art. 3D artists are always under gnarly deadlines and as such experts in delivering high quality assets in a short space of time. The downside to not having all your assets mapped out in the pre-production stage is that you risk having an entire film crew on standby for two days until your asset is created that is vital for your script. This is yet a strong argument for starting all VP productions with a virtual shoot previs no matter how you wish to shoot your movie. Can I change the Actor? -Yes & No. An actor in VP is an asset, so therefore, No. However, you can change the mocap movement and you can also change the face of an actor if necessary. It's not easy, and it's not cost effective, but yes it is possible.
Can I change the camera angle? -Virtual Shoot: -Yes. -Virtual Studio: -No. A virtual shoot takes place within the game engine and is the same technology wether it is previs, principle photography or postvis. In a virtual studio production you are able to move the camera while shooting principle photography. No you can not change the camera angle in post.
Can I change the lighting?
-Virtual Shoot: -Yes.
-Virtual Studio: -mostly No.
Thanks to the incredible technology of ray tracing in game engines you can move the sun and simulate any time of day with accurate bounce of sunlight reflecting from the surfaces and textures in your virtual environment. You can also add some smaller cheat lights in your environment for cinematic effect. This part of the VP process you can change right up until end of post-production. In a virtual studio you will have set the light to reflect the time of day in the script and balance out the IRL stage to reflect and blend with the lighting environment in the virtual location behind the actors. To a degree the lighting can easily be altered or adapted in post production, but you do run the risk of the lighting on set to not blend or match with the backdrop.
