Virtual Reality (VR) is hard. The devices are pricey (but becoming more affordable), the workflow is nightmarish and what exactly should be filmed in 360 degrees remains more or less unknown.
In the last century of cinema, we’ve developed a vocabulary for the screen. We experimented and figured out how to utilize the medium to tell amazing stories. VR, like the invention of cinema, has its own vocabulary and language for directing and focusing attention. So, it’s important we treat VR as a separate medium. What works with 2D video cannot simply always be ported over to 360-degree video. There are no close-ups, pans or zooms to focus the viewer’s attention.
360 Video Production:
The first thing that is unique to the VR filmic language is proximity. What’s closer to the camera seems more important and we focus our attention on that subject. Things farther away are less detailed and become less interesting. The right proximity to grab attention may even be a matter of a few feet, because just a few steps away from the camera can drastically minimize a subject. Such is the nature of fisheye lenses used on 360 rigs.
Secondly, because viewers can literally look anywhere in 360 degree space, centering is a crucial part of the VR filmmaking process. Centering is directing the viewer’s attention to the center of the scene through either cutting, movement or proximity (as mentioned above). The first shot of a VR film is the most important because it’s the only time you have complete control over where the viewer will look first. After the first shot, use cuts to keep the focus on your subject(s). And take advantage of your subject’s movements to grab and maintain attention.
Creating a spatial audio experience:
We’re not cool like dolphins with their echolocation abilities, but humans rely heavily on hearing to locate and place where sounds come from. Try closing your eyes in a room and have a friend throw something on the floor. You’ll likely be able to place the origin of the sound just from hearing it. In VR, this means creating a binaural audio experience, which means we hear things just as if we were really there. Turn your head left, the sounds changes. Turn your head right, the sound changes accordingly. Now what’s really cool is how accessible recording and encoding spatial audio (VR term for binaural sound) is. This is a huge step forward for VR storytellers!
Post-production pipeline:
There’s a big difference between editing standard video and 360 video. It’s surprisingly simple once you know where to start.
- First, stitching: If you shot your video with a 360 video array, you’ll need to stitch all that video together before you can edit it. Currently, the best option for stitching video is Autopano Video from Kolor, a company that’s part of GoPro family. It lets you quickly stitch footage shot on multiple cameras. Equirectangular video is the most popular 360 video format. It essentially manipulates an entire 360 video into a rectangular format that is very reminiscent of HD or 4K footage. In fact, the exact aspect ratio can vary based on your specific project. Unrendered equirectangular footage can technically be watched on a normal video player, but if you want to view the video in 360 or experience it on a headset, you need to watch in a player or app that supports 360 video, like YouTube. There are also a few other types of 360 video formats, like circular and bicubic, but none of these formats are particularly user-friendly or practical compared to equirectangular video.
- Editing 360 video: Premiere Pro is not only the most popular professional video editing software in the world, but it’s also one of the only professional video editing softwares that allows 360 previewing. Editing 360 video is actually very similar to editing standard video. All of the same concepts like syncing, cutting, dissolves, and levels still apply to 360 video. It’s just important to know that if you want to add in special objects (like graphics) you’ll probably want to use a third-party app like After Effects paired with Skybox Studio to make sure your graphics aren’t distorted in playback.
- Previewing 360 video: Previewing 360 video is incredibly easy from Premiere Pro. Simply drop your equirectangular video into your timeline, and select the “Toggle VR Video Display” button found in the menu in both the Preview and Program monitors. Once you select this button, you’ll be able to pan and tilt the camera around in 360 degrees with either your mouse or the dials on the side of your preview or program monitor.
- Exporting 360 video: Exporting 360 video is a little different than exporting standard video. Namely, you need to add the appropriate metadata to your video. To do this, simply go to File>Export>Media. This will pull up an export panel. Adjust your video settings as you see fit. I recommend using a less compressed format like ProRes, since 360 video can accentuate compression artifacts, and of course use 4K if possible (the higher the better). With 360, you can only see a portion of the raw video at once. So even if the raw file is 4K, the portion within your viewing angle will be closer to 1080. Think of the raw file as a map, while the 360 player shows one angle of a globe. Once you’re ready, it’s time to upload.