The Future of VR Filmmaking

These are my notes on the “Future of VR Filmmaking” panel at the NYVR Expo, along with some some of my own thoughts interspersed. Rather than attempt to convert my notes to coherent paragraphs, I’m just presenting them here the way I recorded them in the interest of getting the info out there.

Panelists:

• Matt Braly (Littlstar, https://twitter.com/LittlstarVR)
• Brandon Henriquez (Pretend Labs, https://twitter.com/brandonlyfe_)
• Daron Jenkins (New York Film Loft, https://twitter.com/nyfilmloft)
• Jodi Schiller (https://twitter.com/jodischiller, http://www.newrealityarts.com/)

Some definitions of terms used in this post:

360 Video – you can look around (up, down, left, right) but you can’t move (yet!) [the word “yet” was used many times this weekend. Clearly there are expectations for big developments in VR and AR in the future]

VR – Virtual Reality – allows you to move around in the space, and possibly interact with your environment and the objects around you. Also sometimes used interchangeably with 360 Video

AR – Augmented Reality – overlaying virtual objects into the real world.

MR – Mixed Reality. Sometimes used interchangeably with AR. Microsoft uses this term to refer to AR that is contextually aware of the world around it. It recognizes real-world objects and allows interaction with AR appropriate to those environmental elements. For example, in mixed reality, a virtual person might be smart enough to recognize and sit in a real-life chair.

Notes:

VR vs. Film:

• VR and Film are different storytelling mediums. [ie. what works for one won’t necessarily work for the other]
• We need to determine what are the best kind of stories for each medium.
• VR provides for more presence, empathy, interaction, and discovery.
• VR allows more agency and world building  [both are opportunities and challenges]
• VR is more ephemeral but more emotion-changing.

We have to determine: How can you most effectively/efficiently change the emotional state of the viewer?

Jodi, who comes from a theater background, referred to VR/AR as “Theater times 1000.”

Jodi talked about the under-representation of women in so many fields, and her group “AR VR Women and Allies” is aimed at preventing that in VR starting now, on the ground floor, from the beginning of this medium. Facebook group: https://www.facebook.com/groups/womenarvr/

In VR, the film maker loses control of the viewer’s focus. Since the viewer isn’t just starting straight ahead at the screen, they can look anywhere. As a result, Matt said, “the environment is always going to be a character in your story.” [I thought this was a great way of describing a profound truth about VR!]

Referring to VR filmmaking techniques, “Everything is still so new, just try lots of things out!”

Getting the viewer’s attention

• Filmmakers need to pay more attention to narrative magnetism to grab the user’s attention. [Traditional magnets, like making an object brighter, or centering something in a frame, or using movement in an otherwise static scene won’t work if the viewer isn’t already looking in the right direction.]
• Spatial sound (where the viewer can tell where the sound is coming from) to draw attention is one of the most effective tools VR filmmakers have currently.
• Some creators have used arrows pointing back to the action when the viewer is looking in the wrong direction. [Really, I should say “wrong” direction, in quotes, because there really is no wrong direction for a viewer to look; it just might not be where the film maker prefers them to be looking at any given moment.]

“VR video is a marriage between gaming and film.” [I understand the sentiment of this statement, essentially that VR viewers will be able to take a more active role in the story or at least the storytelling experience itself, but really, gaming itself is a marriage between gaming and film, and has been for many years.]

On Funding a VR Film:

• Good VR video costs a lot of money (this specifically refers to VR, not 360 video.)
• We are still in the research and development phase of this medium/industry,
• it’s hard to find people (companies) who are willing to pay for R&D. They want to pay for finished products. So funding is difficult for VR projects.

Directing 360 film:

• There is a difficulty for the director of a 360 film, because they can’t be near the scene/action without also being in the shot. It would be like a stage/theater director on the stage with the actors during a performance.
• One panelist recounted a story of a director who literally crouched under the tripod/360 camera to direct so that he wouldn’t be in the shot.
• [traditionally the camera and tripod are edited out of the final film in a process called “nadir patching” so that the camera doesn’t become part of the movie itself and distract the viewer from what is important. In this case, the director would be removed during nadir patching]

In the future:

• viewers will bounce back and forth between VR and AT in the same film.
• Since the software knows where the viewer is looking, it can potentially bring the story to the viewer instead of waiting for the viewer to look in the right location. (For example, a character can walk over to where the viewer is instead of needing the viewer to look in a certain spot to understand the story you are trying to tell.)
• “Unity will be the new Adobe,” in terms of baseline skills needed to create VR experiences.

Unasked Questions:

Because the panel was only an hour long, lots of things weren’t discussed, and unfortunately the moderator didn’t leave time for questions at the end. I’m listing here some of the thoughts and questions I’d assembled but never got a chance to ask the panel:

• How do you effectively translate filmmaking conventions from 2D film into 360/VR (do the same type of transitions work? Where do you position subtitles when you don’t know where the viewer is looking? How do you camera movement—both panning and forward movement, given that it can have a nauseating effect on the viewer if not done carefully, etc.)
• VR films potentially have much more “replay” value because the viewer can literally watch a different film every time. This can work against the storyteller (see above re: narrative magnetism) but could also be used to tell a much more complex story, or multiple stories requiring multiple viewings to see them all or to capture all the details and clues hidden in the scene.
• What genres work best in 360/VR? (I can tell you from experience that horror can be INCREDIBLY effective in 360, as can documentaries.)
• What technological and social advancements/changes (or lack thereof) are holding VR/360 video back from becoming better and more popular?
• What have you seen in VR video that is generally NOT being done well?
• How will movie theaters respond to a medium that (currently) excludes them by taking the film viewing experiences to people in their homes rather than in movie theaters? What can theaters add to the experience that won’t be available in homes? (For example, you previously had to go to a movie theater if you wanted to see 3D films, but now they can be watched at home.)
• How can AR enhance live-action, stage-based performances?
• With the lack of ability to directly control a viewer’s focus, can we have VR film experiences where film makers are really just “guiding” a viewer rather than directly telling a specific story from a specific point of view? What other decisions can we turn over to viewers other than just where they’re looking? Can a viewer become an active character in a VR film (the answer to this is already yes; they’re called video games)