'Star Wars' and the coming holographic cinema revolution

"AR is going to hit us like a big bang," says ILMxLab creative director John Gaeta when I ask him whether augmented reality, as that holographic technology is known, has been undervalued by the public and press. "We're just trying to point out right from the beginning that there will be a form of AR that will be as hi-fidelity as the cinema that you see at some point. I can't say what year that'll be. But at some point, we'll have intimate holo-experiences with performance and things like that."

And what better way to usher in that next shift in entertainment consumption than on the back of the Star Wars franchise. Gaeta, along with a handful of others from the immersive entertainment-focused ILMxLab team -- which combines talents from Lucasfilm; Industrial Light and Magic, the visual effects studio behind The Martian and Avengers: Age of Ultron; and audio post-production house Skywalker Sound -- is at the Sundance Film Festival's experimental New Frontier exhibit showing off 'Holo-Cinema.' It's an AR installation that places viewers within the sandy world of Episode VII's Jakku.

Unlike Microsoft's HoloLens, an AR headset that combines a Windows 10 computer and projection system to create its mixed-reality world, Holo-Cinema relies on a pair of lightweight, sensor-laden active shutter glasses. These glasses, in combination with a motion-capture system used to track positioning, let viewers watch the inhabitants of Jakku -- in this case, the droids BB-8 and C-3PO -- pop out from the projections on surrounding screens and move around in 3D space as if they were physically present. It's an impressive technical feat that, quite literally, has to be seen to be believed.

Holo-Cinema is merely a preview of the kinds of immersive entertainment Disney-owned Lucasfilm, ILMxLab's parent company, believes will become commonplace in the very near future. In fact, Gaeta and Rob Bredow, the lab's head of new media, say their team's already at work on consumer-facing projects that could accompany the studio's next blockbuster films. So when Episode VIII of Star Wars hits theaters, according to Gaeta, there's a very good chance it'll arrive with a Holo-Cinema offshoot.

"What we're experimenting with and planning around is to actually give you more of what could be happening in the convergence of that story," says Gaeta of AR storytelling's promise. "What happened just before. What happens just after. What's happening around the boundaries. ... Without getting too into 'Choose your own adventure' and going into crazy multi-branching, what we're trying to do is show you that, in this moment in time, in this space that you happen to be standing in, there are things still going on around the periphery of the story for you to find. And if you wanna go free, you can do that, too."

ILMxLab demos its virtual reality production tool, VScout, at Oculus Connect 2 in Los Angeles.

This past summer, at Oculus VR's Connect 2 developer conference in Los Angeles, ILMxLab took the wraps off another immersive production tool, dubbed VScout, that Bredow says was developed in parallel with Holo-Cinema. It's the virtual reality companion to what ILMxLab's doing with AR. And, like Holo-Cinema, it can be used either as a standalone entertainment experience, allowing viewers to explore and follow separate storylines in the world, or as a tool for pre-production.

"We've used it for some of the other filmmakers," says Bredow of the platform's filmmaking potential. "Gareth Edwards has leveraged it for some of the work that he's done on Rogue One."

Using both VScout and Holo-Cinema, Bredows says Edwards was able to make certain pre-production decisions for the next Star Wars film simply by being able to visualize "how big doors should be" on set, for example. The key advantage of which was that the crew didn't actually have to "build the things at full scale," thus sparing the production's budget.

To create the scene taking place on Jakku at New Frontier, artists and engineers at ILMxLab used "real elements" from The Force Awakens, blending both photographed and computer-generated assets. That environment was then combined with a mocap performance of C-3PO and BB-8. The result is a pre-recorded scene that viewers can participate in.

Actors perform in a motion-capture studio to create a scene from Star Wars.

"When we do this for production, we'll have the person acting for C-3PO and the person driving the droid [BB-8] standing just off set, acting in real time. So you'll get the interaction of being immersed in the environment and the interaction with a real person. So it's basically like being inside of a movie," says Bredow.

As for why ILMxLab chose Sundance to show off the tech, Gaeta says it's really about community outreach and education. He refers to Holo-Cinema as a "portal" that filmmakers can use to transport audiences into the worlds they're seeing onscreen. It can also be used, Bredow suggests, as a means of repurposing "things that might've ended up on the cutting-room floor."

Using a tablet interface, ILMxLab can control elements of the projected Holo-Cinema scene.

"This is our way of simulating augmented reality to come," says Gaeta. "So when glasses come online and are enabled to handle the type of fidelity that we're looking for, we're trying to imagine how we would bring cinematic emotional moments into your spaces, your world. This is just kind of like ... it's a little bit of a format teaser."

Image credits: Lucasfilm (mocap studio)