Microsoft Mixed Reality Capture Studios create holograms to educate and entertain

In sci-fi movies and TV, holograms have long embodied a bevy of characters, including Princess Leia and The Doctor on “Star Trek: Voyager.” 

A big part of the appeal of these 3D representations is that they occupy the same space we do and that we can interact with them in a way we can’t with 2D videos and photos.

Beyond the fictional world, holograms are starting to help humans in real life. They’re translators, teachers and preservers of memories. 

“Our children are the first generation to start to expect spatial – or 3D, volumetric – content,” says Jason Waskey, a principal creative director for Microsoft and creative director for the company’s Mixed Reality Capture Studio in San Francisco, where the company creates holograms for various uses.

“When we think about why people would want a hologram, it’s the same reason you want to see your videos in sound and in color. It’s because it reflects the world the way we move in the world. Having someone life-sized in the same room with you, remembering them the way that they were or are – that’s really powerful. As we continue to recreate the world around us with as much fidelity as possible, adding depth is the next stage. Anywhere you expect to see video, you should also expect a truly volumetric experience.”

Microsoft’s studios are leading the way toward this future. You can see the technology at work in The New York Times through its “Ashley Graham: Unfiltered” experience, the most recent British Open and Madonna’s performance at the 2019 Billboard Music Awards with holographic versions of her past incarnations.

“Our job is to make sure creators can do what they want to do,” Waskey says. “When we look at what Microsoft is doing in this space and the future of human computer interaction, that experience will be spatial. It allows us to make eye contact and be more present with others.”

The studios, which operate like professional stage or video production sets, started in Redmond, Washington, in 2010, but are now headquartered in San Francisco with higher-resolution cameras and the ability to record longer makes for more experiences.

They are setting up a a state-of-the-art volumetric stage in Redmond to create souvenir holograms for corporate visitors at Microsoft’s new Industry Experience Center (IEC), which recently opened to reveal a 23,000-square-foot facility that showcases nearly 100 real-world examples of customers and partners that are innovating their businesses and disrupting markets with Microsoft technologies.

There are also partner studios in London and Los Angeles, and a pop-up location in New York.

The main studio in San Francisco provides hair, make-up and 106 infrared and RGB cameras to record 360 degrees of a person. Clients dress and move the way they normally would, such as children laughing and playing.

When Microsoft exec Julia White came in to create a hologram version of herself to use on stage at a recent conference, she spoke as she would to an audience, pivoting back and forth, using her hands to gesture naturally. In a memorable moment at the conference, her hologram appeared on stage beside her and continued her keynote speech – in Japanese.

“We move the camera for you, versus you are the camera,” Waskey says. “It’s a very similar process to creating a video, but in the end, you’re getting a hologram.”

The process brings in several cross-company initiatives and strengths. Most recently, they’ve added Azure to the mix to boost the work.

“One of the pieces of tech that really proves to be important is the way we compress and package the results,” says Steve Sullivan, general manager of the Mixed Reality Capture Studio. “Some other providers struggle with heavy files that are hard to get to consumers. But we can crunch it down to about the size of a video stream from Netflix. So basically, anywhere you can stream Netflix, you can stream our holograms. And that puts us onto phones, web, HoloLens and – you name it. We’ve built players for every major device and platform, so if you’re a creator or performer wanting to reach an audience, we can get you there.”

Azure is a new element, Sullivan says, that gives the company and its partners the advantage of better scale and broader reach. It allows them to process much more data, as they’re able to rely on the cloud to process more content, more flexibly, than an on-site render farm. This also allows them to provide services to other capture providers who may already have their own infrastructure.

“Even those who capture with different technology can process their data in our Azure pipeline to leverage our compression and playback ecosystem, giving all creators more options,” Sullivan says.

And now, they’re taking this technology on the road with two mobile stages, for more informal sessions. New as of this year, these packaged configurations are easier to deploy and able to take advantage of the cloud, versus huge on-premise computing. The mobile capture studios can have fewer cameras (64 instead of 106), are lighter in weight and built for portability.

“This gives us the opportunity to go where the action is,” says Waskey, pointing to events like the British Open, in which Dimension demonstrated golfers’ swings.

“When we look at sports as a particular scenario, we’ve generally seen it at a very flat angle. Golf is truly three-dimensional, there’s an arc and plane that gets described, there’s a full range of motion to hit the ball. Every golfer is subtly different in how they approach that task. One of our partners, Dimension, used a mobile stage and the power of Azure to process very fast huge amounts of data to show golfers’ angles that are normally ‘too dangerous to film.’”

It’s as if you were standing right in front of him. You can get up close and see exactly where their hands are placed, exactly where the driver smacks the ball. And that’s not safe to do that with standard, conventional 2D photography.

There are also those who license Microsoft’s technology stack, including Dimension in London and Metastage in Los Angeles. Some have their own stages. Clients come to them to shoot the content to produce, and Microsoft provides tools and plug-ins that unblocks them and empowers them to do more. Microsoft also provides services for clients at these studios, as a fee for service.

The expansion of these mobile stages is part of the road to a future where memories, educational opportunities and entertainment come into our world fully, rather than viewing them passively.

Sullivan sees a similar not-too far-off world where annual school holograms, for instance, replace those first day of school pics you put on the fridge. As captures become more pervasive, he imagines hologram recordings of family vacations and other milestones becoming the norm.

He says the “the end-to-end technology is here today,” but access, pricing and an effective consumer ecosystem will take time to take hold, culturally.

“Even in small doses, it really changes your experience and perceptions,” Sullivan says. “Having a hologram of somebody is a more authentic, complete representation of them than photos or videos.”