Fantastic voyages: how VR became a reality

from our May 2016 issue

Credit: Sander Koning / Getty Images

“Last night I was in the kingdom of the shadows,” Soviet novelist Maxim Gorky wrote in reaction to the Lumière brothers’ first films at the Nizhny Novgorod Fair in July 1896. “If only you knew how strange it is to be there. Everything vanishes and a train appears on the screen. It speeds straight at you – watch out! It seems as though it will plunge into the darkness in which you sit, turning you into a ripped sack of lacerated flesh and splintered bones… But this, too, is but a train of shadows.”

Gorky’s astonishment at the Lumières’ The Arrival of a Train at Ciotat (1896) conveys the exhilarating sensations of early cinema. The new medium generated wonder, unease, terror, excitement, anxiety. Moving objects disappearing beyond the edge of the frame were inexplicable, noteworthy, a stimulating and thrilling marvel.

We are now entering a similar era of wonder as millions experience the medium of virtual reality for the first time. But what is VR? Put simply it’s a set of goggles you wear or hold over your eyes that allows you to view a 360-degree filmic environment. Up, down, behind you, there are no screens – the viewer chooses where to look. Add a pair of headphones and the feeling of immersion is intensified, so much so that it is advisable to be seated or standing with someone you trust.

Done well, VR can transport the wearer away from reality in a way no other medium can. ‘True VR’ is stereoscopic, meaning viewers get a different image in each eye, adding the illusion of depth. Headsets made for VR determine the quality of the experience, but it is possible to watch VR films simply using a smartphone and a piece of foldable cardboard, such as Google Cardboard.

Google Cardboard

Lingering in the minds of sceptics are the more muted incarnations of early VR technology of the 1980s and 90s, which caused many to dismiss the phenomenon as solely a gaming medium. After all, VR was born from Doom, John Carmack’s influential first-person shooter.

Yet in 2015, investment in VR was rampant in Los Angeles, with this year expected to be decisive as consumer-ready headsets such as Oculus Rift, Samsung Gear and HTC Vive reach the market. Google Cardboard – the £12 version of the more sophisticated headgear – is readily available. In November, the New York Times delivered 1.2 million cardboard viewers to its subscribers, with a range of documentaries ready to watch on its NYTVR app. There is a distinct air of a gold rush as film industry players work out whether, and how, to adapt, and how to make it pay beyond the most obvious genres: games, extreme sports and pornography.

Watching a well-produced VR film is overwhelming and best enjoyed in short bursts. Headsets completely envelop the field of vision and the ability to look in any direction can surprise and delight. With the right film, the first encounter is wholly unfamiliar and astonishing, although vertigo and motion sickness – the bête noire of the industry – are common side effects and immersed viewers can become so disoriented they can walk into walls.

Yet the effect can also be phenomenally beautiful. In one popular silent YouTube video, the viewer is perched on top of a mountain in the Arctic archipelago of Svalbard to witness a perfect solar eclipse. The natural world – alongside horror, live music and deep space – is a regular theme of these early films. We are invited to take space walks, visit African savannahs or swim with dolphins.

“VR has incredible potential. It takes you to places you could never have dreamed existed,” says Sir David Attenborough, who has narrated First Life, a 15-minute VR film exploring the dawn of life in the earth’s ancient oceans, made in collaboration with the Natural History Museum and production company Atlantic Pictures.

Early-cinema historian Stephen Bottomore has argued that “when a new medium is introduced to an ‘inexperienced viewer’, his senses need to adjust until, through some kind of learning process, they build up enough experience to reintegrate themselves into a new mode of operation”. This process is now happening all over again.

Chris Milk, a veteran music promo director, is leading an ambitious charge to persuade a sceptical establishment of the medium’s cinematic value. In a TED Talk in March 2015, he declared he is “done with rectangles”. He went on to co-direct a series of films with the United Nations senior adviser and filmmaker Gabo Arora, the first of which is a powerful documentary called Clouds over Sidra.

Sidra, a 12-year-old Syrian living in the Zaatari refugee camp in Jordan, guides the viewer through the features of her semi-permanent existence in the tent city – an austere living room, a dirt football pitch, a bakery so vivid it’s hard to believe you aren’t there breathing the aroma of fresh flatbread. Dignified and humble, Sidra shares a meal with the viewer as a voiceover details her hopes and dreams. The film won an Interactive Award at Sheffield Doc/Fest 2015 and has been seen by international policy decision-makers the world over. One by one, they were bowled over by the experience. Fittingly Milk has also evoked the Lumières’ ‘train effect’ knowingly in Evolution of Verse, a 360-degree CGI-rendered short that screened at the 2015 Sundance Film Festival.

“Behind the bells and whistles, we’re creating stories,” says Milk. “What’s special and different about Clouds over Sidra is the level of immersion the technology provides. Clouds is like stepping into another world, walking a mile in someone else’s shoes. Once we found Sidra, her life informed how we captured the environment. We agreed that virtually placing people within this camp might greatly impact how the world sees Syrian refugees. The goal from the outset was to create a sharper and more intimate form of empathy.”

One of the many affecting moments in Clouds over Sidra is when a group of Syrian refugee children surround the viewer, ghost-like, full of weary smiles and innocent curiosity, albeit pixelated. It’s a familiar scene for anyone who has done field photography and has proved capable of reducing viewers to tears.

“So much of Clouds appears startlingly simple and uncurated,” says Arora. “We wanted the viewer to feel as if they were actually there alongside Sidra. Our intention was to make a documentary, but we wanted to do something more in the spirit of Werner Herzog’s documentaries, which work on the notion of ‘ecstatic truth’; that facts are static while the truth needs to be revealed by artistry.”

VR not only represents a potentially major shift in exhibition (from viewings in dark screening rooms, to solo and synchronised headset displays), but also in how cameras capture their subjects, as Arora explains. “We started filming in December 2014, after other celebrities and filmmakers had already visited and filmed in the camp. Most people I approached were just tired of being filmed. They felt their sufferings were being exploited, that they were telling their story repeatedly yet their situation remained unchanged. But they reacted differently when I showed them the virtual reality camera and explained our purpose. There was immediate enthusiasm and intrigue. The camera system is less invasive and can be left for minutes at a time alone, which leads people to forget it’s there, allowing them to have a more natural experience with filmmaking.”

Aspiring VR filmmakers are faced with a bewildering array of competing camera equipment, content platforms and viewing hardware, with new products launched all the time. “We use whatever’s necessary for the story we’re trying to tell,” says Milk. “A lot of what we use, we build in-house. How we rig, how we lens, how we ingest the footage, what software we use, what software we create – all depends on the needs of the project. We’ve gone out with a rig of eight GoPros before, which is more of a run-and-gun type set-up. We’re currently shooting in Prague with a rig that didn’t exist two weeks ago.”

Spike Jonze last year produced Millions March, a blood-pumping 360-degree documentary created with Milk for VICE News about New York’s ‘Day of Anger’ protest demanding justice for unarmed black men killed by police. It’s an electrifying use of the medium. New York’s vertical geography combined with the sheer strength of feeling among protesters makes for a jolting experience. What is remarkable is how the film captures an intangible sense of danger and personal suffering – from seeing the whites of the eyes of impassioned demonstrators to hearing drums of angry protest behind you. Despite the guarantee of safety, the mind is alerted to the ever-present threat of police brutality. It’s a deeply moving and immersive experience.

Other documentary pioneers include American journalist Nonny de la Peña and Oscar Raby, from Chile, both exploring the possibilities of interactive documentary via 3D-modelled environments layered with real audio, locations and characters. De la Peña is known for another examination of the Syrian conflict, Project Syria, as well as Hunger in Los Angeles – a critical study of America’s neglected poor – and Kiya, the story of two women who attempt to rescue their sister from a violent ex-boyfriend. Touted by some as the ‘Godmother of VR’ for her extensive work in virtual reality and augmented reality, de la Peña has developed a highly charged, political visual language.

“When people put on our goggles, they are fully transported to another world,” she says. “There is a solid academic, neuro-scientific grounding for the notion of ‘presence’, of your mind tricking your body into believing that you’re somewhere else. That presence is the defining element of our work, and of all good VR. In Hunger in Los Angeles, for example, a piece about waiting in line at a food bank in downtown Los Angeles, we’ve seen countless audience members get to their knees to try to help someone who has had a seizure and collapsed.”

Raby’s film Assent is a more personal affair, a challenging autobiographical piece which looks like a traditional video game. The spectator takes the role of Raby’s father, guided by the filmmaker through the former’s very worst memory – a day soon after the military coup in Chile in 1973 when he witnessed the execution of a group of prisoners as an officer in the military.

“VR is of course a tool for all sorts of story makers, but it is noticeable the enthusiasm with which it has been embraced in the activist space,” says Raby. “At least in its early days it has been quite a disruptive medium – one where you can gain attention just by making a piece of work. That won’t last forever, but it is important now, when the competition for eyeballs and attention is so vast.” Yet he does not entirely agree with Milk’s soundbite-friendly description of VR as the “ultimate empathy machine”. “VR as the ultimate empathy machine obscures what theatre has excelled at for ages,” he says.

This analogy with the theatre is one de la Peña also recognises: “VR is a whole new kind of spatial storytelling that has more in common with interactive, experiential theatre than traditional filmmaking,” she says. “The traditional notion of cutting doesn’t really apply in VR, since you can’t control your user’s point of view.”

Raby, whose Melbourne-based production studio VRTOV is a nod to Soviet documentary pioneer Dziga Vertov, correlates the popularity of immersive events practised by the likes of the Punchdrunk theatre company to our collective desire for more complex narrative experiences in the digital age.

Heeding this call perhaps, the New York Times produced The Displaced – a VR documentary about three children who have fled their homes. The stories from South Sudan, Lebanon and the Ukraine revive an empathy that has seemingly been in short supply over years of dehumanising media treatment. One astonishing scene shows an aid airdrop from the point of view of a beneficiary on the ground. It’s among the best scenes to date in VR documentary.

At the light entertainment end of production, Oculus Story Studio – the creative arm of the VR hardware firm Oculus Rift – produced Henry, directed by former Pixar animator Ramiro López Dau, whose credits include Brave (2012) and Monsters University (2013). The film, narrated by Elijah Wood, recounts the story of a cartoon hedgehog who has lost his friends. The relationship between Pixar and Oculus will be interesting to watch.

Mesmerising abstract pieces are emerging all the time, as designers, developers and artists begin to experiment and collaborate. One such piece, Tana Pura, is a psychedelic swirling lightshow – created by New York-based Mike Tucker and set to an original composition by Radiohead’s Jonny Greenwood – which explores the moments following death, and the transition of the soul into the afterlife.

Raby anticipates more work of this kind: “We’ll see a lot more variety in the narrative VR work that’s being created over the next 12 months than we have in the past couple of years.”