Thu 23rd Oct 2014, by Rory Fellowes | Production
CGSociety talks to to Jeff Wagner of SideFX And Alex McDowell (Man of Steel, Minority Report) of the 5D Institute, USC about new ways to tell stories.
New Technology, New World
Such huge technological leaps-forward have been made in the last twenty years or so, and especially in the last ten, that they make the next ten a thrilling cornucopia of possibilities. What will we be watching, reading, hearing ten or twenty years from now?
In my previous articles in this series I discussed how games and film are converging in the fidelity of the images, and the scale of the data and the sophistication of the delivery systems that are coming onto the market and altering the nature of media.
Delivery has been perhaps the major game changer. The new technology is proving to be amazingly good at keeping up with the demands of the users. Games have led the charge in providing home delivery of HD images with realtime gameplay and rendering, while film has continued to hone the bleeding edge of image quality. But now they are almost running together We have seen the development of Blue Ray and internet streaming, MMPOG, 4K images on plasma and LED screens. We have home entertainment delivery and reception systems that are becoming familiar in the average living room, ranging in scale from a small widescreen TV streaming satellite TV, up to, I imagine (and if you have the money), a full scale digital cinema in your house, like a Hollywood movie star. Sony have already prototyped a 4K TV screen, and I recently heard of an 8K screen in development in another company.
Or perhaps we will see a version of an idea that was going around in the 1990s, what was called then, “intelligent paint”, a paint-like liquid solution of receptors, which could convert a whole wall, or just sections of it, into a screen or lots of screens of any size, by what I assume would be some sort of system of polarizing the receptor cells, so that media can be streamed to them. Imagine the various windows as resizable and moveable as the boxes and icons on your computer screen, controlled by an all-purpose games-type console.
I say a console because of a personal experience some 8 years ago, that started off this line of thinking for me. I was at a party where there was a room full of adults and one young child, a boy. Soon bored with being introduced to and patronised by the adults, he got off into a corner on his own and proceed to play a sort of “air” console, his thumbs twiddling as he pretended he was “playing” us adults. It occurred to me then that by the time he reached 21 (about 6 years from now), he will expect to be able to control all of his media through some sort of console, whether to switch on the TV, or choose a movie, or play an MMPOG, or close the curtains in his bedroom, or whatever else is on offer.
Home systems are going to be what the media industry mainly delivers to, they will be the principal interface with the consumer from now on. Now add that kind of personal home access to high speed, high definition streamed data, and give it to a 21st Century multi-tasking teenager; then recall what Alex McDowell said to me – “If you can imagine it, you can make it” – and who knows where this whole industry is going in the next decade or three.
Jeff Wagner & Alex McDowell
Proceduralism
I would be surprised if the word “Proceduralism” hasn’t been used and published before now, but maybe it is a neologism, in which case, bear with me, a new word seems to me what is needed to describe a new way of thinking. I’m just being cautious because Jeff Wagner, Senior Technology Consultant at Side Effects Software Inc., said to me, “Proceduralism is not a word. If you look it up in the dictionary, you won’t find it. It doesn’t exist.”
I don’t care, I’m going to use it. My idea here is that the original concept of procedural calculation as applied to mathematical description has become a way of thinking that goes beyond the bounds of mathematics and has become, at least within the media industries, an influence on many forms of creative and descriptive activity. A philosophical concept, if you like. I see the idea of Proceduralism describing the way of thinking that forms the core of Disruptive Technology, the subject I first heard discussed at FMX 2014 last May in Stuttgart, Germany, at the annual FMX Conference and which I have been discussing since then in the series of articles of which this is the latest,
It seems that what is being disrupted is the entire methodology of film and games production and conceptualisation, of drama and news and documentary, of everything you can think of in terms of moving image delivery, and most likely some innovations of which we have not yet dreamed, Horatio. Everything is changing, hybridising, into new forms.
You may have noticed in that first quotation, Jeff Wagner is the kind of guy who says things three times, in three different ways, to drive the point home. I never met a man who wanted so much to be understood as Jeff. But of course, he is up against it with most of us, certainly with me. If someone had told me Jeff was raised in a secret government facility beneath the Nevada Desert to make him into a hybrid human computer, I’d be tempted to believe them.
“I hate to tell you, but that’s the way my brain is wired,” he said to me. “I learnt the forebear of Houdini, [a 3D software called] Prisms, literally in two weeks. I mastered it from zero to knowing almost every single intrinsic part of that software in two weeks. For me it was how my brain gets used.”
Houdini
Jeff and the team at SideFX have developed the software Houdini. Houdini has been part of the landscape since the final years of the last century. When I was learning Poweranimator on a Silicon Graphics Indigo machine, Houdini looked like a computer science software programme to me. I couldn’t make head or tail of it Well, that’s because it is rooted absolutely in the world of computer thinking. I was coming to it after twenty years as an animator in the analogue world of stop motion and optical cameras, but I suspect current generations are more than ready for Jeff and his co-developers and thinkers, and it is my prediction that anyone wanting a job in the moving image production industry 15 years from now had better learn how to work with Houdini. Whether it is that software or some other like it, the procedures, the techniques, the thinking, will be similar.
“When we were doing the geometry rewrite,” Jeff said, “[designing] the actual architecture that runs the geometry processing nodes, what we did is, we created an environment that allows you to run any of these programmes in a thread safe environment. In other words, you can write these nodes utilising threads. [Houdini] provides an architecture that allows you to do that. The funny thing is, that when we first shipped Houdini 12, with the geometry re-write fully in place, the majority of our own nodes were not threaded, although the architecture that can invoke these nodes was thread-safe. So hopefully that simplifies that.”
Except, what is a thread? “If you’re running an application, that application can basically fork its process, it can sub-divide its process into separate sub-routines [called threads], running in isolation from any other thread. In other words, it allows you to parallelize your operation. Each parallelized operation can be run on a thread.
“And a thread isn’t tied to a CPU. You can have, let’s say eight cores, but you could be running two thousand threads, and then what you’re doing is you’re allowing the operating system to optimise how those threads get executed. Same thing with GPUs. The Nvidia GPU in itself has a mast of parallelised cores. You could get an Nvidia GPU that has 10, 24, 256Mb cores, and you can run 2,256 threads if that’s what you want to do, [all running at the same time], or you could run 100,000 threads and let the GPU sort out how it’s going to execute those threads.”
What Jeff is describing here is a highly complex Interrupt procedure, as each thread takes its turn to pass on the data it has calculated, and then stands down to let other threads go, before being allowed to go again. All this being measured in nano-seconds, you understand.
“It’s not for the faint of heart. Writing threaded applications you can get into what are called deadlock situations, whereby one thread is dependent on another thread and the other thread never tells the first thread it can go.
Fortunately for the operators, he tells me, “Handling and managing threads [is an issue] you don’t have to worry about when you’re writing a geometry node inside of Houdini. You just basically have to worry about how you manage your memory and how you call these threads. You don’t have to worry about how Houdini deals with this.”
What it means to the person using the programme is that they are obliged to begin with some kind of fairly detailed abstract conception, a broad canvas of inception. They have to fully understand what they mean to produce in purely descriptive terms. Something of a Platonic exercise if you think about it, a dialectic way of thinking through an idea. A computer has no idea what you mean by the word “Waterfall”, but if you describe it well enough, you can tell a computer what the river looks like, how it comes to look like that, and in all its permutations, and from that information it can create an image of a waterfall that is pretty damn convincing these days, and only getting better.
[embedded content]
I have quoted Jeff at length here because I believe what he is describing is the core of what I mean when I say we are beginning to think in a new way. This assembly of parallel calculations leading to an end result is not unlike the way all of our brains are “wired”, at least, in terms of the way they function, but with the advantage to us (for now) that the human brain can run, thread safe, millions of simultaneous thoughts/ neural messaging activities. The computer hardware has yet to be built with an equivalent architecture, but I guess it will come.
The 5D Institute
Alex McDowell, at the USC 5D Institute is trying to take this new way of thinking, this multi-dimensional algorithmic way of plotting what you are trying to realise, and apply it to the processes of creativity, bringing all the contributing departments into the moment of, as he calls it, inception.
He is looking for new ways to create narratives using procedural techniques of analysis and collaboration, and most importantly, expanding the human base for the creative process, involving a wider range of people.
Interestingly, I have recently come across two theatre groups in the UK. One is a travelling show that goes from town to town creating a play out of the local history and staged all over town with the local people taking part alongside the actors. The other takes over buildings and makes stories out of the history of that building and also involving the audience so that they influence the flow of the play, such that it is different in every performance. As well as the effect of procedural thinking there is a social awareness in all this, a sense of community and the desire to all act together.
Until recently Alex was a Hollywood Production Designer of great distinction, justly famous for many Hollywood blockbusters and independent productions. For myself, he is to be admired particularly for his work on Minority Report.
Minority Report was an unusual film in many ways, as the Writer Scott Frank and Production Designer Alex McDowell began working on the same day. Without any script in place, the creative team built the world first by defining a global narrative space – Washington DC in 2050 – within which the linear story could be carved out as the script and the world developed in lock-step. Image courtesy of Twentieth Century Fox, illustration by James Clyne.
In Steven Spielberg’s Minority Report, Alex McDowell and his team imagined a Washington DC in around 2050 which had undergone rapid vertical urban development due to the unique nature and tight radius of influence of the Precogs and their ability to predict murder and violent crime. As a result, the city has advanced socially and technologically in ways that not only predict the future but engender new ways of negotiating urban life: transportation, architecture, retail, communication, crime, and ecology. Image courtesy Twentieth Century Fox.
Alex now fills his time under three titles, Professor of Practice at the USC School of Cinematic Arts, Media Arts and Practice division; Director of the USC World Building Media Lab; Director at USC’s 5D Institute; and Creative Director at the 5D Global Studio. To be honest, I was amazed that he found time to talk to me, but delighted that he did. We first discussed his ideas at FMX 2014 and followed up that conversation later.
At the World Building Lab, Alex and his students and fellow tutors are working with a variety of softwares, the Unity game platform, Oculus Rift, and Metaio, to work with Augmented Reality, as well as Houdini, to build procedurally developed environments, based on what Alex described to me as “the DNA of cities and landscape”, and to evolve their narratives from these worlds.
Built from the DNA of real world cities Los Angeles and Rio, the Rilao Project is a vast world initiated and developed by the USC World Building Media Lab. A fictional city that is meant to function as a narrative laboratory for the future, Rilao is situated on a mountainous archipelago that it has extended through terraforming processes, Rilao is a multicultural, technologically advanced city that is at the forefront of developments in materials science, energy use, urban planning, expressive personal robotics and procedural architecture. While Rilao has the conditions and elements of a real city, its fictional status allows us to imagine possibilities that aren’t governed by already existing parameters. It is a space that dares cities, institutions and industries more broadly, to dream.
Lao Chapters.png – Artists: Bryan Edelman, Eric Adrian Marshall, Chris Cain, Ascot Smith; Image c. World Building Media Lab
Camouflage.jpeg – Artist: Behnaz Farahi; Image c. World Building Media Lab
Why does Rome look so different to Paris, or Berlin so different to London? In their latest creative experiment, The Rilao Project, the World Building Lab have created a city called Rilao, that stands on its own island. The city combines the so-called DNAs of Rio de Janeiro and Los Angeles. Out of this process of building and inhabiting a new city, the tutors and students are finding narratives and other possible creative strands to develop into various forms. As well as several drama concepts coming out of this, Alex told me one student has written a recipe book of the local diet. The possibilities, seemingly, are endless.
The 5D Institute is essentially a research unit organised by the University of Southern California. It is a discussion and outreach space.
“It’s primarily about building a network for discussion and enquiry. We look into the future of narrative. How should we be thinking about the technology? How can we tell new stories and what is the audience for those stories?” This in turn, brings in the delivery systems, how they affect the narrative methodology of the storytelling. “What are the delivery systems? Do the delivery systems change? Do the delivery systems require a retraining of the audience, to better exploit the narrative content?
“The Science Of Fiction event is about, how do we tell new stories and how do we imagine new worlds. And how do those new worlds inform our present reality. This is the second event. Last year we looked at LA, this year we’re looking at an fictional city called Rilao. That’s becoming super-interesting. We’ve already had 60 or 80 students working in that space, we’ve got about 60 projects that were developed in three or four months from this fictional world, and they’re wildly varied, but they’re all treating Rilao as this sandbox space where whatever you imagine you can make, but you’re making it in an informed way.
“We’re looking at a near horizon, we’re looking 10 years, 15 years out, but in a fictional way informed by real world conditions. So what happens if you try and imagine the future of, say, the cell phone or of architecture or of transportation? You get very liberated by the fact that it’s in a fictional framework, but what you learn from it is very applicable to the real world.”
Er, sometimes… Below is an image from Upside Down, an independent feature film, a French Quebec co-production, that imagines a magical realist love affair in two opposing gravities.
Mapping a world in two gravities required detailed prototyping and massive complexity of design throughout production, that was framed by world building the narrative.
In Floor Zero of Transworld Tower there is a constant interaction between actors in two gravities, where merely establishing a coherent eye-line required complex technological and design solutions.
Inception
“My interest is in inception, the front end of the creative process. Inception, ideation, prototyping and then making, the executing of that. It’s very much about the weave between the components of storytelling, how do you tell better stories? How should we rethink the tradition of storytelling? We’re looking at the way in which you make stories change the way you tell stories, which is very much about [the production] process and technology meeting the creative process, and how they inform each other.
“We’ve discovered to a great extent that the demands of the story actually change the technology, that we have a very fluid relationship now with the technology and capability component [of story creation]. We’re only constrained by imagination. There’s no physical, tangible, experiential constraints really remaining.”
That was when he uttered his deathless line: “If you can imagine it, you can make it.”
Basically, Alex’s assumption is that the technology develops to fulfil the dreams of the creative imaginations at work. He believes the new technology can contribute to the way in which we conceive narratives. Writing these articles, I see where what Alex is proposing very much resembles the way games create narratives.
Alex and his students and fellow tutors are trying to answer such questions as “How do we tell new stories and how do we imagine new worlds? And how do those new worlds inform our present reality?
“So if we’re liberated in the way we tell stories and if we don’t self-constrain,” Alex said, “then the demands we make on the technology from the story, [will be met]. So for instance, asking why can’t we tell this kind of story, why can’t we imagine this kind of world, forces the technology to adapt. So that’s a big piece of the research that we’re doing. [We want] to put pressure on the technology space by using imaginative processes to make demands on the technology.
“Of course, since what we are talking about is at bottom, the entertainment media (though I concede the new technology, the new thinking, is having its effect in many walks of life, if not the whole damn lot), we have to consider the audience. The audience has to be ready for these new ways of creating and presenting media entertainment, and quite as much of course, they are ready for it all.”
Alex is a man who forms his ideas by way of asking a succession of questions. I suspect he has answers for them all, but he does not choose to assert his thinking so much as to coax your thoughts on the subject (he must be a wonderful tutor to study with).
So he asks, “What is an audience? Who is the audience if you have a multi-layered, multi-platform, multi-media, multi-narrative space? How do you first train an audience to think differently or to access the stories differently? And how does the audience’s needs change the way we think about the story as well. What kind of new relationship can we form between a large creative unit and a very much larger audience?”
Assuming we answer that, I asked Alex how does any project begin, who starts the process?
“I think the initial story impetus [comes from] the author. The author could be multiple people. With Leviathan, we started with the NY Times bestselling YA novels, written by Scott Westerfeld.”
Leviathan is a steam punk fantasy about a flying whale, set around the time of World War I. It imagines a Whale Airship on a one week voyage from London to Moscow, as a contained eco-system, and laboratory. For the people at 5D Institute, it provides a world for a deep exploration of the future of narrative.
“We, as a group of about ten people, then conjured a narrative that built a backstory to that [series], that the original had never imagined, that went back twenty or thirty years and looked at the origin story of the flying whale. So we took the story impetus from [Scott Westerfeld], but we built an entirely new story in a collaborative way, and that story now has handed off to three generations of students, and that remains a very volatile narrative space where the original author is still distantly connected, but he’s not actually having any direct impact on the narrative space at the moment. We built a virtual world which can be experienced through Oculus Rift mostly. We built it in Maya, we explored it and developed it with a virtual camera called Photon, which is a Microsoft Beta system that we’ve been testing, and then we pushed it into Unity, and we built a procedural AI character system that runs through that, and then we’re actors, our characters etc., our creators,
“My agenda is to redirect this conversation. I’ve been involved for at least ten years now, with this very broad inquiry space. It’s nothing to do with 5D specifically, though 5D have taken on some of this, but this notion essentially, of what is virtual production, or, in a very technical way, where do we go with the new capabilities, and how do you think about changing the paradigm. I feel like I’ve been a lone voice in a collection of VFX Producers, VFX supervisors, pre-viz artists, cinematographers even, and conventional producers, who all are stuck in the idea that digital means post-production. So I’ve fought tooth and nail for the understanding that virtual production starts at the beginning.
Slicing through a horizontal view of the world and its connective logic. Image c. Alex McDowell 2011
“I think you will be shocked by this, but there was a very powerful contingent who were insisting that virtual production starts on the first day of shooting, saying this was about stage and post, and I was saying that’s madness. If you don’t know what you’re doing; if your conditions haven’t been completely prototyped from the very, very beginning of inception; if you haven’t considered the design space to be the story space and started looking at virtual production as the way those two things flow together, then you’re going to be flailing around and just wasting money on the stage. And it happens on production after production after production, that millions are lost, billions are lost, in shooting stuff unnecessarily, in bad planning for stagecraft, not thinking about the right tools for the right jobs.
“We see it with every new production. They start up and everyone has their own opinion, and the first thing they do is call the visual effects company and say, ‘What should we do?’ And the visual effects company says, ‘Well, these are the assets we need; this is the way we want you to deliver your content to us.’ They don’t think about planning, because planning involves so many things that are not visual effects. A big part of our job is in [finding ways] to save shots, how do we maximise our funds, our budget, so that it doesn’t go to the massive costs of post-production, to solve it in camera and so on. So obviously there is a conflict of interest when you ask the visual effects company to get involved in shot planning on stage.”
Future Hurdles, Exciting Challenges
The major difficulty that confronts all this high thinking and ambitious prediction for the future of media is the hardware requirement to make all the possibilities into realities. All forms of delivery over the internet rely on speed, overcoming the “lag” as data is streamed to the player or viewer.
But for now this is a practical consideration that Alex McDowell need not concern himself with as a theorist, but a consideration for the rest of us, the audience. Besides, I don’t think Alex’s ambitions lie in this particular direction. As far as I could tell, he is looking only to open up the inception process, to liberate it from our prejudices and expectations, to allow new ideas, new ways of thinking to enter the forum, but ultimately he is only looking at how we develop what must become in the end a fixed idea, a final draft script, a plan, a production, a project. He talks of finding 60 stories in a new world, not an infinity of possible stories, unrelated and uncontrolled, and all happening at the same time.
But then again, that sort of infinity of possibility is the concept behind Star Trek’s Holodeck, and in fact it is already being looked into:
[embedded content]
[embedded content]
Alex does not speak of the multiple stories generated in the World Building Lab all being available within a single encompassing production, be it a movie or a game or whatever new name is given to this form of multimedia entertainment. Nevertheless, the methodology that makers of narrative games use is reflected in the inception process that Alex envisages.
Anyone interested in all of this could do no better than to attend this year’s Science of Fiction conference on October 25th in LA, where it and many other related subjects will be discussed.
As Alex said, if you can imagine it, you can make it, and I suspect that the new kids on the block, the seven to eighteen year olds of today, are imagining delivery systems that would boggle your mind.
Production companies, next time you’re having a blue sky session, I suggest you head down to the local high school and ask the 8 to 15 year old students what they’re dreaming of……Then make it!
[embedded content]
Related Links
Enter the world of Rilao
The 5D Institute
The 5D Global Studio
Alex McDowel on World Building at USC School of Cinematic Arts (Video)
Alex McDowell on IMDB
Rory’s Coverage of the 5D Science Of Fiction 2014 Conference
Making Order out of Chaos
SideFX – Houdini
Kinetic Sports Rivals on Youtube
Kinetic Sports Rivals – Launch Trailer
Kinetic Sports Rivals – Island Tour
Discuss this Feature on the Forums
