The acclaimed film director, Stanley Kubrick, has created some of the most memorable moments in cinematic history. He created and produced masterpieces across a multitude of film genres including war, political satire, sci-fi and horror. Without context, “Heeere’s Johnny!” is able to invoke an image of a murderous Jack Nicholson peering through a door wielding an axe. Kubrick’s creativity has certainly impacted pop culture and at the times often dabble in the field of Artificial Intelligence will his most highly regarded film “2001 A Space Odyssey” produced in 1968. Although murderous in its intention, HAL 9000 shocked audiences with its sentient being. At that time, it was revolutionary to consider a computer to be capable of thought however with the introduction of virtual assistants such as Siri, Alexa and Google Assistant, it is a very common way modern people are interacting with their devices.
He peered behind the curtain and at that time it was considered pure science fiction however, it has to some extent become our reality. Kubrick mixed elements of science within the creative process to further storytelling, thus in homage, a project dubbed “Neural Kubrick” mixes elements of the creative process within their science to examine furthering storytelling. That is, using the latest in deep learning technology they set out to create an AI film crew to reinterpret and redirect Kubrick’s films. Three positions, namely the art director, film editor and director of photography, was replaced with three machine learning algorithms.
AI as Film Editor
The role of the film editor aims to sequence and edit the raw film to create a finished work of art. The editor’s job is highly regarded as they have control of the narrative. Badly edited films fail to connect with audiences and contribute to a disaster on the silver screen. Anirudhan Iyengar, the inventor of Neural Kubrick, used a popular image processing neural network, known as the convolutional neural network, that would classify visual similarities between a given scene and the scenes from various movies from a large dataset. Here, the idea is simple re-interpret a Stanley Kubrick scene. A data set of frames from 100 movies, was generated to be used to train the neural network. Given a movie clip, the model would output a series of images which were similar to the input. Below, is an example of this. The scene of the left comes from “2001 A Space Odyssey” and the scene on the right comes from “ A Star Wars: The Empire strikes back”.
The colour composition of the scenes are identical and in certain cases, it is able to match the actor’s similarity. The bowler hat that Alex adorns in “A Clockwork Orange” is matched to the iconic bowler hat Charlie Chaplin wears in his films.
Film editing can be described as an art form and the best, most powerful editing is hidden and should not be noticeable to the audience. William Goldenberg, Oscar-nominated film editor, mentioned in an interview: "Why am I seeing this? What is this about? What does this scene mean?" If you have questions like that, probably the editing isn't good.
And that is very true, the objective of editors is to create a harmonious marriage between film, sound, dialogue and in modern time, graphics. So although we commend the use of deep learning in film editing, we have to consider the context of a scene and how to tell a story.
AI as Director of Photography
The director of photography is responsible for making artistic and technical decisions related to the image. Also known as the cinematographer, their job looks to create and hone the visual aesthetic of the film. Wes Anderson is famous for using cinematography to create a very stylised piece of art across his films.
Iyengar and team used a recurrent neural network( RNN) to replace the role of the cinematographer. Using Kubrick’s films, the camera positions of a scene was extracted.
Using the Reality Capture and Untity software, images from the Kubrick films can be imported and a 3D model can be generated. The XYZ coordinates can be exported and used to train an RNN. The model will be able to generate new co-ordinates thus effectively re-shooting the scene. An example of this in action can be seen in the side by side comparison below, the image on the left was extracted from a Kubrick film and the right was the modelled scene.
The model was highly effective in this by creating zooming in effect for the above scene.
AI as Art Director
Possibly, one of the most important aspects of film making. The art director facilitates the production designer's creative vision for all the locations and sets that eventually give the film its unique visual identity. They are also responsible to create colour stories across the film to make a cohesive experience. A Generative Adversarial Network was trained on frames of Kubrick’s films. The data set was divided into three datasets based on the shot length. The model would then re-imagine the composition of the scenes and generates new images. I can be seen from the below image, more information is needed.
The team has made massive strides in the field of creative AI. They have taken existing deep learning techniques applied in the field of film. Some successful and some not so much. It can be seen that film making is hard and not formulaic. Every year new groundbreaking advancements are made to export audiences to new weird and wonderful worlds. However, humans don’t want to see the same thing on screen despite the noble efforts of the Fast and Furious franchise.
But this brings us to the point of algorithmic collaboration of industries, and should we actually continue. My answer would be an astounding “YES!”. John Lassetter, director of the beloved classic Toy Story (1995), said in an interview for the behind-the-scenes making of the film, “One of the things that make Toy Story so unique is the collaboration between traditionally trained artists and the animators and these amazing computer geniuses”. Technology is ever-evolving and our medium for storytelling is changing. Before 1995, computer-generated animation would have never been imaged, now we are sitting in a position where some of the greeted stories every told was through CGI. We need to be constantly breaking ground and not simply generate AI for the sake of knowledge but collaborate where needed. So to all budding filmmakers and computer scientists, I say to you, “To Infinity and beyond.”
Authored by:
Bryce Ramgovind - Data Scientist & Machine Learning Expert at DotModus
AI can be used to add an intelligent layer of automation to your business today. Here's how we're using ML and AI to help our customers.
