How de-ageing VFX works and how it could change cinema forever

The first trailer for Martin Scorsese’s The Irishman has given us an early glimpse at the extensive de-aging work that’s being done to turn back the clock on Al Pacino, Robert De Niro and Joe Pesci.

The visual effect will allow them to play Jimmy Hoffa, Frank ‘The Irishman’ Sheeran, and Russell Bufalino respectively, requiring them to be 30 years younger than their actual ages at key points in the film.

The work is being done by Industrial Light & Magic and VFX Supervisor Pablo Helman, who will have a significant workload for the movie.

When Yahoo Movies UK spoke to the film’s editor, Thelma Schoonmaker, back in February, she revealed that the effect is going to be used extensively.

Read more: Thelma Schoonmaker: Scorsese’s ‘Irishman’ is 'not Goodfellas'

“We’re youthifying the actors in the first half of the movie. And then the second half of the movie they play their own age. So that’s a big risk.”

“We’re seeing some of it, but I haven’t gotten a whole scene where they’re young, and what I’m going to have to see, and what Marty’s going to have to see is, ‘How is it affecting the rest of the movie when you see them young?’”

“Interestingly, we’ve only been able to screen for very few people, because they’re wearing some things on their faces, and on their clothes, that tracks their movement… Nobody minds. Nobody minds watching them play young, because they’re gripped.”

But how is the effect achieved? We’ve done the research, spoken to some key visual effects artists, and the implications for the future of cinema are fairly staggering.

The 2D approach

This technique involves using footage of the actor on the set of the film, combined with reference material from movies or TV shows they made when they were the age they’re supposed to be in the movie.

A library sample of skin, eye colour, face shape, body shape, hair colour, hairline, and more detail will be created, and cross-referenced.

“We used [Kurt Russell] from ‘Used Car’ era,” James Gunn explained during the promotion for Guardians Of The Galaxy Vol 2, which features an impressively de-aged Kurt Russell. “He’s a bit older than Meredith Quill at the beginning of the movie.”

Read more: How Guardians of the Galaxy Vol. 2 de-aged Kurt Russell

Then, the hard work begins - the actor needs to be isolated from each frame, with every incongruent element of their face and body replaced by mattes, before using VFX techniques such as patching, cloning, digital paint, blur, glows, or a combination of all of them, to achieve the desired effect.

It’s possible to use tracking dots on the actor’s face to automate the work, allowing the computer to do the tough stuff, but it’s not always possible, requiring the digital artist to use digital paint to hand-draw on each frame.

Perhaps the most impressive recent use of this technique came in Captain Marvel, which took Samuel L Jackson back to the mid-90s for a significant part of the movie.

“It was a team effort,” Captain Marvel’s VFX supervisor Chris Townsend said. “There’s no magic bullet here. It’s all about artists painstakingly frame-by-frame going through and trying to make sure that you youthen, but you don’t step on the performance at all.”

“We’re transforming the actor’s appearance to look like another character, but just using digital technology,” said John Knoll - chief creative officer at Industrial Light & Magic and Star Wars: Rogue One’s visual effects supervisor, who added that the process is basically “a super high-tech and labor-intensive version of doing makeup.”

And it’s not just the look of the performer that’s being matched, but their individual tics and traits. “When Peter Cushing makes an ‘aah’ sound, he doesn’t move his upper lip. He only opens his jaw about halfway, and makes this square shape with his lower lip, that exposes his lower teeth.”

So, that’s a lot for the digital artists to think about - but the process can get even more detailed.

The 3D approach

This involves digitally scanning your actor and creating an ultra-realistic 3D model. This has the advantage of making integration easier - artists don’t have to worry about matching what’s already there in the physical performance, because they’re creating a character from scratch.

However, it’s less used than the 2D approach, partly because it’s more time-consuming (and therefore more expensive) and partly because it’s considered preferable to have some real element in there (and often performers are needed to provide a match for their movements anyway).

According to Gary Brozenich, the visual effects supervisor on Pirates 5 who worked with VFX house Lola to create a younger version of Johnny Depp’s Jack Sparrow for the movie, working with actors is preferable.

“[Lola’s] approach is to get the actual actor to do the actual performance and then they work on top of that. So underneath everything that you’re seeing there, it isn’t like he’s a computer-generated Johnny Depp, it’s actually Johnny Depp’s performance of Jack Sparrow manipulated to look like a younger version of himself,” Brozenich explained.

Read more: How Terminator Genisys lessons improved young Jack Sparrow

“So it’s his eyes, his mouth, his lips. They’re enhanced but that is the actual man underneath it. For me the big difference is that audiences are so in tune with Jack Sparrow, and it’s all about his performance – there’s no way to replace that. No matter how good you are technically, to try to recreate that spirit is not something that I never wanted to go near.”

The future of de-aging

Of course, there’s a slightly more unnerving aspect to the de-aging process, which so far has been used (mostly) on actors who are still alive, and able to give their permission.

But could we see stars being resurrected from the dead? Knoll, who brought Peter Cushing back to CGI-life for Rogue One, doesn’t think it’s an issue.

Read more: Star Wars guru on ethics of bringing Peter Cushing back to life

“I don’t imagine that happening, Knoll said. “This was done for very solid and defendable story reasons. This is a character that is very important to telling this kind of story. It is extremely labor-intensive and expensive to do. I don’t imagine anybody engaging in this kind of thing in a casual manner.

“We’re not planning on doing this digital re-creation extensively from now on,” Knoll added. “It just made sense for this particular movie.”

And if the reaction to Grand Moff Tarkin is to be taken into account, completely recreating a dead actor adds significantly to the uncanny valley effect.

Even Martin Scorsese has his doubts about how far the technology can go. “Does [the technique] change the eyes at all? If that’s the case, what was in the eyes that I liked? Was it intensity? Was it gravitas? Was it threat? And then how do we get it back? I don’t know.”

Still, with contracts including the rights to use actors’ images in any context (it’s how Rogue One was able to include a Princess Leia cameo without Carrie Fisher’s involvement), we might not be far off seeing ensemble casts featuring Hollywood icons of the past, decades after their actual final performances.

Cinema is changing - let’s hope that filmmaking ethics don’t.

The Irishman will hit Netflix UK this autumn.

(Editor’s note: This story has been edited since it was first published. It erroneously claimed that Rob Legato was the VFX supervisor on the film, not Pablo Helman.)