The first time you see Denis Shiryaev’s videos, they feel pretty miraculous. You can walk through New York as it was in 1911, or ride on Wuppertal’s flying train at the turn of the 20th century, or witness the birth of the moving image in a Leeds garden in 1888.
Shiryaev’s YouTube channel is a showcase for his company Neural Love, based in Gdansk, Poland, which uses a combination of neural networks and algorithms to overhaul historic images. Some of the very earliest surviving film has been cleaned, unscuffed, repaired, colourised, stabilised, corrected to 60 frames per second and upscaled to vivid 4K resolution.
For viewers, it almost feels like time travel. “That is something that our clients and even the commenters on YouTube have pointed out consistently,” says Elizabeth Peck, one of Shiryaev’s colleagues at Neural Love. “It brings you more into that real-life feeling of, ‘I’m here watching someone do this’, whereas before you’re looking more at something more artistic or cinematic.”
But these vivid videos and images haven’t wowed everyone. Digital upscalers and the millions who’ve watched their work on YouTube say they’re making the past relatable for viewers in 2020, but for some historians of art and image-making, modernising century-old archives brings a host of problems. Even adding colour to black and white photographs is hotly contested.
“The problem with colourisation is it leads people to just think about photographs as a kind of uncomplicated window onto the past, and that’s not what photographs are,” says Emily Mark-FitzGerald, Associate Professor at University College Dublin’s School of Art History and Cultural Policy.
Peck says Neural Love makes clear to clients the huge difference the company sees between “the restoration aspect and the enhancement aspect”. They see the removal of scratches, noise, dust or other imperfections picked up during processing as a less ethically fraught process to upscaling and colourising. “You’re really returning the film to its original state,” she says.
That’s not a view many academics hold, however. Luke McKernan, lead curator of news and moving images at the British Library, was particularly scathing about Peter Jackson’s 2018 World War One documentary They Shall Not Grow Old, which upscaled and colourised footage from the Western Front. Making the footage look more modern, he argued, undermined it. “It is a nonsense,” he wrote. “Colourisation does not bring us closer to the past; it increases the gap between now and then. It does not enable immediacy; it creates difference.”
The colours that suddenly flood into the streets of 1910s New York aren’t drawn from the celluloid itself; that information was never captured there. The extra frames added to smooth those New Yorkers’ 60 frame-a-second strolls are brand new too.
Neural Love uses several other manipulation programmes on its clips, slowly mending and finessing one stage at a time. To colourise its clips, it uses open source software DeOldify, whose developers Jason Antic and Dana Kelley are closing in on an all-purpose image restoration and colourising tool that would handle the whole process. “It’s very painful, though,” says Kelley. “Lots of training and lots of failed experiments.”
For Mark-FitzGerald and other historians of photography tools like DeOldify and Neural Love might make pictures look amazing, but they risk obscuring the past rather than illuminating it. “Even as a photo historian, I look at them and think, oh, wow, that’s quite an arresting image,” she says. “But always then my next impulse is to say, ‘Well, why am I having that response? And what is the person who’s made this intervention on the restoration actually doing? What information has this person added? What have they taken away?”
DeOldify and Neural Love, though, see their tools as a means of bridging the gap of understanding opened up by a century of technological advancement. Their tech is a means of making jerky, jittery images seem suddenly modern, but for historians, the distance between now and then is the whole point. “It’s the effort that creates the understanding,” McKernan writes. “Without that there is no true sympathy, only false sentiment. Film that looks like it was shot last week belongs only to last week.”
Peck compares Neural Love’s work to an installation at the Salvador Dali Museum in Florida which manipulates images of the artist to make him ask visitors for a selfie: “It’s making something that’s more palatable to a modern generation, which is used to interacting with media in a really different way.”
“We consider our work to be an adaptation of the original, similar to a modern take on Shakespeare or the translation of literature into another language,” Shiryaev adds via email. “The choices (or in this case, tech) involved with transforming the original have their own artistic merit, but the source content is still its own independent art form (and deserves to be experienced as such). Our work seeks to transform access to and awareness of the originals, not pose challenges to their authenticity or artistic merit.”
Antic and Kelley aren’t under any illusions that images treated by DeOldify will come out historically accurate, though their reservations are with the practicalities of training a neural network. Making sure colourised films are accurate is “a literally impossible problem,” Antic says. DeOldify uses modern images to train its AI on, he explains, “and we know that’s a big weakness, because, amongst other things, it biases people to wearing blue jeans.”
Rather, they say, they’d prefer DeOldify to be used as a platform to get the time-consuming bits of colourising taken care of – buildings, trees, skies – before a professional, human colourist makes sure everything is informed by proper historical research.
Neural Love spells this out to clients who want video colourising and smoothing. “We’re very clear about this: it’s not historically accurate,” says Peck. “It’s a neural network making a best guess estimation based on the vectors that are present in the film.”
To guard against anyone taking DeOldify’s images at face value, users have the option of leaving a watermark on any image they adapt with the software. However, as DeOldify is open source, many other developers using it don’t bother to add the watermark. “We can’t control what the rest of the world does with it,” says Antic. “The best we can do is just try to be thought leaders of a sort.”
The original videos still exist, of course, and Shiryaev’s YouTube videos break down each step in the process of restoration so no-one could mistake them for originals. Kelley compares their work to reading transcribed diaries rather than the original illegible scrawl.
For historians though, there’s a gap between the limitations and compromises their software has, which Antic and Kelley are happy to acknowledge, and the assumptions that anyone bumping into the images on social media might make. Getting people interested is one thing, but Mark-FitzGerald says there’s a need to critically assess what you’re seeing rather than passively absorbing whatever comes onto your Twitter feed.
On the internet these images, she says, “come unmoored” from how and why they were made, and how and why they were changed. She has already had students submitting essays which include falsely colourised images without realising it. “There’s something that’s gained, but there’s also something that’s lost,” says Mark-FitzGerald. “And I think we need to have a conversation about what both of those things are.”