“There must have been about 10 apartments and there were 600 people,” he tells me. “And the situation looked so bad that a number of old people climbed up onto the roof and jumped off.” 

When not chewing the fat with me in the pub, John spends his time speaking in schools for the Holocaust Memorial Trust. Lest we forget. 

Of the remaining 200,000 or so Holocaust survivors, most are in their late eighties, which means that these vanishing witnesses to Nazi evil may all be gone within a decade. That is why, a few years back, John took part in Testimony 360, an innovative project to create a digital record of his experience. Thanks to some sophisticated AI and virtual-reality technology, future students will be able to interact with a virtual John, ask him questions, and learn from his testimony.

More from this author

Tommy Robinson is a cynical Christian

By Giles Fraser

But the use of AI in Holocaust remembrance is becoming increasingly controversial as fake images circulate around the internet. Such is the level of alarm that the International Holocaust Remembrance Alliance has issued a warning to schools, concerned that memes are being used “for political or ideological purposes, including denial and distortion”. Holocaust memory, they say, is under attack and false narratives are on the rise at a time of significant jeopardy for the Jewish community. 

For the time being, those with a keen eye and some knowledge of historical events can still tell the difference between fake images and real ones. There were only a handful of real photos taken within Auschwitz, and so images of children hiding under floorboards or playing violins ought to be treated with great caution. Fake ones are all too obvious, clearly designed to tug at the heartstrings, and often contain small mistakes. But as AI improves such giveaways will be ironed out. Inevitably, it will become harder to tell the difference.  

For some, the fictionalization of the Holocaust — in literature for example — is a useful way of introducing the young to the reality of the horrors of the genocide perpetrated by the Nazis — take The Boy in the Striped Pajamas, for example. It offers an accessible way into a historical reality that is too dreadful for children to absorb in its entirety. But even this can be controversial. As Imre Kertész, another Budapest-born Holocaust survivor, said in Who owns Auschwitz?: fictionalization can mean that “survivors watch helplessly as their only real possessions are done away with: authentic experiences”.

Whatever we conclude on this tricky question, bot-promoted deepfakes are different; it’s not clear that these images are a fiction at all. And so, as fabricated content floods social media, the truth becomes harder to recognize and easier to dismiss. Suddenly, a whole new world of Holocaust suspicion opens up. The fear is that certain forms of Holocaust remembrance could even be conscripted into Holocaust denial. 

In church, we know how technology can complicate memory. Wedding videographers promise to preserve a perfect record of that special day. Yet many couples find that, over time, this vivid recording crowds out any organic memories of the day, leaving only what the videographer saw. Ten years on, wedding couples admit they can’t actually remember those little details that made the day so real to them. Recorded “memories” can easily supplant real ones. 

In handing over responsibility to the videographer, you also relinquish control over how the day is curated and, potentially, remembered. AI imagery can do the same with history: creatively and powerfully interpreting historical events through particular perspectives, until interpretation muscles out the truth. 

Memory is the lifeblood of Jewish togetherness. Jews speak of the deceased as being “of blessed memory”. Passover is an act of collective liturgical memory, remembering the liberation from Egyptian bondage. “Remember the Sabbath day and keep it holy” exhorts the book of Exodus. The Hebrew Bible is full of instructions to remember what God has done. In the UK, we mark Holocaust Memorial Day today, the 81st anniversary of the liberation of Auschwitz. In Israel, remembrance takes place on Yom HaShoah in April or May (27th Nissan, to be precise), the anniversary of the Warsaw Ghetto uprising. The different dates register different emphases: one the liberation of passive Jews by others, the other the heroism of Jews who resisted the Nazi regime. But memory is the common thread. And threats to it are rightly regarded as existential. 

Suggested Reading

History has caught up with British Jews

By Tanya Gold

This is particularly salient today, amid the continued fallout from the October 7 massacre. Even though the Holocaust is the only historical event that is compulsory within the KS3 history curriculum, schools are becoming increasingly nervous about commemorating it. Only this month, a visit by a Jewish MP to a school in Bristol was canceled following protests by local pro-Palestinian groups. And over the past few years, the number of schools marking Holocaust Memorial Day has halved. 

There are some, Simon Jenkins for example, who argue that we should do less remembering and more forgetting. It is an unusual take for a historian. He warns us not to “drink from the rancid well of grievance”, and that “almost all the conflicts in the world are caused by too much remembering: refreshing religious divisions, tribal feuds, border conflicts, humiliations and expulsions”. I take the point but I don’t agree. Memory makes us who and what we are. It creates shared identity. That’s why so many of us fear dementia. To lose our memory is to forget who we are. 

There used to be graffiti on the towpath near our pub. Six million lies, it read. The graffiti has been washed away. But the lie has not. Memory, then, is both a reminder and a warning of what we are capable. Memory is what renews our conviction: never again.

And not just any old bikini deepfakes, even though many AI platforms are capable of creating them. It’s easy to do: you just say to the AI “Put her in a bikini” and the politician, celebrity, or — sometimes — child is suddenly wearing only a bikini. Yes, it’s creepy. But also striking is the way Labour’s ire is laser-focused on Grok, the AI attached to X, formerly Twitter. It’s also striking how, despite pleas from campaigners for years on the platform’s abundance of obscene material, it’s only now that Labour is swinging into outraged action. So grievous is the danger posed by its bikini deepfakes in particular (including of Starmer), that the Government now plans to outlaw them, even as it’s  threatened to ban X from Britain altogether.

This is where the political expediency bit comes in. In Starmer’s shoes I too would be itching for an excuse to ban X. Under Musk’s ownership, the site has made Starmer’s political life a misery, more or less from the moment he entered Downing Street, from stirring the pot during the Southport riots to amplifying the grooming scandal to the point that Labour were forced to respond with the Casey review. And the response to the review, again largely convened on X, forced Starmer to promise the national inquiry his party is now doing its best to avoid delivering.

This is a far cry from pre-Musk Twitter, which was a liberal-dominated hub of respectable media-class opinion, one with a direct line for governments and intelligence services seeking speech-control influence. But since Musk bought the site in 2022, it’s morphed by degrees into something far more pungent and anarchic, where Right-wing voices predominate and many of the former progressive Twitterati have fled to the Bluesky echo-chamber or simply quit posting.

This has not been an improvement in every respect. I don’t miss the sanctimonious tone of old Twitter, but it was more playful. Even so, the rambunctious Right-wing X is politically significant, not least in being the sole large-scale digital exception to the general rule that major institutions are controlled by the Left. As such, the site plays a pivotal role in UK politics. Whether feeding the “posting to policy pipeline”, or drawing attention to ham-fisted efforts at propaganda or speech control, X convenes so large and vocally anti-Starmer a subset of the British Right that we can hardly blame Keith for wishing, like Henry II, that someone (in this case Ofcom) would rid him of this turbulent website.

More from this author

Elon Musk: neo-feudal prince

By Mary Harrington

So is this just about Starmer’s ailing premiership? Yes and no. He’s at the helm of a regime that is, in its entirety, holed below the waterline by the subversive uncontrollability of digital public life. He can hardly be blamed for lashing out at a website whose denizens, and owner, so obviously loathe him. But the proposed ban is best understood as part of a wider war — as was the killing of Thomas à Becket.

Henry II had his Archbishop murdered in 1170 in the course of a dispute over the competing political and spiritual prerogatives of Crown and Church: a contest that obliged both parties to turn to the Pope in Rome for arbitration. And today Starmer finds himself at the mercy of not exactly a religious but certainly a cultural force, one whose extra-territorial leader also has an agenda that doesn’t always (or even very often) align with Starmer’s own. There is of course nothing very holy about Musk, or the trolls of X. But the contest over bikini deepfakes points toward a larger-scale struggle, with cultural, political, and economic dimensions and between overlapping Old and New World jurisdictions.

The field on which this is generally litigated is “free speech”, typically referred to as if it were an abstract moral value. But while there surely is a moral dimension to free speech, in an information age the power to free or control “speech”, and to determine what counts in that definition, is of immense political and economic consequence.

We are, in other words, once again litigating the extra-territorial reach of something intangible, much like Henry II. But in the intervening period Britain also gained her own territorial empire: and then lost it again, at the hands of our own former colony, in a way that also provides lessons for our situation now. In 1941, the British Empire, struggling against the Axis, was obliged to negotiate for aid from its secessionist progeny America. A key dispute within the negotiations was over “imperial preference”. This was a set of tariff arrangements enacted within the British Empire which gave preferential trading rights to polities that were part of the Empire. This so gravely disadvantaged American trade that its dissolution was a cornerstone of mid-century US foreign policy. Britain gave up on imperial preference and secured American aid.

I mention this not to re-litigate long-dead geopolitical quarrels, but to draw a parallel with the broader context of Starmer’s proposed ban of X. This modern argument doesn’t concern physical trade in goods, but the rules under which the digital kind are permitted to circulate. Many of these “goods” — including everything that happens on X — fall broadly under the heading of “speech”. In such a context, who gets to set the rules on “free speech” is obviously every bit as important as the 1941 battle over who gets to determine tariffs across British imperial territories.

As things stand, the digital “imperial preference” favors the polity in which the internet was first developed: the United States.  The question of geography is notoriously vexed when it comes to trade. Even determining who has tax and regulatory jurisdiction over (say) a pile of grain in a warehouse forms the topic of intense negotiations between trade experts. Intangible goods are still more complex, though: when a website can be served in multiple physical locations simultaneously, who gets to regulate it?

Extensive material and political interests rest on how this question is settled. And this is the real geopolitical backbeat to Starmer’s feud with Elon Musk: a backbeat that, in truth, encompasses not just Britain but also the EU to which Starmer clearly wishes Britain still belonged. It’s a feud currently playing out in a series of bitter skirmishes between the White House and the EU, as well as, in microcosm, between Musk and Starmer.

Recently, for example, US Secretary of State Marco Rubio announced sanctions against foreign individuals or institutions involved in censoring US digital properties: a clear shot across the bows of European politicians who believe they have a right to censor US websites and fine the companies  where these are served on EU territories. In the pandemonium that erupted following Rubio’s announcement, pro-EU supporters pointed out that no such challenges had been issued against Russia or China, despite these polities firewalling large portions of the American internet in its entirety.

But this is precisely the rub. Do the European states want to be as detached from the US as Russia and China, or not? There is no freedom of speech battle between these polities and the US, because neither Russia nor China tries to control what’s published on American platforms (at least, not openly). They just block them, and build their own platforms. By contrast, European states seem happy for America to supply its digital platforms, even as European politicians demand a say in how they are regulated. It’s as though 20th-century Canada was demanding, on the basis of its extensive wood and paper exports to Britain, a veto over everything published in London.

Until relatively recently, this relation of center and periphery was broadly mutually beneficial — not least in that it enabled American politicians to route round inconvenient constitutional clauses at home, in regulating the internet. Under Biden, the State Department encouraged and fostered European online speech controls, as a means of developing censorship tools both unconstitutional in the USA but nonetheless applicable domestically where desired, and aligned with an overall policy of digital speech control. I’m old enough to remember when something being illegal somewhere in Europe could result in its being deleted altogether from X even if the poster was in the USA. In effect, the Biden regime used Europe as a way of offshoring politically sensitive speech controls, somewhat like Guantanamo Bay offshored the torture of suspected jihadists.

More from this author

The tragedy of Keir Starmer’s Substack

By Mary Harrington

Now, though, priorities have changed — much to the bewilderment of Europe, which now has a well-developed censorship architecture but has discovered its American sponsor suddenly in a different mood. Even so, they have pressed on: two heavy-hitting US websites, X and Cloudflare, have already been threatened with fines under, respectively, the EU’s Digital Services Act and Italy’s “Piracy Shield”. The US-based forum 4chan has also been slapped with fines under Britain’s Digital Services Act. All have responded, essentially, “GFY”. Starmer’s threat to Musk on the deepfake image-making falls in the same category. It is an attempt to exert sovereignty in a space that’s both nominally open, but also in practice wholly dominated by American “imperial preference” — in the context of an American presidency taking a radical new direction.

Seen thus, the actual contest becomes clearer. Is Europe an American imperial subject, or a rival imperium? If it’s part of the empire, then obviously it doesn’t get to regulate its hegemon. If Europe is an imperial rival, it can hardly expect to demand its opponents supply services even as Europe dictates the terms. This is the real subject of the ongoing diplomatic wrestling-match over “free speech”, censorship, and regulatory frameworks such as the EU’s Digital Services Act and Britain’s Online Safety Act. If Britain, or any other European entity, really wants information sovereignty outside the American imperium, the only route there is the one now threatened by Starmer: banning American digital properties entirely from our territories, the way Russia and China do.

Does Starmer really want to go there? Does any EU state? This is perhaps uniquely difficult for Britain to answer. Geographically and economically speaking, Britain is closer to Europe, which accounts for around 41% of UK exports in 2024, compared to 18% for the US. But linguistically and culturally we’re closer to the USA. To take just one of countless examples, the BBC pays scant attention to EU politics, but Trump only has to raise an eyebrow and it’s front-page news.

In this context, can we really imagine an Atlantic firewall? I doubt it. Back in 1170, ridding himself of Thomas à Becket backfired badly on King Henry; I suspect that ridding himself of this turbulent X would prove similarly unpopular with the British media and political class. And yet our cultural, linguistic and digital enmeshment with an increasingly unpredictable US regime has introduced an acute new set of tensions. It is difficult to see how they can be resolved peacefully. But let us hope they can be. Having seen one bikini’d AI Starmer, I have no desire to see any more.

Today, fake barns are replaced with fake videos and images. “Pics or it didn’t happen” is a social media cliché but may soon become an outdated one. The use of “deepfakes” is growing — and the opportunities they bring for epistemic chaos are legion. Entrepreneurial types have already used them to put celebrities in porn, impersonate a CEO’s voice to make fraudulent money transfers, and hack bank facial recognition security software. We are all living in Fake Barn Country now.

As well as the risk of being fooled by misleading clips — watch deepfake Tom Cruise coupling up with Paris Hilton, for instance — there are also obvious worries about exponentially increasing the amount of misinformation out there. In an age already saturated with it, many fear where this is all going, including in Beijing. This week, the New York Times reported that the Chinese authorities have recently introduced strict rules requiring that any deepfakes used “have the subject’s consent and bear digital signatures or watermarks”. It turns out there are some advantages to having one of the world’s heaviest internet censorship systems. As a commentator told CNBC without apparent irony: “China is able to institute these rules because it already has systems in place to control the transmission of content in online spaces, and regulatory bodies in place that enforce these rules”.

Libertarians in the US, meanwhile, suggest that any attempt to control deepfake technology must be an infringement on free speech. Their main point seems to be that the law has no right to punish speech simply on grounds that it is false. But this is to treat a deepfake as if it were just any old kind of false statement, when in fact it’s potentially a falsehood squared — not just in terms of what’s being said, but also in terms of who’s saying it. It’s hard enough these days to get people to stop believing that reptilian aliens are secretly in control of things, without also showing them convincing videos of their favourite politician saying it too.

Equally, unlike with verbal or written falsehoods, most people won’t have any alternative way of checking whether Nigel Farage really does endorse the existence of lizard people, or whatever. Deepfakes affect the viewer on a visceral level, hacking every hardwired visual and aural recognition system you have. And there’s another problem, too. Part of the worry about undisclosed deepfakes is to do with audiences’ unfamiliarity with the technology involved, leaving them especially vulnerable to deception. At the same time, however, once general public literacy about deepfakes improves, then without clear and reliable signposting, there’s a real chance people won’t trust anything they ever see again — even when it’s completely kosher.

Perhaps wisely fearing the onset of paralysing public distrust, the general media position now appears to be one of disapproval towards covertly introduced deepfakes in factual contexts. When it was discovered in 2021 that three lines of deepfaked audio had appeared in a documentary about chef Anthony Bourdain, mimicking his voice undetectably, there was a lot of subsequent criticism; to this day, reviewers don’t seem to know exactly which lines of his were faked. In comparison, the response to the use of an AI-generated simulacrum of Andy Warhol’s voice reading out his own diaries in Netflix’s The Andy Warhol Diaries has been relatively positive — worries presumably disarmed by the fact that the presence of deepfaked audio was announced early in the first episode (or perhaps by the fact that apparently Warhol sounded like a robot anyway).

In this second case, though, director Andrew Rossi was able to offer a blanket disclaimer relatively easily because he was faking audio for all of the diary entries in the series, not just some of them. Similarly, in a recent BBC documentary about Alcoholics Anonymous members, all of the faces of the AA member participants were deepfaked to preserve anonymity, but not other participants, again allowing filmmakers a relatively easy way to differentiate for viewers at the beginning of the film.

Suggested reading

Tár and the triumph of amoral artists

By Kathleen Stock

A harder case for documentary-makers is where they might wish to deepfake some elements more haphazardly — perhaps because, as with the Bourdain film, particular bits of video and audio are missing from the archive, yet narratively important — and don’t have any easy way to indicate to the viewer which bits in particular these are. The problem is that telling the audience in advance that some bits of a documentary are deepfaked, without specifying exactly which, will leave them back in Fake Barn Country, unsure whether to trust anything of what they are seeing, real or not. (Indeed, this was my accidental experience with the Warhol documentary. I’d heard that some bits of archival footage were deepfaked but missed the disclaimer about the voice in particular, so spent the whole series fruitlessly trying to work out what was real Andy or fake Andy).

Faced with the haphazard deepfake, then, it might seem that the Chinese are right, and the use of a watermark is a good way forward — or at least, some visual aspect, simultaneous with the viewer’s experience of the deepfaked image or sound, indicating its lack of authenticity. In the 2020 HBO documentary Welcome To Chechnya, for instance, the visual effects supervisor Ryan Laney deliberately used “underblur” and a “halo” effect to indicate to the viewer that deepfaking was involved in representing the faces of persecuted Chechen gays and lesbians. He said: “There were times when we actually backed off on some areas, the machine did better or sharper, and we were concerned that we weren’t telling the audience that something was going on.”

I wonder if this requirement presents a bit of a conundrum for filmmakers though. Effectively, deepfaking somebody in a documentary, when it isn’t a deliberate attempt to mislead people, is best understood as a form of dramatic re-enactment. It’s done in the spirit of immersing viewers more completely in a narrative — making them feel imaginatively like they are right there, watching history unfold. It’s a creative decision on a continuum with using actors in disguise to re-enact key historical moments, as indeed The Andy Warhol Diaries also does copiously. Through the use of actors, we “see” him lying with lovers or staring depressively into space, just as through the use of deepfaked audio, we “hear” him reading out his diary. In neither instance is there any intent to deceive viewers, but rather to immerse them imaginatively in the moment. Even with the deepfaked alcoholics or Chechens, the documentary makers’ goal was to get viewers to relate to them as naturalistically as possible, given anonymity constraints.

Suggested reading

The art of empty apologies

By Kathleen Stock

Yet while introducing something like a watermark throughout a deepfaked scene may well be optimal from the perspective of responsible epistemology, it is pretty unsatisfactory from the perspective of aesthetics. For any such constant visual reminder is bound to inhibit the viewer’s imaginative engagement in a scene. If, during a dramatic re-enactment using actors, a director added a contemporaneous sign on-screen saying “these are just actors”, the whole point of the exercise would be undermined. I don’t see how it’s much different with a reminder to viewers that what they are now seeing is faked.

If I’m right, then the perhaps unsurprising moral of this story is that, just like forged paintings, or cosmetic surgery, or Andy Warhol’s wig, deepfakes only really “work” where their status as fake is at least somewhat hidden — whether because it was mentioned only once to viewers and then half-forgotten about, or because it was never mentioned at all in the first place. What’s perhaps more surprising is that this seems true even where the intent is mainly to get viewers to imagine something. If the viewer is fully conscious that an image is faked, she will be less likely to believe it; but she will also be unlikely even just to suspend her disbelief in the way that imaginative immersion in a dramatic re-enactment requires. When it comes to deepfakes in documentaries, then, unless you can find a way to use them cleverly, it seems to me you should possibly save your money altogether. For some creative purposes, it’s pointless to keep reminding people they are in Fake Barn Country.

One man who has done this countless times, to countless women, without their consent, is one Mr Deepfakes. As the founder of the most prominent deepfake porn website in existence, he has chosen to remain anonymous. He built the burgeoning community “from scratch” as a side hustle, after deepfake porn was banned from Reddit in 2018. It is, he claims, a place where “users affected by the ban could go”, to ensure the technology wasn’t unfairly “squashed”. MDF, as he styles himself, cites a commitment to free speech, and a desire to advance machine learning, as his sole motivations.

But in Deepfake Porn: Could you be next? MDF comes across as a man struggling with “the more moral aspect” of his work. On the one hand, he ardently claims to respect women (“100%”!) and insists this principle is reconcilable with his passion project. On the other, he has no plans to tell his wife — who would “probably be against” deepfake porn, “to put it bluntly” — about his work. “I’m afraid of how it would affect her, knowing I work on something like this.” He admits that “the content is actually in a grey area, and I think we’re on a fine line”.

Despite this ethical “indecision”, MDF’s website is thriving. It has amassed more than 20,000 deepfake videos of women loosely defined as “celebrities”, who are divided into 23 categories that include “Cosplay”, “Threesome” and “Asian Celeb”. Each day, an average of 25 new videos are added by a team of deepfake porn producers. There are 13 million original visitors who view this content every month, 10,000 of whom are online at any given moment.

Suggested reading

Porn will destroy you

By Sarah Ditum

In these videos, the facial expressions, mannerisms, and idiosyncrasies of the deepfaked subject do not belong to the victim whose face we see. They are the creation of a male fantasy. It not only looks like the victim is performing porn, when they never have, but also that they are engaging in the producer’s favourite sexual acts. Their identity is hijacked. When a victim sees herself embodied in the form of a porn performer, she describes it to me as a feeling of extreme disassociation. That is her face, but that is not her body.

Mr Deepfakes rakes in a high four-figure profit per month, mostly from ad revenue. He’s probably earning upwards of $100,000 a year from the venture. This money goes mostly to “maintaining the servers”.

Most of his effort, meanwhile, goes to imposing strict ethics on his website. Or so he claims. MDF recites his boundaries to minimise harm like a script: no porn niches that are “defamatory”; only “celebrities” allowed; the age limit is absolute; and producers must “make sure people know that these are fake videos” by ensuring “every video is watermarked”. But ultimately, MDF’s defence always comes down to separating the videos’ potential consequences from their creators. “We’re not all bad people!”

MDF claims he is simply an advocate of technological progress. He is, he says, facilitating the improvement of AI, while “other communities like Reddit wouldn’t allow it”. It just so happens that the best way to do this is through the “porn niche”: a convenient assumption that MDF treats as an awkward but inevitable fact. The “community” forum, the section of the website where content creators discuss how to improve results, is therefore what he cares about most. It fails to generate any revenue, at least for now. But this is hardly a charitable venture. This “community” is working to develop a tool that’s being used as a weapon against women.

Suggested reading

Test article

By RICHARD V. REEVES

And the ethical boundaries imposed by Mr Deepfakes fall apart on close inspection. There is no clear definition, for instance, of a “celebrity”. It includes women in the “mainstream media”, like those who appear in “Hollywood films”, but also “social media influencers”. And of course, “politicians are in the public domain”, so they can be targeted too. It is, in short, any woman with a public life. Many of these female “celebrities” already receive so much sexualised publicity in the media that MDF’s community think they are “fair game”. Or in the words of one user on the forum: “If you plaster your face everywhere and thirst trap me into seeking out your visage, then I’m just gonna deepfake you into porn.”

The majority of Mr Deepfakes community are “probably men”, he admits. Male users who are becoming radicalised by getting to “pick and choose aspects of different girls”, and digitally manipulate them into performing sexual acts.

Disturbingly, one regular poster on the site admits to deepfaking his co-workers. “Walking into work after having deepfaked these women, it did feel odd, but I just controlled my nerves. I can act like nothing is wrong, no one would suspect a thing.” The video is for his eyes only, he insists; it now exists on his hard drive, which he sees as merely an extension of his imagination. Asked if he would create a custom deepfake of an ordinary woman for someone else, he replied: “From a moral standpoint, yeah, I don’t think there’s anything that would stop me.” He went on to make a deepfake of someone after a Zoom call.

Suggested reading

The horror of deepfake nudes

By NINA SCHICK

The forums on MDF’s site seethe with misogyny, illustrating how utterly ineffective his ethical code is. “She didn’t let me smash her during highschool so cool. ima just deepfake u on porn and masturbate to it? CHECKMATE HOE,” one user writes. “It is never better when deepfake kicks in, and you get to make your dream celebrity be a mindless robot, and obey masters orders,” posted another.

Even those who have built a living on the porn industry are alarmed by the havoc deepfake technology could wreak on women’s lives. “The principal concern I have is how quickly the technology is evolving. It’s like a runaway freight train,” warns Viktor Zafirovski, a reviewer at the world’s largest porn directory website. Not only is it increasingly accessible, but they are getting easier to create.We will reach a point when “everyone will have to be paranoid about sharing their image,” says Viktor. And having made a small fortune developing this latest tool in the misogynist’s arsenal, even the man at the helm of the deepfake porn industry agrees. “It will be so convincing, that eventually you can’t identify what’s real and what’s fake,” MDF says. “You know, it’s scary to me as well.”

Underneath this cloak of concern for women, Mr Deepfakes must know that he empowers misogynists every day. “Porn runs the world right now. And it’s not something that I agree with with,” he says. Regardless, it’s something he leverages, even as he strives to distance himself from the culture he enables.

MDF doesn’t like to imagine how his wife would feel “if she found a video of herself on the internet — or maybe even a deepfake porn video of someone she knows”. But there are hundreds of women who, if they visited his site, might have the kind of horrifying realisation he wouldn’t want his loved ones to experience. The most frustrating thing is, he admits to feeling some discomfort. “I think I need to, you know, look deep down and see what I’m okay with,” he reflects. But his site remains live, and he keeps making money from it.

It is no exaggeration to say that soon almost everything we see or hear online will be synthetic — that is, generated or manipulated by AI. Machines that can “learn” to do almost anything when ‘trained’ on the right data — and they’ve never had access to more data, nor so much power to churn through it all. Some experts estimate that within 5-7 years, 90% of all video content online will be synthetic. Before long, anyone with a smartphone will be able to make Hollywood level AI-generated content.

One synthetic-text generating model can already generate articles that appear to have been written by a human. AI can be trained to clone someone’s voice even if they’re already dead: an old recording of JFK’s voice has been used to make a clip of the former president reading the Book of Genesis. AI trained on a dataset of human faces can generate convincing fake images of people who do not exist, and it can be taught to insert people into photographs and videos they were not originally in. One YouTuber is working on a project to insert actor Nicholas Cage into every movie ever made.

More from this author

Could Russian race-baiting tear the US apart?

By Nina Schick

All this sounds weird at worst and hilarious at best, but this technology has a dark side. It will, inevitably, be misused, and for that most obvious of male-driven reasons. It was reported last week that the messaging app Telegram is hosting a “deepfake pornography bot,” which allows users to generate images of naked women. According to the report, there are already over 100,000 such images being circulated in public Telegram channels; considering that they are almost certainly being shared privately too, the actual number in existence is likely much higher. The women who appear to feature in this cache of publicly-shared fake porn are mostly private individuals rather than celebrities. More disturbingly, the images also include deepfake nudes of underage girls.

Henry Ajder, the lead author of the report, told me that “the discovery of the bot and its surrounding ecosystem was a disturbing, yet sadly unsurprising, confirmation that the creation and sharing of malicious deepfakes is growing rapidly.” When he wrote to Telegram to request a takedown, he “received no response”. The bot and the fake pornography — including that of minors — is still live at time of writing.

Deepfake pornography first emerged less than three years ago, when an anonymous user started posting it on Reddit. He later revealed his methodology: he was using open-source AI software to insert celebrity faces into pornographic films, by training AI on images and videos of his intended target. The end result was surprisingly convincing. When other Redditors started making their own deepfake pornography and news of this AI-assisted community broke to the world, there was a furore: Reddit shut down the community and banned deepfake pornography. But the genie was out of the bottle.

Since its early days on Reddit, an entire deepfake pornography ecosystem has developed online. With a few clicks, it is possible to access deepfake pornography of every (female) celebrity imaginable, from Ivanka Trump and Michelle Obama to Ann Coulter. And celebrities are not the only targets. AI can clone any woman: all that is needed is some training data, and the rapid acceleration of the technology means that less and less training data is required. These days a single picture, a few seconds of a voice recording, or a single video would be enough.

Suggested reading

Just how harmful is porn?

By Tom Chivers

The tools to make synthetic media are becoming more and more accessible. Anyone can try to create their own by using free software; support communities online explain how to use it. It is even possible to buy “learn to deepfake” courses, or to commission a “deepfake artist” for a bespoke “creation” from little as $20. Recently,entrepreneurs have begun to wrap up the technology in easy-to-use app interfaces, so millions of consumers will be able to experiment with making their own AI-generated fake content.

One such app launched last year, calling itself “DeepNude”. It allowed users to upload photos of a woman to generate a deepfake image of her naked, and for a fee of $50, users could pay to remove a watermark so that the image would look authentic. (Because the underlying AI was developed on training data of female bodies, the app only worked on women.) When the DeepNude was released, the demand was so high that the app’s servers crashed under a stampede of downloads. Facing a barrage of negative press, its developers eventually took their creation offline, saying “the world is not ready for DeepNude app”. Weeks later, they quietly sold their software in an anonymous auction for $30,000.

While deepfake pornography ostensibly seems to be a “woman’s issue”, it provides an early and worrying case study for how synthetic media could be weaponised against us all. It is inevitable, for example, that deepfakes will be used as a potent new tool of identity theft and fraud.

Last week, federal prosecutors in the US revealed the case of a Californian widow who was scammed out of nearly $300,000 by an unidentified overseas con man who romanced her using deepfake videos in which he posed as the superintendent of the US Naval Academy. The widow, only identified as “M.M.”, thought she was building a relationship with an admiral named “Sean Buck”, who told her he was stationed on an aircraft carrier in the Middle East. They communicated for months via Skype, during which “Buck” always appeared dressed in his military uniform. According to the prosecutors, “While M.M. believed she was communicating with [Buck] via live chat on Skype, what she was seeing were actually manipulated [deepfake] clips of preexisting publicly-available video of the real Admiral Buck, and not the live video chats that M.M. believed them to be.”

Suggested reading

Why doesn't porn ever get cancelled?

By Sarah Ditum

The costs of identity theft and fraud are vast and on the rise. According to an annual report by Javelin Strategy & Research, identity fraud-related losses grew 15% in 2019 to $16.9 billion — in the US alone. This is largely due to the fact that financial institutions’ methods of identifying and responding to fraud are no match for criminals’ high-tech schemes to steal money, which increasingly incorporate deepfakes.

Of course, they will not only be utilised against consumers, but businesses too. The first serious case of deepfake business fraud emerged last year when the Wall Street Journal reported that a British energy company lost €250,000 when scammers used AI to clone the voice of the company’s CEO to demand (over a phone call) that the money transfer be made.

Libel, identity theft and fraud are nothing new — but the potency of such ventures will increase exponentially as synthetic media becomes prolific. Because of its unique ability to “clone”, AI presents a dire threat to an individual’s right to privacy and security. At the moment it is almost impossible to distinguish between authentic and synthetic media, and the quality of the latter is improving. This, then, is the alarming reality: a world in which our identities can be “hijacked” by almost anyone and used against us.

We have reached the critical moment to set standards — to create ethical and legal frameworks — to define how synthetic media should be created, labelled and identified. Given that the AI behind synthetic media is still nascent, we still have (a little) time to influence the effect this technology will have on our societies and the individuals within them. Too often we build exciting technology without considering how it might amplify the worst parts of human nature, or hand weapons to the cruellest impulses.