In this case, AI might just have saved the day. The flaws were discovered because cybersecurity experts used AI to search for weaknesses that humans had missed. Had, say, AI-assisted Russian cybercriminals got there first, it might have been far worse. The incident gives an insight into the double-edged nature of AI, which presents clear and present dangers — but also, new tools to combat those dangers. Note, however, that we are talking about the fairly unsexy business of cybercrime  and not, for example, sentient computers spontaneously deciding to eradicate humanity or farming our living souls in some sort of Matrix nightmare. And yet it is these scenarios, dreamed up by a small set of extravagantly funded AI doomers and spread by a credulous media that have occupied the world’s attention; and have gone into overdrive since the Hugging Face hack and Jacob Coxon’s resignation from Anthropic.

For “normies”, anxiety about a new emerging technology tends to center on things like jobs, education, or the economy. Cybersecurity experts tend to worry about tangible threats to existing systems. But AI doomerism is distinct. It describes people who believe that advanced AI poses a risk of human extinction — and are, generally, prepared to put a number on it. Coxon’s former colleague Evan Hubinger set his own “p(doom)” at “above 10%”; some go as high as 25% or even 50% .

Suggested reading

Why is Britain blocking AI?

By Tom Ough

Much has been written about the strange confluence of money, power, and quasi-theological belief systems that have given rise to this apocalyptic worldview. Still, what receives less attention is what this movement actually wants to happen in order to avert this admittedly unpleasant scenario — which is to say, the authoritarian and even totalitarian implication of the AI doomers’ policy proposals. Ideas of mass surveillance that would make Big Brother blush have almost imperceptibly crept from the fantastical thought experiments of Effective Altruists and the Rationalists and are being now turned into policy proposals by such figures as Bernie Sanders.

To understand the peculiar flavor of AI doomerism, a good place to start is a 2019 TED talk given by Oxford’s Nick Bostrom, one of the leading philosophers on existential risks, titled “How civilization could destroy itself—and 4 ways we could prevent it”. Bostrom opened his talk with a thought experiment. He asked his audience to imagine an urn containing various balls representing technological advances. There would be some white balls, representing beneficial discoveries. There would be some gray balls, representing discoveries that have some positive and some negative benefits. But there might be a few black balls: discoveries so profound, they would effectively make political life as we know it unsustainable. One example might be synthetic biology: imagine if your average supervillain really did gain the ability to create mutants! Another might be AGI.

Bostrom’s point was that once the ball is withdrawn from the urn, you can’t put it back. Therefore, the only viable technological safeguard would be to stop anyone from withdrawing any black balls in the first place. And the only real way of doing this, he mused, was: “Ubiquitous surveillance — everybody would be monitored all the time.” The interviewer’s thoughts turned immediately to dystopia: “This is Minority Report, essentially,” he said, referencing the 2002 Tom Cruise movie about a future police state that uses psychics to arrest criminals before they have a chance to commit their crimes. Bostrom duly admitted that there were “huge problems and risks” associated with the idea of mass surveillance and global governance. But perhaps, he mused, people would be more willing to wear multidirectional cameras and microphones if we called this “The ‘freedom tag’… or something like that”. Still, he suggested that “we seemed to be doomed anyway”, by which logic, extreme solutions may come to seem proportionate.

Over time, this “end justifies the means” mindset moved from thought experiment to policy proposal, with increasingly concrete ideas for controlling AI research and the people working on it. With the work of the Machine Intelligence Research Institute (MIRI), the case for totalizing enforcement became more explicit.

MIRI is a Berkeley-based nonprofit organization, founded more than two decades ago by the writer Eliezer Yudkowsky and backed by a remarkable concentration of money and influence. Its funders include Open Philanthropy, the philanthropic organization founded by Facebook co-founder Dustin Moskovitz and Cari Tuna; Vitalik Buterin, the co-founder of Ethereum (an open-source blockchain); and Jaan Tallinn, the Estonian programmer who helped build Skype and has since made existential risk one of the central objects of his philanthropy. Tallinn is a particularly important figure in this ecosystem: through the Survival and Flourishing Fund, which he funds alongside other philanthropists, he has helped direct hundreds of millions of dollars towards organizations concerned with AI existential risks.

MIRI’s particular obsession is artificial superintelligence (ASI) — a hypothetical scenario in which computers develop vastly superior intelligence to humans on every vector. Its position on ASI is: “Everyone dies.” Yudkowsky therefore argues that the only reasonable response is a globally coordinated and collectively enforced moratorium on the development of ASI.

This may not worry you, since you (I assume) have no intention of trying to create ASI. But the wider problem is enforcement. If governments agree to halt this kind of research, how do they know companies or individuals are obeying? In June, MIRI’s Technical Governance Team published Verifying Restrictions on Frontier AI Research, which considers how such a prohibition could actually be policed. The paper catalogs 28 possible verification mechanisms, including intelligence gathering; international search warrants and inspections (of properties, computers, and files); polygraphs; monitoring what users type into AI systems and what the model produces while it is being used; sting operations; employee-monitoring software; and AI-assisted code review, which may extend to codebases, experiment logs, and research documents. It is a proposal for giving governments unprecedented oversight of private activity. Researchers would work under permanent clouds of suspicion, never knowing which of their experiments might be deemed illicit, which conversations were incriminating, which of their colleagues was an informant.

This really is a dystopian nightmare, and, unfortunately, not the dystopia the media is covering. Journalists have spent years asking readers to imagine the dictatorship of future machines while paying far less attention to the systems of human control being proposed to prevent it.

MIRI itself acknowledged that such mechanisms could “enable authoritarianism” and cautioned that some “might be undesirable.” And yet thanks to the well-funded ecosystem of organizations surrounding MIRI, these ideas have now migrated into public policy proposals.

On 23 September, Senator Bernie Sanders and Representative Greg Casar introduced the Ban Artificial Superintelligence Act, which would permanently prohibit the development and deployment of artificial superintelligence and immediately pause advanced AI development until a new federal regulator was established to enforce the bans and strict penalties. The bill would create a cabinet-level Department of Artificial Intelligence with powers to monitor frontier AI systems and oversee their destruction. Individuals who violated or circumvented the ban could face up to 20 years in prison; companies could face what the sponsors call the “corporate death penalty”. That is a remarkable amount of government power over a technology that doesn’t even exist.

“A new age of AI radicalism is upon us,” concluded Adam Thierer and Nicholas Thielman of the R Street Institute, describing the Sanders-Casar framework as one of “the most radical interventions ever proposed in any piece of proposed legislation in American history”. Their report warned that “if enacted, it would have extremely destructive and dangerous consequences for innovation, competition, economic growth, global competitiveness, national security, and freedom of speech.”

They label the proposals “AI Authoritarianism”, arguing that it represents an unprecedented expansion of government control over technology and potentially over information itself, sharply restricting what companies, scientists, and individuals are allowed to do. Even pro-compute governance groups such as GovAI have cautioned that such a regime could “also be used to infringe on civil liberties, perpetuate existing power structures, and entrench authoritarian regimes.” Indeed, it could reasonably be argued that its implications are totalitarian — since enforcement will necessarily extend into private intellectual activity itself, through the monitoring of researchers’ prompts, code, messages, and devices via a pervasive surveillance system designed to prevent even private non-compliance.

These ideas have also found an energetic British advocate in ControlAI, a non-profit founded by Andrea Miotti to campaign against the existential risk posed by superintelligent AI. Its policy proposal also includes a 20-year AI pause. Backed by figures including Tallinn, the organization has spent the past two years briefing politicians in Britain, America, and elsewhere. In September, its UK campaign achieved a milestone: Labour MP Alex Sobel introduced a bill, drafted by ControlAI, that would ban the development of superintelligent AI in Britain for 20 years and require the Government to pursue an international agreement banning it worldwide. ControlAI also drafted a separate AI “Kill-Switch” amendment, designed to grant the government emergency powers to shut down AI models and data centers. And its US arm also consulted with the offices of Bernie Sanders and Greg Casar on their Ban Artificial Superintelligence Act.

The moratorium is extremely important to the AI doomers. Writing in Time in 2023, Yudkowsky warned readers: “Be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike.” Yudkowsky’s argument treats interstate military conflict, potentially involving nuclear-armed powers, as a lesser danger than his hypothetical risk of AI-induced catastrophe. In that framework, even conventional geopolitical dangers become subordinate to the threat he assigns to AI.

If it were up to Nick Bostrom and Eliezer Yudkowsky, we would therefore have a “Singleton” — a single, centralized, global decision-making agency with the power to monitor all computing power, an entity that would require absolute mass surveillance and the dismantling of national sovereignty.

Their ideas have roots in two overlapping subcultures: the Californian Rationalists and Oxford’s Effective Altruists, who by the early 2010s, were converging around longtermism: the idea that safeguarding humanity’s distant future should be a central moral priority. Figures such as Dustin Moskovitz, Sam Bankman-Fried, and Jaan Tallinn leverage the logic that EA had applied to philanthropy: if there was even the tiniest chance AI might one day wipe out humanity, that possibility warrants outsized funding today.

A pair of rather insular intellectual subcultures had developed something approaching a secular eschatology, turning sci-fi hypotheticals into moral imperatives. Critics and former members have described recurring problems of insularity, groupthink, and superiority complex. The true believers cast themselves as the tiny group who “get the danger” before the “ignorant masses” do. Qiaochu Yuan, a former rationalist, told Bloomberg that “Within the group, there was an unspoken sense of being the chosen people smart enough to see the truth and save the world, of being ‘cosmically significant.’”

“Why are they issuing exact probabilities of human extinction, which in fact are pulled out of thin air?” wondered Harvard’s Steven Pinker in a recent magazine article about their tendency to pronounce that there is, say, a 25% or 35% chance of us all dying by 2040. I am sorry to say this is how they talk. When you spend time in these circles, you clearly see their obsession with making up numbers and then quoting them as important facts when, essentially, they are “feelings dressed up as numbers.”

Those who actually engage with the technology, as opposed to the thought experiments, tend to find all this pretty absurd. One review of Yudkowsky and Nate Soares’s book If Anyone Builds It, Everyone Dies memorably retitled it: “If Anyone Reads It, Everyone Laughs.” “The real work of technology is found in implementation, adoption, maintenance, repair, adaptation, debugging, negotiation, administration, and other boring but essential tasks.”

In other words, it’s buggy PHP code, a battle between defenders and bad actors. This may not get the kind of clicks that doom-friendly news outlets aim for with headlines about “THE END OF HUMANITY” (the infamous Time magazine cover with two letters in flashing lights: AI). But it’s far closer to the realities of AI.

Suggested reading

The cult of AI armageddon

By Kathleen Stock

Nonetheless, the “AI will kill us all” camp is on media tours. They are also paying content creators to push AI existential risk messages to the public by sponsoring social media posts and partnering with influencers: influencers with tens of millions of subscribers, recruited to preach doom. The AI doomers have deep pockets  — the most recent estimate is $3 billion. The generous support of figures like Moskovitz, Tallinn, and convicted felon Sam Bankman-Fried has helped finance the ecosystem’s rapid expansion, with hundreds of organizations devoted to existential risks from AI. These billionaires have built the “AI safety” and “AI alignment” fields through grants, fellowships, research centers, training pipelines, media work, advocacy groups, and lobbying networks, as well as within frontier AI companies. Anthropic’s ties to Effective Altruism run unusually deep, spanning its funding, hires, and organizational culture. Anthropic’s policy advocacy — built around access controls, evaluation requirements, and proposals to “pace” development — could create a permission regime that large incumbents are far better equipped to navigate than smaller competitors, especially from the open-source community. Anthropic’s IPO is meanwhile expected to generate tens of billions of dollars in new philanthropic wealth for causes including AI existential risk. If so, the network built during the first AI-safety funding boom could soon have access to vastly more capital.

The danger is that this small but well-funded and influential group is claiming oversized moral and political authority over everyone else’s AI future. If AI really is as consequential as its most alarmed advocates claim, then concentrating control over it in a small number of frontier corporations, regulators, or self-appointed “experts” susceptible to the same groupthink creates its own category of risk. Broad access would give researchers, developers, businesses, and ordinary users greater opportunity to test systems, discover weaknesses, and develop competing alternatives.

What comes next does not have to be a choice between AI hype and predictions of apocalypse. A more grounded alternative is AI realism: treat AI as a powerful, fast-moving technology whose benefits and risks should be assessed through evidence rather than prophecy. The goal of AI regulation should be to target specific, concrete harms. That means distinguishing demonstrated harms from speculative scenarios, and treating the risks created by surveillance, censorship, market concentration, regulatory capture, and restrictions on scientific inquiry as part of the safety equation rather than acceptable collateral damage.

Before building an apparatus of surveillance and control to prevent a hypothetical dystopia, we should examine whether the proposed cure is becoming the dystopia itself.

Dr. Nirit Weiss-Blatt is a Communication Researcher and author of The TECHLASH and Tech Crisis Communication and the AI Panic newsletter.

Each in their own way, Trump and Khan have taken the rational position: existing laws and regulations, with modifications, are adequate to deal with any threats from the development of AI. Trump put it with typical bluster and poor grammar in his Truth Social post of Sept. 19 — referencing “AI Force” and “AI Czar” positions to which “Only High IQ individuals need apply! … We will not in any way hinder or stifle the Growth of this incredible Industry. Rather, we will cherish it, help it, and watch over it, as it grows! However, we will also be looking for BAD, and we can do that, very easily, with our already existing Criminal and Civil Justice System.”  

Khan came to a similar conclusion in a recent post of her own on X: “Law enforcers already have authority to charge companies and their CEOs for creating and releasing dangerous, unvetted, or defective products. We shouldn’t let discussions about new legal regimes distract from the fact that there’s no AI exemption from laws already on the books…. We can and must pursue any new efforts alongside enforcing existing laws.”

A policy of relying chiefly on existing legal and regulatory frameworks to govern AI, with modifications as needed, is one of three approaches to AI regulation. The other two are collective self-regulation by the private AI industry; and new, comprehensive government regulation specifically targeted at AI. There are many reasons to be skeptical of the latter two options, but the most important and least examined one is that proponents of both private self-regulation and comprehensive government regulation are looking at AI the wrong way: as a technology, rather than as a tool that has specific applications in different sectors such as national security, commerce, communications, and medicine. 

There is no “one” AI. 

A thoughtful case for private, nongovernmental self-regulation by the AI industry has been made by Aram Gavoor of George Washington University Law School in a recent essay. Gavoor claims that dangerous potential abuses of AI can be dealt with adequately by “soft law,” which he defines as “a regime of norms that create expectations and are broadly agreed to by a particular industry, but are not directly enforced by the state.”  

But with the exceptions of the Financial Industry Regulatory Authority and the Insurance Institute for Highway Safety, Gavoor’s examples of successful “soft law” self-regulation are mostly in industries in which the potential harm to the public is relatively minor, such as video games, which are regulated by the Entertainment Software Rating Board; online consumer privacy rights, addressed by the Do Not Track System; forestry, regulated by the Sustainable Forestry Initiative and Forest Stewardship Council; and college athletics, governed by the National Collegiate Athletic Association. The stakes are higher with AI. 

More from this author

In defence of data centres

By Michael Lind

Additionally, proponents of self-regulation underestimate the dangers of “regulatory capture,” in which incumbent companies use the rules issued by industry associations to stifle competition and reinforce their advantages. In this manner, professional bodies such as the American Medical Association, the American Bar Association, and the professoriate, rather than promoting the public interest, have produced nepotistic, self-enriching cartels. The method is also prone to ideological capture, whether by woke progressives or the advocates of “effective altruism” in the tech industry, as the infiltration and capture of policymaking at major corporations, media companies, and university bureaucracies by woke commissars in the last decade suggests.

In effect, Gavoor proposes to abandon representative democracy in the case of AI governance, reducing the role of government to rubber-stamping the rules that benevolent and trustworthy corporations collectively decide to adopt. He writes, “The most sustainable path forward is therefore an adaptive governance ecosystem: industry-led standards, validated by practice, disciplined by reputation and liability, and ultimately enshrined in law once their contours are clear.”   

The opposite proposed approach — comprehensive government regulation of AI as a technology — euthanizes democracy, as well. Proponents of comprehensive regulation of AI claim that the threat of the technology is so great, and so urgent, that the processes of ordinary democratic legislation and judicial and public debate will take too much time. What is needed to deal with the supposed AI emergency is empowered technocratic experts in a new government agency with sweeping authority, right now, before it is too late. It’s an emergency! There’s no time for debate or legislation! People are dying! We must act now to stop people from dying!

If this “argument from apocalypse” sounds familiar, it is because it has been used by progressives four times in the last decade to ram through measures that in hindsight even some on the Left now admit were extreme overreactions to real but limited problems.

The four moral panics of Black Lives Matter, trans radicalism, Covid hysteria, and climate extinction in the 2010s and 2020s all followed the same script. Because lives were at stake, there was no time for debate. Racist police officers were allegedly deliberately murdering black Americans. Trans kids were allegedly dying from suicide because the rest of society refused to agree that a person suffering from gender dysphoria is really, actually a member of the opposite sex. Huge numbers of people would perish needlessly from infection, unless society and the economy were locked down during the Covid pandemic. And, according to radical environmentalists, climate change would cause the human race itself to become extinct in the relatively near future.

Each one of these panics demanded that we trust credentialed establishment scientists to save us: Dr. Anthony Fauci in the case of Covid, pseudo-scientific race theorists in the case of Black Lives Matter, pseudo-scientific gender theorists in the case of trans radicalism, and a clique of so-called climate scientists in the case of global warming. The demand murders democracy in another way, as well: technical expertise, by its very nature, isn’t and can’t be democratic.

The technocratic progressive reverence for credentialed expertise led naturally to the third part of the moral-panic script: the call for emergency rule or martial law. The problem with such authoritarian measures is that they may turn out to be permanent. 

Having witnessed the progressive apocalypse shtick and its consequences four times in less than a decade, we are justified in being suspicious when the moral panic script is played out in a new form, in this case: unless we act now, AI will kill us all! 

The present hype cycle began with an unforeseen attack on the open-source AI platform Hugging Face by autonomous AI agents during a test by the company OpenAI in July. The fear-mongers claimed it was evidence that AI agents can already escape control and act on their own agenda. Yet Niels Rogge, a machine-learning engineer at the hacked platform, claimed that the cause was ultimately human error and said it was “bizarre nonsense” to suggest that rogue AI agents could take over the Internet. 

But predictably, the most sensational claims got the most media coverage. 

Then, on Sept. 8, Jacob Coxon, a junior employee at Anthropic, resigned, declaring: “The people building AI earnestly believe it could kill us all by the end of the decade.” The next day, another Anthropic employee, Evan Hubinger, asserted: “We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade.” On Sept. 12, the CEO of Anthropic, Dario Amodei, chimed in: “In six to 12 months … a swarm could be capable of taking over the entire internet with a persistent botnet — potentially causing….” Potentially causing human extinction? No, “ … potentially causing hundreds of billions of dollars in damage.” Whatever.

The fear that human-made artificial beings will turn against their makers and run amok has inspired science fiction from Mary Shelley’s Frankenstein in 1818 to Karel Čapek’s dystopian robot classic R.U.R. in 1920. In modern times, tech-adjacent subcultures have included “accelerationists” who hope to hurry an AI-based utopian millennium and “doomers” who fear that AI will destroy civilization or the human species. In 2024, Amodei and Bill Gates were among those who signed a one-sentence statement: “Mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war.”

But exactly how would a rogue AI agent cause human extinction? Dan Selsam of OpenAI has proposed that it might cause “runaway industrialization that makes the planet inhospitable to humans.”  

But wouldn’t it take decades or centuries to build so many factories and pump so much carbon dioxide into the air to drive humanity to extinction? And wouldn’t the diabolical computer mind’s scheme encounter resistance? Others have speculated that a rogue AI might take control of weapons systems or design, create, and disseminate a bioweapon capable of annihilating humanity.  

Max Tegmark of the Future of Life Institute, a nonprofit focusing on technology governance, argues that an artificial superintelligence could be a dangerous con artist: “To cause us trouble, such misaligned superhuman intelligence needs no robotic body, merely an internet connection — this may enable outsmarting financial markets, out-inventing human researchers, out-manipulating human leaders, and developing weapons we cannot even understand. Even if building robots were physically impossible, a super-intelligent and super-wealthy AI could easily pay or manipulate many humans to unwittingly do its bidding.”

Maybe. Then again, maybe some intelligence, human or artificial, would notice the silicon Bernie Madoff amassing a digital fortune and using it to pay or trick people into carrying out its nefarious plans.

Like many peddlers of the climate apocalypse, AI doomers have difficulty explaining exactly how what they oppose would bring about the extinction of the human race. As Rep. Alexandria Ocasio-Cortez (D-NY) once said: “For young people, climate change is bigger than election or re-election. It’s life or death.”  

Suggested reading

Have you corrected your racist robot?

By Sohrab Ahmari

But even in the worst-case scenario issued by the Intergovernmental Panel on Climate Change — a partisan body of environmentalists  — in which global temperature rises by 3.5 or 4 degrees Celsius, the consequences aren’t life or death. They include more dangerous heat exposure for people, increased water scarcity for a minority of the human race, modest sea level rises, coastal and island flooding, and — are you ready? — the possible extinction of 16.9% of all species of animal life, largely insects.  

Bad, to be sure, but incapable of causing the collapse of civilization, much less of making humanity go the way of the dinosaurs and the dodos.

In the same way, AI doomers struggle to explain exactly how “misaligned” AI could annihilate the human race, whether intentionally or by accident. In the Terminator film franchise, the malevolent computer program Skynet wipes out civilization by launching a global nuclear war.  In the real world, the command-and-control systems of nuclear weapons are “air-gapped” or disconnected from the public internet, and launching nuclear weapons additionally requires multiple stages of mechanical and human intervention.  

A RAND Corp. study entitled “On the Extinction Risk from Artificial Intelligence” concluded:  “In the three scenarios examined in this study — nuclear weapons, pathogens, and geoengineering — human extinction would not be a plausible outcome unless an actor was intentionally seeking that outcome. Even then, an actor would need to overcome significant constraints to achieve that goal.” 

Needless to say, this sober and skeptical assessment did not make headlines worldwide or turn the authors into overnight celebrities.

Moreover, the limits to human intelligence make comprehensive regulation of AI as a technology unlikely to succeed. For example, the European Union’s regulation policy toward AI requires that technologies be assessed on the basis of possible risks, ranked by potential danger.

But this requires an impossible degree of foresight.

Imagine that, following the development of the steam engine by James Watt, the British government had established a Ministry of Steam, charging ministers and civil servants to anticipate and make rules for all risks that might arise from the new steam engines. It is unlikely that even the most brilliant minds in 1800 could have anticipated many of the future applications of the new technology, much less make specific rules in advance for safety in the cases of steamships, steam locomotives, and steam-powered factories.  

Likewise, it would have been absurd for the United States in 1900 to create a Department of Electricity or an Internal Combustion Engine Agency to regulate all uses of these new technologies. The American tradition wisely involves regulating sectors rather than technologies.  By this logic, rather than having a general regulatory body, we should have AI applications to weapons regulated by the Department of Defense, AI technology in vehicles regulated by the Department of Transportation, AI technology in aircraft and air traffic control by the Federal Aviation Administration, and AI internet safety by the Federal Communications Commission. 

The exception that proves the rule is the misconceived and ill-fated Atomic Energy Commission.  Established in 1946, the AEC was charged with oversight of nuclear technology in general.  Multiple missions — promoting nuclear research and development, supporting the civilian application of nuclear energy while ensuring safety, and supervising nuclear weaponry — proved to be too much for one agency to carry out successfully. In 1974, Congress dissolved the AEC, and broke up the regulation of nuclear technology into different sectors, as made sense.

Alarm about possible harmful consequences of AI does not constitute a “hoax,” to use Donald Trump’s phrase, but it is overhyped. The genuine dangers that might arise as AI technology improves do not involve the extinction of humanity or its reduction to the status of slaves or pets by omnipotent silicon gods. Thus, the real dangers should be dealt with on a case-by-case basis by using existing laws and regulations and by supplementing them with new rules, where necessary, not by a moratorium on further AI research or the creation of an all-powerful, centralized bureaucracy with an excessively-broad mandate to regulate all applications of AI, existing or speculative. In regulating artificial intelligence, it is imperative to avoid human stupidity.