In contrast, the Anthropic CEO Dario Amodei had, days earlier, called for his industry to “pace the frontier”: slowing the development of AI technology and giving independent evaluators permanent, employee-level access to AI laboratories as independent monitors of its progress. Sam Altman, the OpenAI CEO, publicly agreed, promising similar access for outside evaluators.

The disagreement between Trump and the CEOs reflects a deeper split in opinion, especially within the technology world, between so-called accelerationists and those more preoccupied with safety. The former tend to see rapid progress as desirable, and attempts to slow it as both misguided and dangerous, especially when it comes to the United States’ position relative to China. By contrast, the latter worry that increasingly capable systems may outrun our ability to understand and ultimately control them.

Wherever one may land on the question, what is striking about current discussions around AI risks is that they return to one dominant fear: that humanity might summon into existence a power whose capacities exceed its comprehension, and discover, too late, that it cannot control it. In this vision of the world, AI would essentially take over the world and after that, potentially destroy humanity. Although I do not regard such fears as frivolous, my own anxiety concerns the rather less spectacular way humans may surrender control to AI. It’s also one that I find more likely and in some ways more frightening. I am worried that we might entrust more and more of our thinking to machines, and gradually cease to sustain the habits and institutions through which we had once cultivated our minds.

Such a surrender might happen amid considerable prosperity and be celebrated, at every stage, as progress. Each concession would come with an excellent justification: an economy of effort, a saving of time or an answer superior to anything we could produce ourselves. Only slowly would it become apparent that, in relieving ourselves of the burden of understanding, we had also diminished our capacity to govern our affairs. And by then, a preference for machine rule might seem entirely sensible. It would materialize not through violent and forced takeover, but by our own volition, with our own diminished abilities the strongest argument for such an arrangement.

We have, of course, outsourced parts of our thinking before. The calculator is the example most often invoked by those optimistic about AI’s impacts on the human mind. Few of us now perform complicated arithmetic by hand, yet mathematics itself, particularly its more conceptual parts, can hardly be said to have disappeared from human understanding. But AI seems different in kind, as well as degree. A calculator relieves us of a quite narrow operation whose purpose we already understand, whereas AI systems can increasingly take over large swathes of mental activity. Often, these are the very activities through which our understanding develops in the first place.

The evidence in this regard is still young, but it has already begun to confirm this intuition. In one eight-week study of 180 university students, those given the greatest freedom to use AI produced the strongest essays while the technology was available. But when the students were later required to write unaided, the advantage reversed in favor of those whose use of AI had been constrained. Given writing’s central role in forming our thoughts and making sense of the world, the prospect of these capacities weakening across an entire generation should be deeply troubling.

How to preserve human judgment, and the cultural and institutional life through which each generation acquires it, seems to me to be among the most consequential questions of our time, and it deserves a far larger place in our conception of “AI safety”. We face a gradual impoverishment of our minds at a time when we are already vulnerable. For decades, education in many developed countries has been deteriorating, a situation broadly attributable to mistaken ideas about what education is.

According to a recent Financial Times analysis by John Burn-Murdoch, teenagers in high-income countries are now roughly a year and a half behind their 2015 counterparts in literacy, knowledge, and reasoning, with the latest PISA results suggesting an accelerating decline. Even Finland, long invoked as evidence that the problem of education had been solved somewhere, has fallen towards the international average.

Suggested Reading

Claude: tyranny’s little helper

By Peter Frankopan

Burn-Murdoch suggests that educational decline reflects more than pandemic disruption or digital distraction. The outliers here are England and the state of Mississippi, which have fared much better than their peers: both have emphasized explicit teaching and foundational knowledge, resisting the fashionable belief that schools can cultivate general capacities such as “critical thinking” at the expense of teaching knowledge and “facts”. Judgment and critical thinking cannot be severed from knowing facts; in order to exercise judgment, you need a working model of the world based in reality.

As preposterous as it is, the idea that we should teach “thinking”, instead of teaching boring old facts has gained a lot of support in modern educational curricula. In an essay for Asterisk, the legal theorist turned mathematics teacher Alec Thompson uses a case study to explain why. He traces the way educationalists on both Left and Right dismantled Scotland’s once demanding, knowledge-rich curriculum from the Sixties onwards. The progressive Left tended to view an inherited canon and strict standards as constraints on children’s individuality and an instrument of class privilege. The market-oriented Right, meanwhile, tended to see education as merely instrumental, and thus an inefficient route to employment. Why immerse pupils in literature and history when schools could teach economically useful skills? So between an extremist egalitarianism, suspicious of intellectual authority, and a utilitarianism impatient with anything lacking an immediate return, education’s deeper purpose was forgotten.

Today, we remain confused about the purpose of education and how the human mind is shaped by learning — and it is into that confusion that AI arrives. We are acquiring the means to delegate more of our intellectual work to machines, just as we seem to have become increasingly confused about the values of cultivating our minds. Recent reactions to advances in AI mathematics have made the consequences of that confusion especially vivid to me.

It turns out that hacking is only one demonstration of AI’s growing capabilities. These systems are also increasingly tackling mathematical questions that have resisted decades of human effort. Last week OpenAI announced an AI-generated solution to the Navier–Stokes problem, a famous mathematical question about whether the equations that describe how fluids move always behave predictably. Such advances have prompted a group of prominent mathematicians, including Terence Tao, a recipient of the Fields Medal, one of mathematics’ highest honors, to warn of a “misalignment” between AI and their discipline in an open letter.

Their concern begins with the individual mind and what becomes of it when a groundbreaking mathematical solution can be obtained without the education and practice once necessary to produce it: “In many fields and activities,” they write, “years of training have traditionally served not only to produce a final answer or product, but also to develop understanding and the ability to formulate new questions and ideas.” In working on a problem, the mathematician had also been working upon himself, acquiring the subtle habits of mind that we call judgment.

“The mathematical community functions, in many ways, as a miniature version of humanity,” the signatories write. “It consists of individuals using a wide variety of different approaches, joined by core values.” Although individuals may continue to pursue mathematics out of curiosity and pure passion, these motives alone may not be able to sustain the institutions through which mathematical understanding develops. If a result can be obtained cheaply, why support years of apprenticeship?

There are reasonable criticisms of the letter. Perhaps mathematical institutions were already too narrowly organized around producing results. Or maybe AI could give mathematicians greater freedom to understand and ask better questions.

Yet much of the reaction I have seen on social media betrays little interest in these questions. What seems to animate many is the prospect of humiliation, the pleasure at imagining the people long admired for their intelligence becoming “jobless”. Others adopt the cooler language of economic interest: they see in the mathematicians’ letter merely a profession defending its status. To this group of people, the cartel feels threatened and the public will benefit from a more abundant supply of mathematics.

These reactions are disheartening. I do not know enough about mathematics to prescribe how the discipline should respond. But the prospect of entire fields of intellectual life being hollowed out, objective “results” multiplying as the communities capable of understanding and valuing them lose their institutional footing, deserves a much more serious reaction. I firmly believe that our capacity to govern ourselves, and the confidence with which we claim that right, depend upon people able to judge what is worth pursuing. Such people must be formed, patiently and uncertainly, within the very institutions whose continued existence is now threatened.

An increasingly familiar reassurance in the tech world is that human beings will remain “agentic”: machines will do the thinking, but humans will choose the goals and, ultimately, make things happen in the world.

One hears this division of labor between the agentic-humans and the thinking-machines described with a confidence that is quite befuddling. Where will the human’s purpose come from? Through what experience will his taste have matured? What will enable him to recognize that the advice he receives is mistaken, or that the goal he has chosen is unworthy of pursuit?

This is the old educational error in a more radical form. Educational reformers once imagined that knowledge and memorization could be ignored, while at the same time leaving something called “critical thinking” intact. We are now being asked to believe that thinking itself can be handed over while human agency somehow would somehow survive untouched. But agency is not an abstract faculty floating free of everything else we do. Our purposes are inextricably linked to our acquisition of knowledge, to our thinking.

More from this author

Why society hates losers

By Ruxandra Teslo

Agency, considered in the abstract, tells us little about what a person will do with it. Someone intelligent and well-read may devote extraordinary energy to building a library that enlarges the lives of his compatriots, while someone unintelligent and indoctrinated by religious extremism might use their agency to wage jihad against infidels. Not only that: by delegating the work of understanding, we may also relinquish some of the means through which we discover what is worth doing. The promise that humans will supply the goals while machines do the thinking treats our purposes as though they were already fully formed, awaiting the means of execution.

It is far from preordained that AI will make us intellectually weaker. But without active effort, the temptation to outsource our thinking will prevail, because it appeals to a common human weakness: the desire to spare ourselves effort.

We can already see that mitigations to our cognitive degradation are possible in small experiments. In one study of school-age math students, an AI tutor offered hints rather than full answers. This seemed to avert most of the harm to students’ mathematical faculties. In this case, preserving the work of learning was an active design choice of the AI systems used in teaching. Making such choices the norm will require institutions that understand what intellectual development demands and considerable resolve to implement and defend such principles. Given the already unfavorable terrain on which we begin, building such institutions will be difficult. It will require at least as much sustained attention as we now devote to other forms of AI risk — including existential catastrophe.

Think about it: when was the last time you praised someone for being noble or honorable? Even the word good has receded. In its place we find the more domesticated commendations: kind and nice. They are agreeable words, certainly, but also modest ones — virtues stripped of grandeur, careful not to be too much of anything. Self-sacrifice is almost unheard-of.

The moral ends that guide modern life now tend to gather around three dominant aims. The first is safety. This reveals itself in a vigilant concern for the preservation of both the body and, increasingly, the mind. Both of these are now understood as biological systems requiring maintenance, hence the growing societal preoccupation with mental health and the medicalization of emotional states. The second aim is utility: the relentless emphasis on efficiency, productivity and optimization. Life is measured in outputs and improvements, and its worth is justified in terms of functionality.

Lastly, there is the obsession with equality — the only one of these aims that still gestures toward something like a moral ideal. The others are largely managerial and materialistic. When I speak of equality as a dominant aim, I do not mean equality in the classical sense of a universal human right. What I have in mind instead is the contemporary drive toward the equalization of outcomes, the growing discomfort with hierarchy, and the proliferation of the narratives of oppression that now function as a central moral grammar in Western societies.

Nietzsche suggested that obsessive egalitarianism was the afterglow of Christianity: a moral residue that lingered after faith itself had faded. But Christianity bequeathed many values to the modern world: humility, charity, forgiveness, self-sacrifice. Out of these, equality, in particular, survived as the key virtue. The question is: why?

It seems to me that equality functions almost as a consolation prize of the secular age. The transition from an enchanted cosmos to a disenchanted, secular world has undoubtedly expanded human freedom. Yet in dismantling the old metaphysical framework, we have also stripped away the cosmic narrative that once conferred dignity even upon those at the margins of society: the so-called “losers”. 

Suggested reading

The age of nihilism is over

By Mary Harrington

The modern valorization of equality, of course, would be bizarre to our ancestors. In A Secular Age, the philosopher Charles Taylor offers a remarkable account of just how foreign the land of the past truly is. For Taylor, modernity is defined not simply by a decline in religious belief, but by a deeper shift in the background assumptions through which people experience reality: a change not merely in what people believe, but in how the world itself is imagined and lived. He describes this transformation as the passage from an enchanted world to one that is disenchanted and secular. 

In the pre-modern “enchanted” world, the world of “spirits, demons and moral forces”, reality was thick with agency and meaning. The cosmos was not neutral or inert, and people understood themselves as vulnerable to powers beyond their control, capable of being corrupted or elevated by forces outside themselves. The boundary between the self and the world was therefore permeable.

Taylor describes this condition as that of the “porous self”: a self embedded within a morally charged universe, open to external influences and aware of its dependence on a larger, divinely structured order. While this dependence implied a certain impotence, it also provided a powerful sense of belonging, situating individual lives within a cosmic order not of their own making. 

Disenchantment dissolves this structure. What disappears is not simply belief in spirits, but the very sense that meaning resides in the fabric of the world. The locus of meaning is relocated from the world to solely inside the human. And this gives rise to what Taylor terms the “buffered self”. The buffered self confronts the world from behind a kind of protective membrane. It no longer sees itself as vulnerable to cosmic forces. In the world of the buffered self, nature is no longer enchanted but inert, something to be explained, predicted, and manipulated. Even when we acknowledge the limits of human power, the dominant modern assumption is that natural obstacles will eventually yield to science and technology. Rather than propitiating unseen powers for a good harvest, we design irrigation systems and engineer crops.

The advantages of the buffered self — and of life in a disenchanted world — are obvious. Human beings no longer experience themselves as helpless before the powers of nature, nor do they feel their destinies to be at the mercy of capricious gods. Nature has become something to be understood and mastered. In this sense, the modern self appears capable of shaping its own fate.

This shift in mentality also helped make possible the extraordinary achievements of modernity. The economist Deirdre McCloskey is among those who have argued that this cultural transformation was crucial in enabling the Scientific Revolution and the economic changes that followed: developments that produced some of the greatest increases in living standards in human history. By any measure, it stands as an immense achievement.

But with such immense empowerment also comes a crushing burden: that of radical responsibility. Once we anchor meaning within the individual and we understand him to be radically able to improve his condition, we can no longer find refuge in helplessness. The secular mind implicitly understands power and merit as matters of will. The loser is solely responsible for their condition.

Formerly, within an enchanted worldview, earthly failure did not exhaust a life’s meaning: the poor and humiliated could still belong to a larger divine story in which salvation, not worldly success, was the ultimate measure. But in a disenchanted world, life’s losers are deprived of both the promise of ultimate reversal — “The first shall be last and the last shall be first,” as the Bible puts it. Even when it came to life on Earth itself, social hierarchy was seen as part of a divinely ordered cosmos, which granted even the humblest positions a certain dignity.

And more disturbingly, without all these, it becomes difficult to resist the conclusion that some lives contribute less, matter less and — even more disturbingly — might even be less necessary.

It is in erecting barriers to this conclusion that post-World War II modernity is at its most morally energetic. This moral energy manifests in the relentless pursuit of equality, the acute sensitivity to oppression and the refusal to accept hierarchy as natural. These phenomena are not merely fashions. They are attempts to supply, through political means, what religion once provided: an assurance that no life is negligible. In a world where dignity no longer descends from heaven, it must be constructed on earth. Or else the weak are left with nothing at all.

The “loser” is not a permanent class of people. It exists, in some degree, within each of us. Human beings are fragile creatures, capable of failure and weakness at any point. Indeed, the anxiety over “loserdom” is often strongest not among the most destitute but among the moderately successful — those who glimpse success and know they might yet fail to reach it. Something of this tension can be seen in the rising levels of anxiety reported at elite universities and other highly competitive environments.

Suggested reading

Why myths can save humanity

By Martin Shaw

Equality in a disenchanted world serves not only to reassure “the weak,” but also to place limits on the strong. It provides a moral vocabulary through which we can say that the strong are behaving improperly. In modern societies, strength is often defined in terms of utility: that is, those who are strong are those who generate wealth or have influence. Yet without a shared conception of the good life, it becomes difficult to explain why such individuals might still be failing in their obligations to the community. In earlier eras, even the most powerful figures were understood to stand under a higher moral order. Kings answered to God and power carried expectations of virtue and stewardship.

Modern liberal societies are less comfortable articulating such expectations. Over the past century, political thought, influenced no doubt by the rise of secularism, has increasingly embraced the idea that the state should remain neutral regarding competing visions of the good life. As the philosopher Michael Sandel has argued, freedom came to be defined largely as the ability of individuals to pursue their own preferences without interference. The result is a society that carefully protects rights but grows hesitant to articulate shared ends.

In this setting, criticism of the powerful often takes an indirect form. Public frustration with elites, whether directed at billionaires or corporations, is frequently expressed in the language of equality and redistribution. A recent flashpoint of this kind has been the proposal for a California billionaire tax. Yet beneath these demands often lies a deeper moral intuition: that the powerful are failing in some civic or ethical duty. Lacking a widely shared language of virtue or the common good, however, that intuition tends to be translated into the one moral grammar that remains widely legitimate — equality.

There remain pockets of the modern world where the obsession with equality recedes, most notably in the sphere of work. In such contexts, individuals can still find a sense of belonging and participation in a mission larger than themselves. Nowhere is this more visible than in start-ups. In such spaces, even the pretense of equality is often suspended in practice. Employees willingly subordinate themselves to a founder, often investing that figure with a quasi-prophetic aura. An entire mythology around the figure is born. “Founder-mode” might be the closest we get in the modern era to a grand hero narrative.

This hierarchy is accepted, even celebrated, because it promises belonging to a grand narrative: building the future and bending the arc of history through innovation. In recent years, companies like Anthropic have inspired precisely this kind of near-religious devotion in their employees. Access to such positions, however, is difficult, even for the fortunate. To most people, it is virtually inaccessible. A society cannot rely on such rare opportunities alone; it must find ways to generate sources of meaning for the many. 

One possible response is a return to a civic form of republicanism — the tradition that animated many of the Founding Fathers — in which the value of citizens is not measured solely by their economic utility but by their participation in a shared common good. Such a vision would require a society willing to articulate what that good consists of, and to speak openly about the virtues and forms of life that sustain it. 

Whether modern liberal societies still possess the confidence to define such shared ends — and whether disenchantment itself might be, even in part, reversible — remains an open question. What does seem clear, however, is that the hunger for such ends has not entirely vanished. Our continued ability to read 19th-century novels and not only recognize but feel drawn to their moral world is itself revealing.

At the same time, new technological forces — chiefly those unleashed by the rise of AI, from the spread of hyper-addictive gambling platforms to more subtle forms of behavioral manipulation — are beginning to unsettle even the most committed of liberals. The pressure to recover a richer moral vocabulary is gathering faster than almost anyone could have expected.