Much has happened in the AI world in the last weeks. This week alone, the President of the United States, Donald Trump, dismissed the idea that advanced AI could escape human control as a “hoax”; the prospect that “robots will be marching into our cities and getting rid of us all” was laughable. There is an eerie feeling for those of us who are online, seeing an official White House account post memes about Effective Altruism.
In contrast, the Anthropic CEO Dario Amodei had, days earlier, called for his industry to “pace the frontier”: slowing the development of AI technology and giving independent evaluators permanent, employee-level access to AI laboratories as independent monitors of its progress. Sam Altman, the OpenAI CEO, publicly agreed, promising similar access for outside evaluators.
The disagreement between Trump and the CEOs reflects a deeper split in opinion, especially within the technology world, between so-called accelerationists and those more preoccupied with safety. The former tend to see rapid progress as desirable, and attempts to slow it as both misguided and dangerous, especially when it comes to the United States’ position relative to China. By contrast, the latter worry that increasingly capable systems may outrun our ability to understand and ultimately control them.
Wherever one may land on the question, what is striking about current discussions around AI risks is that they return to one dominant fear: that humanity might summon into existence a power whose capacities exceed its comprehension, and discover, too late, that it cannot control it. In this vision of the world, AI would essentially take over the world and after that, potentially destroy humanity. Although I do not regard such fears as frivolous, my own anxiety concerns the rather less spectacular way humans may surrender control to AI. It’s also one that I find more likely and in some ways more frightening. I am worried that we might entrust more and more of our thinking to machines, and gradually cease to sustain the habits and institutions through which we had once cultivated our minds.
Such a surrender might happen amid considerable prosperity and be celebrated, at every stage, as progress. Each concession would come with an excellent justification: an economy of effort, a saving of time or an answer superior to anything we could produce ourselves. Only slowly would it become apparent that, in relieving ourselves of the burden of understanding, we had also diminished our capacity to govern our affairs. And by then, a preference for machine rule might seem entirely sensible. It would materialise not through violent and forced takeover, but by our own volition, with our own diminished abilities the strongest argument for such an arrangement.
We have, of course, outsourced parts of our thinking before. The calculator is the example most often invoked by those optimistic about AI’s impacts on the human mind. Few of us now perform complicated arithmetic by hand, yet mathematics itself, particularly its more conceptual parts, can hardly be said to have disappeared from human understanding. But AI seems different in kind, as well as degree. A calculator relieves us of a quite narrow operation whose purpose we already understand, whereas AI systems can increasingly take over large swathes of mental activity. Often, these are the very activities through which our understanding develops in the first place.
The evidence in this regard is still young, but it has already begun to confirm this intuition. In one eight-week study of 180 university students, those given the greatest freedom to use AI produced the strongest essays while the technology was available. But when the students were later required to write unaided, the advantage reversed in favour of those whose use of AI had been constrained. Given writing’s central role in forming our thoughts and making sense of the world, the prospect of these capacities weakening across an entire generation should be deeply troubling.
How to preserve human judgement, and the cultural and institutional life through which each generation acquires it, seems to me to be among the most consequential questions of our time, and it deserves a far larger place in our conception of “AI safety.” We face a gradual impoverishment of our minds at a time when we are already vulnerable. For decades, education in many developed countries has been deteriorating, a situation broadly attributable to mistaken ideas about what education is.
According to a recent Financial Times analysis by John Burn-Murdoch, teenagers in high-income countries are now roughly a year and a half behind their 2015 counterparts in literacy, knowledge, and reasoning, with the latest PISA results suggesting an accelerating decline. Even Finland, long invoked as evidence that the problem of education had been solved somewhere, has fallen towards the international average.
Burn-Murdoch suggests that educational decline reflects more than pandemic disruption or digital distraction. The outliers here are England and the state of Mississippi, which have fared much better than their peers: both have emphasised explicit teaching and foundational knowledge, resisting the fashionable belief that schools can cultivate general capacities such as “critical thinking” at the expense of teaching knowledge and “facts”. Judgement and critical thinking cannot be severed from knowing facts; in order to exercise judgement, you need a working model of the world based in reality.
As preposterous as it is, the idea that we should teach “thinking”, instead of teaching boring old facts has gained a lot of support in modern educational curricula. In an essay for Asterisk, the legal theorist turned mathematics teacher Alec Thompson uses a case study to explain why. He traces the way educationalists on both Left and Right dismantled Scotland’s once demanding, knowledge-rich curriculum from the 1960s onwards. The progressive Left tended to view an inherited canon and strict standards as constraints on children’s individuality and an instrument of class privilege. The market-oriented Right, meanwhile, tended to see education as merely instrumental, and thus an inefficient route to employment. Why immerse pupils in literature and history when schools could teach economically useful skills? So between an extremist egalitarianism, suspicious of intellectual authority, and a utilitarianism impatient with anything lacking an immediate return, education’s deeper purpose was forgotten.
Today, we remain confused about the purpose of education and how the human mind is shaped by learning — and it is into that confusion that AI arrives. We are acquiring the means to delegate more of our intellectual work to machines, just as we seem to have become increasingly confused about the values of cultivating our minds. Recent reactions to advances in AI mathematics have made the consequences of that confusion especially vivid to me.
It turns out that hacking is only one demonstration of AI’s growing capabilities. These systems are also increasingly tackling mathematical questions that have resisted decades of human effort. Last week OpenAI announced an AI-generated solution to the Navier–Stokes problem, a famous mathematical question about whether the equations that describe how fluids move always behave predictably. Such advances have prompted a group of prominent mathematicians, including Terence Tao, a recipient of the Fields Medal, one of mathematics’ highest honours, to warn of a “misalignment” between AI and their discipline in an open letter.
Their concern begins with the individual mind and what becomes of it when a groundbreaking mathematical solution can be obtained without the education and practice once necessary to produce it: “In many fields and activities,” they write, “years of training have traditionally served not only to produce a final answer or product, but also to develop understanding and the ability to formulate new questions and ideas.” In working on a problem, the mathematician had also been working upon himself, acquiring the subtle habits of mind that we call judgement.
“The mathematical community functions, in many ways, as a miniature version of humanity,” the signatories write. “It consists of individuals using a wide variety of different approaches, joined by core values.” Although individuals may continue to pursue mathematics out of curiosity and pure passion, these motives alone may not be able to sustain the institutions through which mathematical understanding develops. If a result can be obtained cheaply, why support years of apprenticeship?
There are reasonable criticisms of the letter. Perhaps mathematical institutions were already too narrowly organised around producing results. Or maybe AI could give mathematicians greater freedom to understand and ask better questions.
Yet much of the reaction I have seen on social media betrays little interest in these questions. What seems to animate many is the prospect of humiliation, the pleasure at imagining the people long admired for their intelligence becoming “jobless”. Others adopt the cooler language of economic interest: they see in the mathematicians’ letter merely a profession defending its status. To this group of people, the cartel feels threatened and the public will benefit from a more abundant supply of mathematics.
These reactions are disheartening. I do not know enough about mathematics to prescribe how the discipline should respond. But the prospect of entire fields of intellectual life being hollowed out, objective “results” multiplying as the communities capable of understanding and valuing them lose their institutional footing, deserves a much more serious reaction. I firmly believe that our capacity to govern ourselves, and the confidence with which we claim that right, depend upon people able to judge what is worth pursuing. Such people must be formed, patiently and uncertainly, within the very institutions whose continued existence is now threatened.
An increasingly familiar reassurance in the tech world is that human beings will remain “agentic”: machines will do the thinking, but humans will choose the goals and, ultimately, make things happen in the world.
This vaguely articulated framework does actually have some support from economic models. Earlier this year, a group of economists from MIT, UCLA and Washington University proposed a model for the transition to the post-AGI economy, with the roles reserved for humans best summarized below:
The answer is not a retreat into obsolescence, but a radical elevation of human purpose. When the marginal cost of generating an answer—and autonomously deducing the optimal policy—approaches zero, the value of human participation migrates entirely to the architecture of intent and the rigor of verification. We are moving from an era where our worth was defined by our capacity to build and discover, to an era where our survival depends on our capacity to steer, understand, and stand behind the meaning of what is created.
Such a future sounds disruptive, scary and unfamiliar. But shall it come to pass in its optimistic form (that is, if humans do actually retain control and ultimately imprint their intent on decisions), I think it would be among the more positive visions of a post-AGI future.
But what I am worried about is that as humans are outpaced and rendered obsolete in the most parts of cognitive labour, the incentives to maintain the institutions and practices that strengthen our minds so that we can even have the imagined intent, taste and capacity to wisely steer vastly more intelligent machines, will fade.
A myriad of questions seem unanswered to me: Where will the human’s purpose come from? Through what experience will his taste have matured? What will enable him to recognise that the advice he receives is mistaken, or that the goal he has chosen is unworthy of pursuit?
To separate our capacity to decide and exercise “agency” from our ability to think seems to me like making the old educational error in a more radical form. Educational reformers once imagined that knowledge and memorisation could be ignored, while at the same time leaving something called “critical thinking” intact. We are now being asked to believe that thinking itself can be handed over while human agency somehow would somehow survive untouched. But agency is not an abstract faculty floating free of everything else we do. Our purposes are inextricably linked to our acquisition of knowledge, to our thinking.
Agency, considered in the abstract, tells us little about what a person will do with it. Someone intelligent and well-read may devote extraordinary energy to building a library that enlarges the lives of his compatriots, while someone unintelligent and indoctrinated by religious extremism might use their agency to wage jihad against infidels. Not only that: by delegating the work of understanding, we may also relinquish some of the means through which we discover what is worth doing. The promise that humans will supply the goals while machines do the thinking treats our purposes as though they were already fully formed, awaiting the means of execution.
Now, it is true that few people (if any) explicitly say: “we should stop thinking” or “it’s ok if human cognitive abilities atrophy.” But it does seem to me like we are not paying sufficient attention to the problem and that implicitly, if not explicitly, it is assumed this is not among the most important considerations.
It is far from preordained that AI will make us intellectually weaker. But without active effort, the temptation to outsource our thinking will prevail, because it appeals to a common human weakness: the desire to spare ourselves effort.
We can already see that mitigations to our cognitive degradation are possible in small experiments. In one study of school-age maths students, an AI tutor offered hints rather than full answers. This seemed to avert most of the harm to students’ mathematical faculties. In this case, preserving the work of learning was an active design choice of the AI systems used in teaching. Making such choices the norm will require institutions that understand what intellectual development demands and considerable resolve to implement and defend such principles. Given the already unfavourable terrain on which we begin, building such institutions will be difficult. It will require at least as much sustained attention as we now devote to other forms of AI risk — including existential catastrophe.
Note: a slightly different version of this article appeared in Unherd two days ago.
Excellent piece. Thank you. An important part of the frame whenever people talk about disempowerment and “loss of control”
I agree that AI presents an immediate and existential threat to our existing education system. I'm hopeful, like you seem to be, that this threat will bring reform and reimagination rather than replacement. I'm curious why you use "safety" as the overarching category. Are you connecting the threat to education with broader safety concerns like those in the "existential threat" conversation? Or are you implying that positioning concerns about education as "safety" concerns is a helpful strategy because "safety" is dominating the discourse right now?