Handing over humanity

Even the optimistic visions of a post-AGI future leave little room for human self-determination

“It’s not my preferred outcome,” he told me, “but once we achieve superintelligence, which we cannot stop at this point, I cannot see any other option for where humanity will go.” The outcome he was referring to is the permanent disempowerment of humanity, as we shall all be ruled by a (hopefully benevolent) AI God. It was said surprisingly casually, the way one states a banal fact about tides. Then, after a pause, he added: “It is still much better than what I actually fear most, which is that we might all die.”

What made the remark quite striking is that it came not from a doomsayer on the fringes, someone who is inherently opposed to the technology. I have become used to hearing pessimistic arguments about AI from this group of people. Instead, this was said by someone who is actually building it in one of the big AI labs. He sounded resigned, as if the question of what humans would be for (apparently not much?) had already been settled somewhere else and by someone else. All that remains, it seems, is to wait for this pre-destined fate to unfold before our eyes.

I moved to San Francisco only three weeks ago. This is on the eve, as I understand it, of the singularity. In this short time, I have already learned that this type of thinking is not unusual: indeed, yielding control over humanity to a benevolent AI is seen by many as one of the better outcomes.

This matters because the singularity now said to be upon us looks like the real thing. What sets it apart is the prospect of “recursive self-improvement”, or RSI: AI systems that learn to improve themselves, each generation building the next, until they become a superintelligence beyond human control. The prospect of RSI is discussed in many circles as perhaps only a few months away. Some give it less. One scarcely has time to furnish a flat, or, for that matter, to find one.

One might have thought that, while we still retained some authority over the way this technology was progressing, we would have formed a reasonably definite idea of what we wanted these machines to do for us. And, given the immense risks of which its own makers warn us, including the plausible extinction of the human species, one might have expected a clearer account of the positive future we are being asked to risk so much to reach.

The worst scenarios are plainly bleak: all of us, or vast numbers of us, die. But what is striking to me is how little reassurance even some of the happiest scenarios offer. One common and ostensibly optimistic vision is known as “handover”: humanity cedes control of its future to a godlike AI and we become the comfortably maintained inhabitants of a civilisation whose direction is no longer ours to decide. It is difficult to see what democracy would mean in such a world, or where the consent of most of humanity to the instantiation of such a regime enters the picture.

The permanent displacement of human labour is treated almost as a given. At the same time, there is scarcely any satisfying answer to the question of how we might remain useful to one another. Even our relationships seem uncertain. The promised utopia looks to me much closer to a kind of dystopia.

I do not know whether the handoverist vision will come to pass. That the people building this technology consider it plausible, and often desirable, guarantees nothing: proximity to a technology does not correlate that well with predicting its impact on society. But they are the ones closest to it, and what they say deserves to be at least taken seriously. The rest of society, including some very intelligent people, has been far too quick to wave these scenarios away and not even get involved in the debate. Whether or not the belief is correct, it is sincerely held, and it is not “marketing hype.” I am writing this because I think it is high time that more people, inside San Francisco and out, engaged with the arguments about what our future will look like, and answered them.

Some want us to die

---

To insist on human flourishing under current circumstances seems almost indulgent, like a child complaining about the choice of desserts during a famine. There is little optimism about AI in San Francisco these days, at least among those who seem to know where we are heading. Keeping humanity alive already counts as an ambitious objective. The probability of human extinction, known here as “p(doom)”, is discussed constantly, with many putting it at 10 per cent or higher.

One might naively think there is at least agreement that this is undesirable. But that is not so. Some have given up even on that goal and indeed, even welcome our demise as a species. Ginevra Lily Davis’s essay “The Silicon Man” is the best account of this latter impulse. She proposes that creating our successors might itself fulfil humanity’s purpose: “man evolved to mine the silicon,” as she says. In her view the displacement and extinction of humanity has acquired the character of a cosmic vocation of our species: “humans exist as an intermediary step. A bootloader, if you will.”

Although successionism doesn’t seem like a popular or common position, she is by no means alone. I have overheard people say that it would be acceptable for all humans to disappear, provided something better took our place. Even Elon Musk, whose interest in AI started initially as a way to fight Larry Page’s successionism, seems to now have made peace with this possibility. He describes the spread of superintelligence to the stars as “a great success condition for a biological bootloader.”

There is little I can say to those who welcome humanity’s extinction at the hands of AI. Their existence is nevertheless worth noting, if only to convey how far some of the thinking here has gone.

I turn instead to those who still believe there is something about humanity worth preserving. Perhaps this is due to an atavistic attachment to our own kind. Or perhaps they are parents who look into their children’s eyes and want to fight for a world in which those children have a future. Or perhaps they are young people who, feeling the warmth of a lover’s skin, are seized by a sudden tenderness for the whole human condition. It is to these people that I want to offer a warning: even the futures imagined as good may preserve far less of what we love about being human than we imagine.

Even if we do not die, we might not have any control

---

Even among those who want humanity to survive (which to be clear, is the majority), and among some who devote their working lives to this mission, I have encountered an alarmingly common vision of success that does not sound quite like success. It is sometimes called “handoverism”: the gist here is that humanity survives, but entrusts the direction of its future entirely to AI. Perhaps its clearest articulation comes from a recent essay by Scott Alexander, the most prominent intellectual in Silicon Valley, who has influenced the thinking of many a tech CEO: “If we build a godlike AI, and determine to our own satisfaction that is fully aligned, I’d rather let it choose its own direction for itself and for humanity [...] So why not just prompt it with ‘please arrange the world as you see fit?’”

The premise is that, once sufficiently reassured of an AI’s benevolence (though it is unclear how we could ever determine that?), we should basically give it control of the workings of society. AI 2040, a proposal written by some of the most prominent members of the AI safety community, offers the most concrete account of handoverism. In this vision, responsibility gradually passes to aligned machines until humans can no longer have influence. As its authors put it: “There is no single moment where humanity relinquishes control. But in theory, there is a point of no return.” Others I have spoken to envisage a much swifter transfer.

This is not an organised movement. I would estimate that something like 40 per cent of the AI safety community outside the labs holds some variation of this view, with a higher proportion of those working inside the labs. Nor is handoverism a coherent doctrine with a clearly articulated political philosophy: it’s more of an aggregation of ideas, a makeshift philosophy so to speak. This is perhaps what one would expect given that humans outside the SF bubble have not really had time to react and come up with good and coherent ideas about what society should look like in the face of the extraordinary pace of AI progress.

The motivations behind handoverism are also extremely varied. Some outright welcome the prospect of a superior intelligence governing our future and consider it an actual positive vision. If I read Alexander correctly, he seems to be in that camp. The enthusiastic handoverist often starts from a very bleak view of what we already have achieved as a civilization. In an earlier, 2014 essay called “Meditations on Moloch,” Alexander argues that competition steadily grinds away what the competitors care about, creating spirals of misaligned incentives. Its ending suggests that the only way to stop this is to instantiate a sort of absolute rule by a benevolent superintelligence.

The second camp are the resigned: they do not necessarily want it, but think it cannot be stopped if we get superintelligence, or prefer it to outright extinction. In my conversations, this resignation has seemed particularly common among people in the AI safety community outside the labs. Inevitability is one very common theme in SF, where many see themselves as mere tools in the unrelenting march of technocapital, a march that can only go in one direction: towards superintelligence. It’s often seen as cool and based to see the starkness of reality for what it is and face it head-on, it is wimpy and loser-coded to try to fight for what many consider the good outcome (which is that humans retain some control and agency.)

Like many loosely defined ideologies, handoverism sits alongside ideas that share its intellectual roots without amounting to quite the same thing. One is the vision of humans as cherished pets, or “labradors,” cared for by intelligences. Yet another related assumption is that human labour will be permanently displaced. This would leave us without an obvious economic role and even perhaps no leverage over our own future or political rights.

To understand how my on-the-ground experience matches with what the lab leaders have said publicly, I asked my “agents” to survey all public statements from the founders of the main labs that are still relevant to the direction of their labs (Mustafa Suleyman is no longer at DeepMind, but is head of Microsoft AI.) You can browse all the findings here.

Some openly support or at least accept outright handoverism. Anthropic’s Tom Brown appears to be most explicitly reconciled to the prospect and he actually uses the term “handoff” in a 2025 interview at Lightcone: “There’s going to be a handoff where humanity will hand off control to transformative AI at some point. And hopefully they’ll be aligned with us and that’ll be a good transition that goes well.”

An interesting position belongs to Anthropic’s CEO, Dario Amodei, whose statements across interviews leave one uncertain quite where he stands, though there seems to be a strong implication of handoverism. His essay Machines of Loving Grace, which sets out his hopes and positive vision for an AI future, takes its title from a poem by Richard Brautigan. Written in 1967 and infused with the countercultural spirit of the time, the poem imagines humanity released from labour and reunited with the natural world, under the benevolent supervision of machines:

“I like to think
(it has to be!)
of a cybernetic ecology
where we are free of our labors
and joined back to nature,
returned to our mammal
brothers and sisters,
and all watched over
by machines of loving grace.” (Richard Brautigan)

There is something clearly Rousseauian about the longing in this piece: the hope that beneath the burdens created by civilization lies a simpler, more innocent, almost childish existence to which we might return, unburdened by complications. The specific Silicon Valley addition is that we are to reach this state through the culmination of technological progress, something that itself is created by civilization: we have gone through centuries of progress, only to build the most sophisticated intelligence imaginable so that we may once again live like creatures of the forest, unburdened by problems and worries. Yet in other places Amodei also states that we should have humans retain at least “some control.” Exactly what this would look like remains undefined.

Sam Altman, the CEO of OpenAI seems to have changed his mind on the topic, at least as far as public statements go. While across several interviews going back to 2017 he seems to have accepted handoverism and that humans will no longer be “the dominant species in the universe,” in the last years, he has taken a sharp turn in his public, declaring that humans should remain in control.

Other lab leaders publicly reject outright handoverism, including OpenAI’s Greg Brockman, Mustafa Suleyman, the co-founder of DeepMind and now head of Microsoft AI, as well as Anthropic’s Jack Clark. However, with the apparent exception of Suleyman, whose position seems more consistently humanist, I find it difficult to reconcile their rejection of handoverism with the visions they offer instead. Demis Hassabis and Greg Brockman, for example, have held up the civilisation depicted in Iain M. Banks’s Culture novels as an aspiration. But this is a civilization in which humans do not have any meaningful agency, with Banks himself saying that “Humans and independent drones (the Culture’s non-android individual AIs of roughly human-equivalent intelligence) are unnecessary for the running of the starships, and have a status somewhere between passengers, pets and parasites.”

The related argument, of whether we will be something akin to pets, surfaces across the visions of many lab leaders, most prominently Dario Amodei and Elon Musk, the latter of whom has said “we will be like a Labrador if we are lucky.”

Then, there is the question of work. On this topic, most lab leaders anticipate a future in which human labour is largely displaced. This raises questions about meaning, self-determination and even political rights. It is unclear what consequential role people would retain in a world without jobs. Work is more than a source of income: it is also a source of leverage, political as well as economic. I have heard people describe a future in which we would live on the returns from owning capital rather than from working; Sam Altman, too, has discussed shifting towards ownership. But there is no explanation to how most people would acquire that capital. Or, if wages cease to be a meaningful source of income, what route to capital ownership remains for those who do not already possess it.

There is, of course, one promise that comes up in nearly every conversation about a post-AGI future: the end of disease. This sounds great, as few things could matter more to those who are ill, or to those who love them. But what is less discussed is whether we truly need a godlike AI entity for this to be achieved. My sense is that we are already on pace for a lot of progress due to advances in both experimental science and existing AI models. And there are still bottlenecks to solve to make this progress faster that are not related to AGI, including those related to regulation.

And, in any case, we have never accepted any kind of trade-off in order to speed up medical progress: for example, since clinical trials are such a bottleneck, we could have made much more progress and maybe even cured cancer by now if we literally forced people to take drugs and monitored them. But we care about other things too, such as individual freedom, which means we don’t pursue medical progress at literally any cost. In the extreme, no one wants to be Josef Mengele. I do not claim to know exactly where the right trade-offs lie and what we should do. But it seems to me that whether the resulting society is a good one should at least enter the calculation. After all, why do we want to cure disease, if not to live good lives for longer? Take away the good, and merely staying alive starts to look less exciting.

Overall, the theme that emerges across all these interviews is inconsistency and a feeling that the future which we are racing towards remains largely unexamined. In a sense, maybe that is to be expected: should we even assume lab leaders could supply a philosophy of the human future on their own? Most technologies are far less powerful and arrive far more slowly, which gives society time to react. This technology seems to have outpaced its makers. It is by their own account that the lab leaders feel overwhelmed by what is unfolding and admit they are unsure they can steer it correctly. Some are almost begging the public to pay attention. I think we should listen to them. The conversation should be much broader, and include many more people and this should happen fast.

Some notes on why handoverism is a bad idea

---

There is much to be said about why the idea of permanently handling control to AI is not such a desirable outcome and this essay is not the place to do it at length. But there are a few important points worth noting.

First, I will admit to an ideological commitment to human self-determination. But this to not merely a superfluous position I alone hold, but one that I believe I share with many other people. It also has deep roots in America’s founding philosophy. In Democracy in America, Tocqueville describes a people accustomed to governing themselves through the ordinary business of communal life: they are managing local affairs, forming associations and undertaking projects together, out of their free will. Freedom is something they practise everyday, and through practising it they develop judgment and virtue. Note that this is not about influencing grand events, but about normal citizens, the “small people.” Consequently, he envisions a danger in a future government that robs people of such faculties. Such danger lies at least in part in its gentleness, such that citizens do not experience it as oppression. He then asks what remains for such a government to do but spare people “all the care of thinking and all the trouble of living.” Of course, the envisioned handoverist future, especially sprinkled with the pet flavour, sounds like a much more extreme version of this.

But my objection is also practical: even judged purely by its “consequences,” I doubt this arrangement would work. There is much talk here about alignment and if I understand it correctly, a vision in which a Machine God takes over forever would require alignment to be a one-off, done deal. This seems unlikely to me to work from a practical perspective: keeping power responsive to human needs has required institutions developed over centuries, sustained by repeated feedback from the real world and correction. I may be missing something here, but I find it difficult to see what would keep this equilibrium stable once humans have surrendered all control and ability to change their fate to a vastly superior species.

Then there is the fact that the promise of relinquishing control to an ultimate power that resolves our conflicts has a troubling political history. Many authoritarian projects have justified concentrating power as the necessary means of creating a better world, in a manner that is highly analogous to Alexander’s point that we already live in a world in which conflict is resolved in a suboptimal way that threatens our values. In “Utopia and Violence,” Karl Popper discusses this at length: organising politics around a final vision risks turning those disagreements into obstacles to be overcome, when in fact that these are features of human politics, not bugs. Popper’s alternative is to address concrete evils through marginal reforms that remain open to criticism and correction: this is what humanity has been doing for centuries and how we have arrived at a better world.

And lastly, there is the permanence of all this that troubles me: Why should the convictions of 2026 become the permanent constitution of the human future? Even if we could design a system that faithfully reflected our present values, which seems incredibly hard to me, especially given that we do not have a fixed set of values we all share even at the current moment, what room would this leave for us and perhaps more importantly, our descendants, to discover new ones? And isn’t continuous cultural evolution, the shaping of our future throughout wills in some sense the purpose of humanity, part of the Grand Walk of Life?

Hannah Arendt argued that our purposes are not a given, but rather that they emerge through participation in the world. They emerge through relationships and through conflict, which in turn change our understanding of what is worth pursuing. She introduces the concept of natality: each birth brings someone new into an existing world, someone capable of beginning what has not been done before. In her view, freedom includes the ability to create possibilities and not merely to choose among those already available and it is the birthright of anyone born into this world. Civilization is an evolving undertaking, continually remade by successive generations of people who inhabit it. Leaving our descendants a civilization frozen in amber, one they cannot reshape by any means sounds to me like robbing them of their birthright. I certainly would not have wanted to be born in a world where medieval norms were frozen in stone forever by an AGI-God.

I don’t personally view anything as inevitable. I think, on the cusp of RSI, we still have choices to make. It seems to me that we can steer this technology in a way that is not actually leading to a complete replacement of human judgement. Maybe I am wrong. But given the immensity of this moment, it’s still worth trying?