This was a really interesting read - as someone who does not live in Silicon Valley and mostly follows the progression of AI through mainstream news outlets and daily interaction with public models at work, it's hard to tell from the outside how much of these predictions are worth feeling worried about. In the vein of some other commenters, you have to wonder whether these beliefs in handover or humanity being made obsolete by AGI are fuelled by the hubris of researchers who haven't ever had to encounter the inertia and human friction that comes from trying to change human systems.
Recent instances of cyber attacks, etc. are troubling, but for an outsider, it's hard to understand how the future that the leaders you've described imagine will actually come to be, in a realistic "this happens then that happens" way. In the time you've spent in these circles so far, do you feel that there are credible answers to these questions, or is it mostly just prophecy?
Thank you! I personally think there’s an under-appreciation of human frictions, yes. However, one could envision these frictions merely delaying handover as opposed to blocking it forever. And I think starting with a handoverist attitude can shape the future in and of itself and make it lean more towards handoverism on a spectrum, even if full-on handoverism isn’t achieved per se.
I think maybe AI 2040 is a good place to start for concrete scenarios:
I really just think every conversation about AI is driven entirely by anxiety. And even the people who are the closest to the products really have no idea what’s going to happen. Just as so many inventors haven’t really known what the most profound inventions would actually do or how they would be used. It is all just guesses. No one is in a better position to make the guest than anyone else. It’s a true horizon.
Which makes all the conversations about AI these days kind of exhausting, Because no one wants to talk about what they actually do know. All the conversations dissolved into the question that nobody can answer: what’s going to happen?
I take comfort in the fact that society has been on any number of precipices before, and we have rolled on.
An apocalypse is coming. I think that’s obvious. But it’s important to remember that the apocalypse is a myth. It isn’t literally true that in an apocalypse everyone dies. It just means that after the apocalypse is over no one can really identify with the person they were before. It is as if they’ve died, but we all live on.
It seems important though what assumptions we go into this Apocalypse with. For example, it doesn’t sound great to me that ceding control to AI is seen as a good outcome.
Lots of these people have just marinated in too much sci fi and cyberpunk to the exclusion of a more varied literary diet. Teenage nerds who never quite got beyond that.
The ones happy to see humanity die by AI are deformed monsters and it's grotesque that they have any cultural or political power.
This relatively small group of zealots in Silicon Valley knows what they’re building is going to destroy humanity, and not only do they persist, but they are racing towards it as fast as possible. The only conclusion that makes sense is that, for whatever incomprehensible reason, they want to destroy humanity. We have to stop them. Urgently.
To be clear, many of them feel compelled to make sure that their singleton is the one that ascends to the one available eternal throne because there are much worse alternatives, and they'd have to coordinate with them to stop it.
In fact, everyone would have to agree to put it down at once because if anyone defected, they would be able to snatch that eternal throne.
You can disagree with this view, but if you believed this, it would not be incredibly stupid to keep building it.
I feel like the answer why there aren't any positive visions of how life with superhuman AI will look like is here already in this post - we still haven't collectively recovered from the consequences of the last few grand utopian visions, and won't even dare to think of a new one. Ceding control over everything to AI does not sound inspiring at all, but I can imagine that a governance system with AI in the middle could help humanity align with itself, and have us resolve conflicts and come to deeper understanding on an ongoing basis. Democracy as it works so far is not great at catering to individual people's needs, hopes, and desires, because there was simply no way to take it all into account.
If I could hire a superintelligence to do one job, I'd ask it to interview every single person on this planet, listen to their hopes, fears and concerns, then combine all this data points together and come up with a plan how we can all move forward, that everyone can further discuss until a majority agrees. Start small, collect feedback, iterate, keep going. Surely this isn't too much to ask of a creature of supposedly godlike powers?
since our system is so centralized there is no “we” that pertains to almost all people in regards to decision making regarding AI
in a territorially and demographically vast and diversified system, absent widely and deeply federated governance structures and wide institutional plurality with wider and more diversified public access to decision making, and the requisite legal/regulatory variability and policy variability that is required for those things to in-practice be what they stated to be in a real way, well, then for almost all people there wont be any “we” in regards to decision making regarding AI
This was a really interesting read - as someone who does not live in Silicon Valley and mostly follows the progression of AI through mainstream news outlets and daily interaction with public models at work, it's hard to tell from the outside how much of these predictions are worth feeling worried about. In the vein of some other commenters, you have to wonder whether these beliefs in handover or humanity being made obsolete by AGI are fuelled by the hubris of researchers who haven't ever had to encounter the inertia and human friction that comes from trying to change human systems.
Recent instances of cyber attacks, etc. are troubling, but for an outsider, it's hard to understand how the future that the leaders you've described imagine will actually come to be, in a realistic "this happens then that happens" way. In the time you've spent in these circles so far, do you feel that there are credible answers to these questions, or is it mostly just prophecy?
Thank you! I personally think there’s an under-appreciation of human frictions, yes. However, one could envision these frictions merely delaying handover as opposed to blocking it forever. And I think starting with a handoverist attitude can shape the future in and of itself and make it lean more towards handoverism on a spectrum, even if full-on handoverism isn’t achieved per se.
I think maybe AI 2040 is a good place to start for concrete scenarios:
https://ai-2040.com/
I really just think every conversation about AI is driven entirely by anxiety. And even the people who are the closest to the products really have no idea what’s going to happen. Just as so many inventors haven’t really known what the most profound inventions would actually do or how they would be used. It is all just guesses. No one is in a better position to make the guest than anyone else. It’s a true horizon.
Which makes all the conversations about AI these days kind of exhausting, Because no one wants to talk about what they actually do know. All the conversations dissolved into the question that nobody can answer: what’s going to happen?
I take comfort in the fact that society has been on any number of precipices before, and we have rolled on.
An apocalypse is coming. I think that’s obvious. But it’s important to remember that the apocalypse is a myth. It isn’t literally true that in an apocalypse everyone dies. It just means that after the apocalypse is over no one can really identify with the person they were before. It is as if they’ve died, but we all live on.
It seems important though what assumptions we go into this Apocalypse with. For example, it doesn’t sound great to me that ceding control to AI is seen as a good outcome.
Lots of these people have just marinated in too much sci fi and cyberpunk to the exclusion of a more varied literary diet. Teenage nerds who never quite got beyond that.
The ones happy to see humanity die by AI are deformed monsters and it's grotesque that they have any cultural or political power.
Look what the Internet done, Ma.
This relatively small group of zealots in Silicon Valley knows what they’re building is going to destroy humanity, and not only do they persist, but they are racing towards it as fast as possible. The only conclusion that makes sense is that, for whatever incomprehensible reason, they want to destroy humanity. We have to stop them. Urgently.
To be clear, many of them feel compelled to make sure that their singleton is the one that ascends to the one available eternal throne because there are much worse alternatives, and they'd have to coordinate with them to stop it.
In fact, everyone would have to agree to put it down at once because if anyone defected, they would be able to snatch that eternal throne.
You can disagree with this view, but if you believed this, it would not be incredibly stupid to keep building it.
Yes I understand this, that’s why I called them zealots.
I feel like the answer why there aren't any positive visions of how life with superhuman AI will look like is here already in this post - we still haven't collectively recovered from the consequences of the last few grand utopian visions, and won't even dare to think of a new one. Ceding control over everything to AI does not sound inspiring at all, but I can imagine that a governance system with AI in the middle could help humanity align with itself, and have us resolve conflicts and come to deeper understanding on an ongoing basis. Democracy as it works so far is not great at catering to individual people's needs, hopes, and desires, because there was simply no way to take it all into account.
If I could hire a superintelligence to do one job, I'd ask it to interview every single person on this planet, listen to their hopes, fears and concerns, then combine all this data points together and come up with a plan how we can all move forward, that everyone can further discuss until a majority agrees. Start small, collect feedback, iterate, keep going. Surely this isn't too much to ask of a creature of supposedly godlike powers?
Exciting times ahead.
Huh. New to SF! Nice city, nice area, basically unaffordable for ordinary mortals. Look forward to your thoughts on our new AI overlords!
since our system is so centralized there is no “we” that pertains to almost all people in regards to decision making regarding AI
in a territorially and demographically vast and diversified system, absent widely and deeply federated governance structures and wide institutional plurality with wider and more diversified public access to decision making, and the requisite legal/regulatory variability and policy variability that is required for those things to in-practice be what they stated to be in a real way, well, then for almost all people there wont be any “we” in regards to decision making regarding AI