Also, that wild-eyed AGI booster needed to read your piece on bits vs atoms! We in tech who actually work on physical things (building & selling chips, servers, PC's/phones, precision test/manufacturing equipment) quietly nod along to AGI boosters when selling our wares to them (we often see ourselves very consciously as shovel providers) while often looking for hedges when it comes to our own personal finances, bc to be frank it's all obviously quite stupid...
What do you mean by "not solvable by intelligence"? I tend to think of intelligence as "the thing that humans have that chimpanzees don't, that allowed humans to bootstrap themselves up from being naked on the savannah to walking on the moon."
The fact that we can put a man on the moon while many of the world’s richest places are full of traffic jams and homeless people kind of supports the point of the article, though, doesn’t it?
When incentives, organizational structure, and so forth, are right, we can put a man on the moon. If they’re not, we end up with tent encampments in Santa Monica 50+ years later.
I don’t think it’s our intelligence which got us to the moon. It’s the ability to record information about what works and what does not, make mistakes, then iterate on what works and enormous number of times.
My view is that we are successful because we opened up a new evolutionary space, which we believed we planned and created through our genius. In reality, I think genius has been vastly, vastly overrated, just as “God’s design” was in creating a toad or a frog. It’s all evolutionary processes, and they don’t all tend towards progress. They can get us to the moon and can make getting to the moon much harder; there is no contradiction there
I agree with your larger point, but in fairness "he ability to record information about what works and what does not, make mistakes, then iterate on what works an enormous number of times" sounds a lot like intelligence.
It's possible to be highly intelligent without having access to good material, or institutional structures, for keeping long-term records. That's why there's so much variability in how much we know about different historical periods.
Intelligence, yes, but also opposable thumbs and oxygen-rich atmosphere. I wouldn’t be shocked if whales turned out to be as intelligent as Neanderthals, but they aren’t going to the moon, not by their own will and effort in any case.
They certainly had accurate data on overall giant squid population density long before we did, and there's a case to be made that exploring the deep ocean is in many ways a harder technical problem than going to space.
One of the things. Individual intelligence plus an aptitude for collective action, which chimps also have. (Also, you know, a nearby moon and stuff, but let's restrict this to circumstances that a clearly properties of a species.)
In this case, it means the problem isn't that we don't know how to build something, it's that someone else is preventing us from building it. The problem isn't that we can't think of new drugs to try or that we don't know how to synthesize them. It's that we aren't allowed to test them in a fast and inexpensive manner because other people are stopping us.
One solution to that particular problem is regulatory arbitrage - test your magic cure for cancer somewhere with more permissive regulators, such as ::insert country here:: and then blame the FDA for why people need to fly to ::insert country here:: to get your obviously effective magic cure for cancer. :P
The tl;dr version: with enough intelligence, you can make better use of the data you already have, and figure out ways to get more and better data out of whatever capacity for running experiments that you do have.
The scenario: you have the combined brainpower of the entire nation of South Korea sitting in your data center, running at 10,000x real time, trying to figure out how to work around the fact that doing experiments in the real world is too damn slow. Do you think the entire nation of South Korea, running at 10,000x real time, working together on the problem of "how do we work around US government restrictions on biological experiments in order to get the data we need to cure cancer forever as soon as possible" and nothing else, won't be able to think up a solution?
There's an old webcomic that parodies the "enhance" button from police procedurals where a computer somehow turns a grainy photo into a clear one. It features someone trying to "enhance" a photo consisting of exactly four pixels. Maybe 10,000x speed South Korea will figure out a way to make better use of data, or maybe it will be like trying to get an HD image out of a four pixel photo. It's hard to say ahead of time.
Fantastic piece! I don't know how you managed to fit in the difficulty of clinical trials and the culture of San Francisco into one coherent argument, but I really enjoyed reading this.
If I may indulge in a bit of speculative psychoanalysis: it seems natural to me that the folks in the AI sector think that superintelligence will have real-world superpowers, because it resonates unusually well with their own experience. Their own right-tail-for-humans intelligence has brought them wealth, power, and status well above that of almost everyone else. Of course they will intuitively assume that lots more intelligence will bring lots more wealth, power, and status, and of course they will overlook the boring non-intelligence-based bottlenecks which have always been especially boring and especially non-salient to them.
I think it's because of analogy. From the point of view of a dog or a chimp, we do have superpowers. We can do things that they can't even comprehend. We can kill by pointing a tube at something and pulling a trigger. We can let huge wheeled machines swallow us and spit us out at a far away destination. It isn't hard to imagine a superhuman AI might be just as superpowered to us as we are to chimps and dogs.
The issue, of course, is that it isn't just our intelligence that allows us to gain these powers. It's the system of cooperative production that we use to make the tools that our intelligence conceives of. A human on a desert island can probably figure out how to kill a chimp or a dog, but it will be much harder and riskier for them.
This take tallies with my experience working in tech dept of non-tech large orgs. The average coder is inclined by their experience to see the biggest (best?) problems as informational.
They live in information rich environments and not only do they gain wealth from their ability to navigate that, but also that wealth then obscures the fact that physical and social factors are the primary issues that holds back most people's flourishing.
When the basics take up most of your paycheck you're more inclined to think of the basics as the primary bottleneck.
I do the big boring stuff and many smart people have asked me why I prefer that grind instead of more exciting startup stuff (which IMO mostly does very little for the life of the average joe). It's a completely different understanding of what the problem is, let alone the solution.
I don't think this is because of wealth or power, at all. I think it's because bright nerdy types are at a big disadvantage in conventional status ladders (i.e. they're nerdy). So they instead concentrate all their personal effort and ego investment on the intelligence status ladder, and as a result become blind to other virtues.
(I would even say that being a right-tail software engineer is of no real status value, except within STEM/computer/Internet communities.)
You have the collective knowledge and brainpower of entire nation of South Korea sitting in your data center, running at 10,000 real time, with no need to eat or sleep, and completely dedicated to solving whatever problem it decides to work on. Would it make sense to consider that a data center like that would have real-world superpowers?
Actually, probably not! Because "completely dedicated to solving whatever problem it decides to work on" is smuggling in some unwarranted assumptions-- first and foremost, the assumption that there exist materially important real-world problems *that are solvable by pure thought*.
Problems solvable by pure thought? Such as, say, everything Einstein accomplished in theoretical physics? Every result in mathematics ever? Figuring out whether a proposed machine design is going to work, given descriptions of materials and the laws of physics? Taking a list of 140,000 protein structures and turning it into a solution to the protein folding problem?
All of which had to be verified by experiment, and none of which ever conferred on Einstein himself any superpowers whatsoever, nor on anyone else till it was followed up by decades of tough, incremental, experiment-heavy engineering work.
So this piece was brought into inception by experience (likely umpteenth time) of being mansplained to by typical overconfident but ignorant AI/AGI (likely also crypto) booster lol
Does quality of governance scale with intelligence? e.g. Clearly there is some level of intelligence below which you are unable to keep track of basic facts. Maybe IQ of ~80? That's the minimum IQ allowed to serve in the US Military so it's a fair floor. If everyone in the USDA had an IQ of 80, would we expect drug approval rates to go up or down?
(A) as a broader point, governance itself will be impacted by AI in various ways (including for example, more autistic people making money from AI and donating to very specific political causes as a result).
(B) More specifically to your point, I think the intelligence of a population involved in governance changes governance. That being said, I am not sure exactly how this will play with AI systems, as they are currently more like disembodied forms of intelligence. I think as long as humans are in control there will be a lot of friction in terms of diffusing machine intelligence into humans in a way that humans want to follow. What I mean by that, is that the human motivational structure is tied to the IQ of the human, imo. Even if a human knows from an AI that they should want X thing, I am not sure that will hit their core motivational structure in a way that will make the human want to advocate for said thing.
Ruxandra, this was an exceptional piece and one of the best IMO I can remember you having written. Just totally awesome.
In particular I appreciate your emphasis on solving ‘quotidian’-seeming-but-equivalently-existential societal challenges; on the circumstance-specific adaptiveness (but also maladaptiveness) of certain orientations, which we see also in VC; and on the fundamental fact that we exist in a world defined first and foremost by institutions and their highly idiosyncratic behavior.
I am so hype that you are able now to pour your full attention into writing about this stuff.
Regulation and whatnot get in the way of many things. We get in our own way due to greed, malice, and competition. It is likely that Healthcare and Medicine will be bottlenecked by those, directly. It is also likely that the entire approach to medicine is itself 'messed up'. It treats symptoms, not identifying causes and prevention. It looks for single-pill cures, not systemic life-changes. Until healthcare becomes personalized health-CREATION, the entire strategy to improve human life is fallible. AI can help in many domains, especially at the personalized level. Given AI is already accelerating the health of a number of people, it seems likely that a shift may occur where personalized doctors help to identify and prevent issues for individual people before they become 10-100x more expensive to handle.
One other reason that medicine may prove less tractable to AI than its most evangelical adherents believe, is that many diseases remain poorly understood. I accept that AI could design a superior antibody against amyloid but that improvement may correlate quite poorly with genuine clinical enhancement (to take one example). Likewise Parkinson’s Disease.
As you rightly suggest, clinical trials remain highly primitive and while AI will help with both the operational aspects and patient selection, there is a finite limit in the time improvement that such an optimized approach could yield. And that’s before we talk about reimbursement.
One final thought; having watched “the thinking game”, many of the most vociferous AI researchers are simultaneously exceptionally bright but very narrow in their knowledge and world view.
Yes, I agree. But the real confusion is kinda happening a bit before. These people fundamentally imagine that an AI will essentially be computationally omniscient -- ie have the ability to perform any computation virtually instantly -- and yes, if that was true AI would basically be a demigod.
And this goes to a deeper confusion about what is meant by intelligence. If intelligence is the ability to infer the answer to something like mathematical problems then *enough* intelligence let's one do anything. But that's not a kind of intelligent that AI could actually be. There is only so efficient the an algorithm for a problem can get.
Yes, an AI equipped with an infinitely fast computer might be a demi-god -- they could just consider all possible scenarios consistent with their observations and run them forward to maximize their expected outcome -- but so would I.
OTOH if you want intelligence to mean something about the software not just raw computational power all the sudden there is no particular reason that AI's should be particularly amazing. Hell we may already be near the algorithmic efficiency limits for the kind of things we care about. And for abilities like persuasion there is just no reason to believe the AI will even be able to overcome disadvantages like people knowing they are an AI or not being able to be a charming human and trigger all our evolved social reactions.
The "AI box experiment": You talk to someone over a text chat that role-plays an AI, and you role-play someone authorized to let the AI "out of the box" and into the world. If you spend a few hours talking to them and haven't agreed to let the AI out of the box, the human roleplaying the AI will give you $20.
Do you think you could be convinced to let the AI out of the box? The success rate of people playing the AI has been about 50%, I think...
How exactly does that relate to what I said? Yes, people can be pretty persuasive. We know that even very persuasive people face huge challenges in getting large institutions designed to make it hard to get approval quickly give that approval *quickly*. For this not to be the barrier you need so much persuasion you convince people to fundamentally change the system.
---
But on to the whole box thing my first response is so what? I don't see why an AI would be so much more dangerous than a smart religious nutter person nor why alignment should be much harder than it is for people. If it's no worse than a really bad terrorists escaping prison ¯\_(ツ)_/¯.
But more on point, that's why we have multiple layers of approval to do many things. You just stack barriers until probability drops. So far no terrorists have social engineered their way into a nuclear launch.
And sure that wouldn't work if you presume AI is way way beyond human persuasion levels and that it usually acts like a ideal bond villain secretly plotting to escape and that it's a disaster if it does but not clear why any of those should be so.
In other words, intelligence is not a bottleneck, the bottleneck is solving coordination problems, such as how to agree on a better legislation or a better patent system.
And it's not evident whether a society made only of AGIs would be necessarily better at solving coordination problems that one made entirely of humans. (Insert some stuff about game theory, Moloch and all that here...)
But given that we are better at solving coordination problems than chimpanzees (we have laws and states and whatnot, after all) is a case for a mild optimism.
Unless, of course, one would argue that humans fare better than chimpanzees not because of their intelligence, but because of some other aspect of human biology/psychology/etology.
One aspect I wonder about is whether the bottleneck is even broader than regulation or governance.
We tend to treat words, data, models and biomarkers as if they simply represent reality, when in practice they also shape what becomes visible, actionable and worthy of attention. In that sense, style, thinking, speech and action are not merely expressions of an ontology; they enact it—and in turn participate in its continual reconstitution.
The challenge is therefore not only improving our representations, but remaining alert to what they inevitably leave out. This is to 'hold' them lightly and tentatively. Otherwise there is a subtle drift whereby the map progressively becomes the territory, and our interventions become directed by the representation rather than the phenomenon itself.
Thank you for a nuanced contribution to an increasingly polarised discussion.
In the realms of science and engineering, a lack of data -- actual, real data from the world of atoms and organisms -- will be a major bottleneck for the foreseeable future. AGI that is 10,000 times smarter than a human (whatever that even means) will not change this fact. In fact, as AI eliminates bottlenecks in the realm of bits and cognition, those chokepoints in the realm of atoms will become even more acute, and hopefully more evident to the currently oblivious AI boomers.
The good news is that it is by recognizing and eliminating bottlenecks that savvy entrepreneurs build thriving companies and become rich. So new, or newly apparent, bottlenecks are a fertile field for builders. The bad news is that if the legal and regulatory environment does not permit freedom of action in a given domain, this won't happen. So the ultimate bottleneck in the realm of medicine is likely to be regulation, as you noted here.
In a way, it's really funny that you heard this in San Francisco, a place where the obvious bottleneck is the political dysfunction which makes it impossible to build housing there, a technology which has existed in some forms since antiquity.
Very useful piece. This is a point I've been trying to make for a long while: many problems are not solvable by intelligence!
Also, that wild-eyed AGI booster needed to read your piece on bits vs atoms! We in tech who actually work on physical things (building & selling chips, servers, PC's/phones, precision test/manufacturing equipment) quietly nod along to AGI boosters when selling our wares to them (we often see ourselves very consciously as shovel providers) while often looking for hedges when it comes to our own personal finances, bc to be frank it's all obviously quite stupid...
https://freddiedeboer.substack.com/p/we-are-still-living-in-the-long-boring
As they say, intelligence is a necessary but insufficient.
Spoken like a true logician
thank you! I have a lot of data points from medicine
What do you mean by "not solvable by intelligence"? I tend to think of intelligence as "the thing that humans have that chimpanzees don't, that allowed humans to bootstrap themselves up from being naked on the savannah to walking on the moon."
The fact that we can put a man on the moon while many of the world’s richest places are full of traffic jams and homeless people kind of supports the point of the article, though, doesn’t it?
When incentives, organizational structure, and so forth, are right, we can put a man on the moon. If they’re not, we end up with tent encampments in Santa Monica 50+ years later.
I don’t think it’s our intelligence which got us to the moon. It’s the ability to record information about what works and what does not, make mistakes, then iterate on what works and enormous number of times.
My view is that we are successful because we opened up a new evolutionary space, which we believed we planned and created through our genius. In reality, I think genius has been vastly, vastly overrated, just as “God’s design” was in creating a toad or a frog. It’s all evolutionary processes, and they don’t all tend towards progress. They can get us to the moon and can make getting to the moon much harder; there is no contradiction there
I agree with your larger point, but in fairness "he ability to record information about what works and what does not, make mistakes, then iterate on what works an enormous number of times" sounds a lot like intelligence.
It's possible to be highly intelligent without having access to good material, or institutional structures, for keeping long-term records. That's why there's so much variability in how much we know about different historical periods.
Intelligence, yes, but also opposable thumbs and oxygen-rich atmosphere. I wouldn’t be shocked if whales turned out to be as intelligent as Neanderthals, but they aren’t going to the moon, not by their own will and effort in any case.
They certainly had accurate data on overall giant squid population density long before we did, and there's a case to be made that exploring the deep ocean is in many ways a harder technical problem than going to space.
One of the things. Individual intelligence plus an aptitude for collective action, which chimps also have. (Also, you know, a nearby moon and stuff, but let's restrict this to circumstances that a clearly properties of a species.)
Cockroach #1: What are they doing?
Cockroach #2: Building a rocket to fly to the moon.
Cockroach #1: Why would they do that? We can't even survive there.
In this case, it means the problem isn't that we don't know how to build something, it's that someone else is preventing us from building it. The problem isn't that we can't think of new drugs to try or that we don't know how to synthesize them. It's that we aren't allowed to test them in a fast and inexpensive manner because other people are stopping us.
I would also say that even the experimental data to test these in vitro is much harder to get than what software people are used to....
So this is before even considering getting this into humans...
One solution to that particular problem is regulatory arbitrage - test your magic cure for cancer somewhere with more permissive regulators, such as ::insert country here:: and then blame the FDA for why people need to fly to ::insert country here:: to get your obviously effective magic cure for cancer. :P
https://ifanyonebuildsit.com/6/wont-ais-be-limited-by-their-ability-to-design-and-run-experiments
The tl;dr version: with enough intelligence, you can make better use of the data you already have, and figure out ways to get more and better data out of whatever capacity for running experiments that you do have.
The scenario: you have the combined brainpower of the entire nation of South Korea sitting in your data center, running at 10,000x real time, trying to figure out how to work around the fact that doing experiments in the real world is too damn slow. Do you think the entire nation of South Korea, running at 10,000x real time, working together on the problem of "how do we work around US government restrictions on biological experiments in order to get the data we need to cure cancer forever as soon as possible" and nothing else, won't be able to think up a solution?
There's an old webcomic that parodies the "enhance" button from police procedurals where a computer somehow turns a grainy photo into a clear one. It features someone trying to "enhance" a photo consisting of exactly four pixels. Maybe 10,000x speed South Korea will figure out a way to make better use of data, or maybe it will be like trying to get an HD image out of a four pixel photo. It's hard to say ahead of time.
Fantastic piece! I don't know how you managed to fit in the difficulty of clinical trials and the culture of San Francisco into one coherent argument, but I really enjoyed reading this.
thank you!
If I may indulge in a bit of speculative psychoanalysis: it seems natural to me that the folks in the AI sector think that superintelligence will have real-world superpowers, because it resonates unusually well with their own experience. Their own right-tail-for-humans intelligence has brought them wealth, power, and status well above that of almost everyone else. Of course they will intuitively assume that lots more intelligence will bring lots more wealth, power, and status, and of course they will overlook the boring non-intelligence-based bottlenecks which have always been especially boring and especially non-salient to them.
I think it's because of analogy. From the point of view of a dog or a chimp, we do have superpowers. We can do things that they can't even comprehend. We can kill by pointing a tube at something and pulling a trigger. We can let huge wheeled machines swallow us and spit us out at a far away destination. It isn't hard to imagine a superhuman AI might be just as superpowered to us as we are to chimps and dogs.
The issue, of course, is that it isn't just our intelligence that allows us to gain these powers. It's the system of cooperative production that we use to make the tools that our intelligence conceives of. A human on a desert island can probably figure out how to kill a chimp or a dog, but it will be much harder and riskier for them.
This take tallies with my experience working in tech dept of non-tech large orgs. The average coder is inclined by their experience to see the biggest (best?) problems as informational.
They live in information rich environments and not only do they gain wealth from their ability to navigate that, but also that wealth then obscures the fact that physical and social factors are the primary issues that holds back most people's flourishing.
When the basics take up most of your paycheck you're more inclined to think of the basics as the primary bottleneck.
I do the big boring stuff and many smart people have asked me why I prefer that grind instead of more exciting startup stuff (which IMO mostly does very little for the life of the average joe). It's a completely different understanding of what the problem is, let alone the solution.
Well, for a bit more pop psych: the weathy tech sector is rife with nerds whose idea of great literature is sci fi. So of course they are jazzed for
intelligent talking machines, a genre mainstay since its golden age.
I don't think this is because of wealth or power, at all. I think it's because bright nerdy types are at a big disadvantage in conventional status ladders (i.e. they're nerdy). So they instead concentrate all their personal effort and ego investment on the intelligence status ladder, and as a result become blind to other virtues.
(I would even say that being a right-tail software engineer is of no real status value, except within STEM/computer/Internet communities.)
Here's the relevant analogy:
You have the collective knowledge and brainpower of entire nation of South Korea sitting in your data center, running at 10,000 real time, with no need to eat or sleep, and completely dedicated to solving whatever problem it decides to work on. Would it make sense to consider that a data center like that would have real-world superpowers?
Actually, probably not! Because "completely dedicated to solving whatever problem it decides to work on" is smuggling in some unwarranted assumptions-- first and foremost, the assumption that there exist materially important real-world problems *that are solvable by pure thought*.
"Satoshi Nakamoto" became a billionaire by publishing a math paper. :P
Why would we assume that a data center is limited to "pure thought"?
You can do lots these days by flipping bits.
Problems solvable by pure thought? Such as, say, everything Einstein accomplished in theoretical physics? Every result in mathematics ever? Figuring out whether a proposed machine design is going to work, given descriptions of materials and the laws of physics? Taking a list of 140,000 protein structures and turning it into a solution to the protein folding problem?
All of which had to be verified by experiment, and none of which ever conferred on Einstein himself any superpowers whatsoever, nor on anyone else till it was followed up by decades of tough, incremental, experiment-heavy engineering work.
expertise in one subject does not confer expertise to others. once again, econ101 goes underrated
So this piece was brought into inception by experience (likely umpteenth time) of being mansplained to by typical overconfident but ignorant AI/AGI (likely also crypto) booster lol
I guess you can put it this way hahhaha
Does quality of governance scale with intelligence? e.g. Clearly there is some level of intelligence below which you are unable to keep track of basic facts. Maybe IQ of ~80? That's the minimum IQ allowed to serve in the US Military so it's a fair floor. If everyone in the USDA had an IQ of 80, would we expect drug approval rates to go up or down?
This is a good point and I certainly think that:
(A) as a broader point, governance itself will be impacted by AI in various ways (including for example, more autistic people making money from AI and donating to very specific political causes as a result).
(B) More specifically to your point, I think the intelligence of a population involved in governance changes governance. That being said, I am not sure exactly how this will play with AI systems, as they are currently more like disembodied forms of intelligence. I think as long as humans are in control there will be a lot of friction in terms of diffusing machine intelligence into humans in a way that humans want to follow. What I mean by that, is that the human motivational structure is tied to the IQ of the human, imo. Even if a human knows from an AI that they should want X thing, I am not sure that will hit their core motivational structure in a way that will make the human want to advocate for said thing.
my response to this person asking you why you do what you do would've been, "well, what else are you going to use for your training data?"
Ruxandra, this was an exceptional piece and one of the best IMO I can remember you having written. Just totally awesome.
In particular I appreciate your emphasis on solving ‘quotidian’-seeming-but-equivalently-existential societal challenges; on the circumstance-specific adaptiveness (but also maladaptiveness) of certain orientations, which we see also in VC; and on the fundamental fact that we exist in a world defined first and foremost by institutions and their highly idiosyncratic behavior.
I am so hype that you are able now to pour your full attention into writing about this stuff.
thank you, that's very kind
Important piece to consider. Nicely written too!
Regulation and whatnot get in the way of many things. We get in our own way due to greed, malice, and competition. It is likely that Healthcare and Medicine will be bottlenecked by those, directly. It is also likely that the entire approach to medicine is itself 'messed up'. It treats symptoms, not identifying causes and prevention. It looks for single-pill cures, not systemic life-changes. Until healthcare becomes personalized health-CREATION, the entire strategy to improve human life is fallible. AI can help in many domains, especially at the personalized level. Given AI is already accelerating the health of a number of people, it seems likely that a shift may occur where personalized doctors help to identify and prevent issues for individual people before they become 10-100x more expensive to handle.
One other reason that medicine may prove less tractable to AI than its most evangelical adherents believe, is that many diseases remain poorly understood. I accept that AI could design a superior antibody against amyloid but that improvement may correlate quite poorly with genuine clinical enhancement (to take one example). Likewise Parkinson’s Disease.
As you rightly suggest, clinical trials remain highly primitive and while AI will help with both the operational aspects and patient selection, there is a finite limit in the time improvement that such an optimized approach could yield. And that’s before we talk about reimbursement.
One final thought; having watched “the thinking game”, many of the most vociferous AI researchers are simultaneously exceptionally bright but very narrow in their knowledge and world view.
Yes, I agree. But the real confusion is kinda happening a bit before. These people fundamentally imagine that an AI will essentially be computationally omniscient -- ie have the ability to perform any computation virtually instantly -- and yes, if that was true AI would basically be a demigod.
And this goes to a deeper confusion about what is meant by intelligence. If intelligence is the ability to infer the answer to something like mathematical problems then *enough* intelligence let's one do anything. But that's not a kind of intelligent that AI could actually be. There is only so efficient the an algorithm for a problem can get.
Yes, an AI equipped with an infinitely fast computer might be a demi-god -- they could just consider all possible scenarios consistent with their observations and run them forward to maximize their expected outcome -- but so would I.
OTOH if you want intelligence to mean something about the software not just raw computational power all the sudden there is no particular reason that AI's should be particularly amazing. Hell we may already be near the algorithmic efficiency limits for the kind of things we care about. And for abilities like persuasion there is just no reason to believe the AI will even be able to overcome disadvantages like people knowing they are an AI or not being able to be a charming human and trigger all our evolved social reactions.
You might be surprised how persuasive even humans can be over a text chat.
https://www.lesswrong.com/posts/fbekxBfgvfc7pmnzB/how-to-win-the-ai-box-experiment-sometimes
The "AI box experiment": You talk to someone over a text chat that role-plays an AI, and you role-play someone authorized to let the AI "out of the box" and into the world. If you spend a few hours talking to them and haven't agreed to let the AI out of the box, the human roleplaying the AI will give you $20.
Do you think you could be convinced to let the AI out of the box? The success rate of people playing the AI has been about 50%, I think...
How exactly does that relate to what I said? Yes, people can be pretty persuasive. We know that even very persuasive people face huge challenges in getting large institutions designed to make it hard to get approval quickly give that approval *quickly*. For this not to be the barrier you need so much persuasion you convince people to fundamentally change the system.
---
But on to the whole box thing my first response is so what? I don't see why an AI would be so much more dangerous than a smart religious nutter person nor why alignment should be much harder than it is for people. If it's no worse than a really bad terrorists escaping prison ¯\_(ツ)_/¯.
But more on point, that's why we have multiple layers of approval to do many things. You just stack barriers until probability drops. So far no terrorists have social engineered their way into a nuclear launch.
And sure that wouldn't work if you presume AI is way way beyond human persuasion levels and that it usually acts like a ideal bond villain secretly plotting to escape and that it's a disaster if it does but not clear why any of those should be so.
In other words, intelligence is not a bottleneck, the bottleneck is solving coordination problems, such as how to agree on a better legislation or a better patent system.
And it's not evident whether a society made only of AGIs would be necessarily better at solving coordination problems that one made entirely of humans. (Insert some stuff about game theory, Moloch and all that here...)
But given that we are better at solving coordination problems than chimpanzees (we have laws and states and whatnot, after all) is a case for a mild optimism.
Unless, of course, one would argue that humans fare better than chimpanzees not because of their intelligence, but because of some other aspect of human biology/psychology/etology.
One aspect I wonder about is whether the bottleneck is even broader than regulation or governance.
We tend to treat words, data, models and biomarkers as if they simply represent reality, when in practice they also shape what becomes visible, actionable and worthy of attention. In that sense, style, thinking, speech and action are not merely expressions of an ontology; they enact it—and in turn participate in its continual reconstitution.
The challenge is therefore not only improving our representations, but remaining alert to what they inevitably leave out. This is to 'hold' them lightly and tentatively. Otherwise there is a subtle drift whereby the map progressively becomes the territory, and our interventions become directed by the representation rather than the phenomenon itself.
Thank you for a nuanced contribution to an increasingly polarised discussion.
Thank you for this. Excellent article.
In the realms of science and engineering, a lack of data -- actual, real data from the world of atoms and organisms -- will be a major bottleneck for the foreseeable future. AGI that is 10,000 times smarter than a human (whatever that even means) will not change this fact. In fact, as AI eliminates bottlenecks in the realm of bits and cognition, those chokepoints in the realm of atoms will become even more acute, and hopefully more evident to the currently oblivious AI boomers.
The good news is that it is by recognizing and eliminating bottlenecks that savvy entrepreneurs build thriving companies and become rich. So new, or newly apparent, bottlenecks are a fertile field for builders. The bad news is that if the legal and regulatory environment does not permit freedom of action in a given domain, this won't happen. So the ultimate bottleneck in the realm of medicine is likely to be regulation, as you noted here.
In a way, it's really funny that you heard this in San Francisco, a place where the obvious bottleneck is the political dysfunction which makes it impossible to build housing there, a technology which has existed in some forms since antiquity.
"You mean more hyperpersuasive than you? No. That's IMPOSSIBLE!"