A lot of this piece is about how many doomers make poor arguments and overstate the consensus and risks. I very much agree, and I think a lot of the soldier mindset and focus on unworkable solutions I seen in AI risk discourse reminds me of some folks with another existential anxiety, climate change.
That being said I do think that AI is the most consequential issue of our time and does pose serious existential risks, just on longer timelines than most doomers.
One of the biggest biases of a certain kind of Rationalist is that they think and talk a lot about things they find interesting, then argue that those things are as important as they are interesting. We have bigger, more pressing problems than Paperclip Machines, but they aren't as neato to talk about.
I'm not where you were, for one I don't get anxiety over things I can't control so that's helpful. But I am concerned and looking for reasons to be more optimistic.
However I have utterly failed to be convinced by any of LeCun's argument, who's generally eluding points and consistently being proven wrong on specific claims.
Besides "they're full of shit on biorisk!" which is an extremely narrow claim, although it should inform our priors somewhat, what have you found in either LeCun or Recht discourse that you found so compelling?
Because from what you write I understand "well the risk is not imminent so we're good", but capabilities are ratcheting up.
We’re all gonna die, Forrest. One way or another. Nothing ventured and all that. We have yet to see a doomer prediction come true in all of human history. Will unexpected bad shit happen? Yeah, certainly. Will we be OK anyway? Yeah, probably.
If you define "doom" as "human extinction", then obviously no human could ever say "we have yet to see a doomer prediction come true." But is human extinction possible? Yes.
Yup. So? My two favorites are the Jellystone caldera going pop and a GRB that’s not off by just a degree (like the one that microwaved the Earth in the Precambrian). By comparison to real extinction level events we see in the geologic record, I’m terribly unworried about SkyNet. I have faith in John Connor. 🤖✊
Okay. I maintain that it would be a good idea to develop AI safety methods before trying to develop superintelligent machines.
Your solution of "trusting John Connor" doesn't convince me, because you're referring to a movie character that can time travel, which is not apparently possible given the known laws of physics.
There's no evidence that any proposed "safety methods" will actually result in anything safer. They will definitely impede anyone else from entering the game who might raise the bar for the soon-to-be-entrenched incumbents. I'm in favor of simple accountability, but I think that should be applied universally to all software, not just AI. The imposition of standards of care for all software companies, executives, and practitioners is the only reasonable path. Then hammer the accelerator.
Doomers also have a gross misunderstanding of the supply chain required to support the pathetic excuses we have for AGI now and the massive increases required to get closer. The energy suck and fragile hardware supply chain required to keep what little progress we've made functional would be destroyed by anything that's is remotely damaging to humans, let alone any existential impact. I work at chip manufacturer with a primary focus on the AI droplets we produce and I have a passing familiarity other clouds' internals (having worked elsewhere). It's a minor miracle that models get trained and utilized as much as they do today.
We are all John Connor. And John Galt. Also, try playing the game "Everyone is John" it's fun. Doomers seem to be lacking in the fun department.
Geological X risk events happen once every 100M years and X risk scale GRB hits are estimated to happen once every 10B years (longer than Earth will remain habitable). These are if anything events that popular culture vastly overinflates relative to real risk levels, though understandably so (asteroids, eruptions, tsunamis, etc. have something of the sublime about them).
So? You’re saying we’re overdue? OK. Which one is better prepared to mitigate the meteor risk before it becomes a meteorite extinction: humanity alone or humanity with AGI? It’s OK. I’ll wait while you google the math.
Hey Ruxandra, I'm a big fan of your blog. Your beef with the doomers makes me sad, because in my mind, it is the biggest thing you're getting wrong.
But first I'm going to say: I'm sorry to hear about your panic period. I don't advocate for panic -- it doesn't sound pleasant, and doesn't seem all that helpful.
Nevertheless, it's important to have a realistic assessment of risks.
* That said, your emphasis on expert opinion and "empirical data" seems somewhat misplaced to me. I remember in the early days of the COVID-19 pandemic, here in the US, experts were trying to reassure us by saying there is "no evidence of community spread in the United States". I didn't find that particularly reassuring, because I felt there were strong *theoretical* reasons to expect community spread before too long. I feel similarly regarding AI doom.
* AI experts mostly have expertise in incrementally improving existing AI systems. They aren't very incentivized to be good at *forecasting* the development of AI.
* More importantly, solving alignment-style challenges with existing AI systems is playing on easy mode. If something goes wrong, it's relatively straightforward to identify and fix the problem. With superhuman AI systems, I expect new problems to crop up. They'll be harder to identify, and quite possibly impossible to fix. I'm an ML engineer and I've spent hundreds of hours thinking about this on a fairly technical level, but I'll offer a basic analogy: Imagine you tried to fool your cat, and you were pretty successful. You might conclude from that experience that "it's easy for animals to fool each other". That's not the right conclusion: If your *cat* tried to fool *you*, it's not going to succeed for very long. In some sense, I worry that AI researchers may be developing "anti-expertise": Current AI models are causing them to develop intuitions which may not transfer to the superhuman scenario.
* In terms of biorisk -- most doomers aren't biologists. In the Less Wrong survey, you can see the userbase has around 8x as many people who work in AI as people who work in biology: https://www.lesswrong.com/posts/WRaq4SzxhunLoFKCs/2023-survey-results#III__Work_and_Education Among prolific posters I would guess the ratio is much more skewed. I'm not sure we can conclude that they're wrong about AI based on them being wrong about biology.
The AI doom conversation is well over a decade old at this point. The panic you see on social media is just the tip of the iceberg. The submerged part is many thousands of words: papers, books, blog posts. I've recently been telling the doomers that it might be a mistake to abandon their argumentative standards to stir up panic. I think your blog post is evidence that I'm right.
Michael Huemer is a smart guy. But based on the post you linked, he hasn't engaged with the AI risk literature (the submerged part of the iceberg) in much depth.
E.g. Huemer writes: "If we make a conscious machine, we’ll probably design it to be much more benevolent than us." Doomers agree this is a good idea! They just think it will be really hard. A big portion of the AI risk literature is devoted to discussing and debating this. Here's a recent summary from Eliezer re: why he thinks it's super hard https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities
Similarly, in section 4.2 Huemer talks about Ted Kaczynski trying to escape from prison. This is another topic the doomers have discussed amongst themselves in fair depth. Believe it or not, Yudkowsky actually won in a couple of simulations, playing as "Kaczynski" trying to talk his way out of a "prison". Click the "(Read More)" link here for a high-level summary: https://www.lesswrong.com/tag/ai-boxing-containment
As I said, I like your blogging and I'm happy to go back and forth for a bit if you reply to this comment. My overall point is there's more to AI doomerism than meets the eye. Eliezer has some serious character flaws, but having character flaws doesn't mean you're wrong about everything.
I have engaged with the "bottom of the iceberg". This was not a post aimed to comprehensively debunk all the AI arguments. The AI in bio is just something that opened my eyes when I was panicked, not the only thing I took as an argument.
Many, many times through many conversations. I am left unsatisfied. To me it just does not seem convincing. I do not think current LLMs are going to lead us to a superintelligence. I think EY & co are making sort of category errors and their framing is wrong.
I will write a longer separate blog post about my view on this whole thing.
Well you could've fooled me, linking to that Huemer post ;-)
>Many, many times through many conversations. I am left unsatisfied. To me it just does not seem convincing. I do not think current LLMs are going to lead us to a superintelligence. I think EY & co are making sort of category errors and their framing is wrong.
I'm an AI alignment researcher and I often find Eliezer's arguments unsatisfying. He seems really overconfident. Many other alignment researchers feel the same way. On Less Wrong a recent upvoted post stated: "The weird thing here is that [Eliezer's] >90% doom disagrees with almost everyone else who thinks seriously about AGI risk." https://www.lesswrong.com/posts/ye78Dip8YNgLBKGcy/seth-herd-s-shortform?commentId=FpdvoZsmmrNLekkz9
The key thing for me is, I think the burden of proof should be on the people building advanced AIs to show they will be safe, in advance of creating them.
EY fails to *prove* that the AI will be unsafe. But attempting to build superhuman systems without much of a safety plan, like OpenAI is trying to do, is inherently a *very* sketchy activity -- *significantly* sketchier than placing the first, gigantic prototype of your new, non-peer-reviewed nuclear power plant design right in the center of Manhattan. You say you're anti-safetyist, but I presume you aren't insane ;-)
If you build a bridge that's gonna take heavy traffic, the burden of proof should be on *you* to show that the bridge will hold up. There are various practices that have arisen in fields like aviation and civil engineering based on decades of failures and post mortems. As an alignment researcher, I spend my time reading about that stuff. That sort of safety culture doesn't exist to nearly the same degree in the world of AI, and the stakes are much higher because it's not just people on the bridge who could die, it's everyone. Plus, we aren't going to get a chance to learn from decades of failure, at least not in the same way -- as I stated, superhuman systems are likely to present qualitatively different failures. So the risk/reward of the current AI capabilities charge looks unfavorable.
Anyway I'll keep an eye out for your post and try to comment. You can also email me a draft at coasterchatter11 on gee-mail and I can offer pre-publication feedback.
I find Huemer's post better than the counterarguments.
I think getting lost into the weeds of this stuff can make one forget about simpler and fundamentally correct arguments.
The other thing is -- if AI Safety ppl do not consider EY to be accurate, they should distance themselves from them. It is not the outsider's responsibility to go through thousands of blog posts to disentangle who says what. As it currently stands, a lot of important figures seem to be influenced by these ideas, so I consider them an integral part of the movement.
I also fundamentally disagree with this approach to technology -- where one needs to prove future iterations of the tech are safe. At the moment I think chatGPT is pretty safe by normal tech standards. If future models become more dangerous, that should be judged at the time, not now.
I don't see much of a "simple and fundamentally correct" argument in this paragraph from Huemer:
>Reply: [Human immorality is] a fair point [for why AIs will be immoral]. So we’d better not model the AI on ourselves—at least not the average human; perhaps we could design a computer to imitate the best people we know. If we make a conscious machine, we’ll probably design it to be much more benevolent than us. E.g., if we figure out the basis for emotions, perhaps we’d make the AI love us.
It really seems like he's handwaving over the core issue, of how difficult it will be to make a benevolent AI? That said, I only skimmed his post; maybe there's something I missed which I addresses that core question elsewhere?
>The other thing is -- if AI Safety ppl do not consider EY to be accurate, they should distance themselves from them.
The AI alignment community is much bigger than just MIRI. MIRI is unusual since it's focused primarily on public communications and politics. Most alignment researchers have their heads down writing papers and running experiments. See https://www.alignmentforum.org/ for a feed of recent research. It's certainly fair to say that MIRI's ideas have been influential, however.
>I also fundamentally disagree with this approach to technology -- where one needs to prove future iterations of the tech are safe.
I doubt this is your actual position. E.g. if Boeing were to introduce a new jumbo jet, I'll bet you would want someone, somewhere to have good reasons to believe that it won't crash and tragically kill all its passengers. If the main design priority was trying to recreate a child's sketch, I doubt you would want to step on that plane.
Maybe it would be useful to discuss the power plant hypothetical I brought up earlier: a gigantic nuclear power plant prototype right in the middle of Manhattan, which hasn't received any sort of review from regulators or engineers outside of a small design team. We can subdivide the disagreement into two questions: (1) is this power plant hypothetical a reasonable way to deploy nuclear technology, and (2) how does the power plant hypothetical compare with advanced AI development.
You may be correct that people are typically too risk-averse, but it's possible to take things too far in the opposite direction. If your house is too cold in the winter, that doesn't mean you want to set it on fire.
>At the moment I think chatGPT is pretty safe by normal tech standards. If future models become more dangerous, that should be judged at the time, not now.
And then what? Supposing you were to judge a future system as dangerous -- what would an appropriate response be at that point? I suspect by that time, the cat would be out of the bag. We're lucky that nuclear weapons can't be built by the average Joe, for physics-related reasons. Are you confident that it would even be feasible to restrict this hypothetical dangerous AI system, after everyone learns that it's possible to build?
BTW, I recommend reading up on the history of nuclear close calls if you haven't already. See the stories of Stanislav Petrov or Vasily Arkhipov, for example.
You say you're fairly familiar with AI risk discussion. Can you tell me what you remember reading about "hard takeoff" and "soft takeoff"? Presumably you can see how in a world with hard takeoff, your suggestion to wait for clear danger would likely fail. So -- do you have specific reasons why you think hard takeoff can be ruled out? How about AI deception (e.g. the so-called "treacherous turn"), where unfriendly AIs masquerade as helpful and friendly, because they're smart enough to realize that's a better way to achieve their counter-human goals in the longer term? That could also defeat the "wait for clear danger before shutting it off" approach.
Maybe it'd be helpful if you mentioned the specific AI risk intro you read. I feel like most good intros should've covered the topics I'm bringing up.
That’s fair. That’s the key question around an AI pause for me. I’m less concerned about scaling per se, and more concerned about preventing that sort of big improvement in architecture which could lead to hard takeoff. But it’s not clear how best to accomplish that from a policy perspective.
my bias is that AI X-risk is a nothin’ burger but bc it flatters high IQ people, they worry about it. The philosophy chooses the apocalypse that nerds can at least take credit for. It’s a fantasy too if all the dumb normies of the earth (AND ones’ more successful peers) will be wiped into valueless statusless nothing or gruesomely purged. More or less don’t want to understand that bc it’s pretty boring
But the refined tastes that make natalism a hot topic, 🤔 hmmm interesting
(not really) The RW journo sphere accidentally caring about something real is pretty cute. But their reasons are probably uninteresting. My sense is that it’s the perfect whatabouttist retort to climate. But infinitely better than trying to legislate climate. Because instead of having to go after the titans of industry it’s the fault of DINKs, ordinary infecund proles (and women, of course).
What were your primary concerns as a "doomer"? Was it more "the world is going to end" or more "my work will be obsolete," and therefore what is the point of existence?
And besides following new people on Twitter, like Yann, is there anything else that helped you overcome this fear?
It varied. I mostly oscillated between I will be completely obsolete no matter what I do other than smth super physical to AI will enable super catastrophe.
I mean I also talked to eg Ben in DMs quite a bit. I paid attention to what chatgpt could actually do -- and whether the claims abt its intelligence at the time were warranted.
For every claim I encountered I started to apply more critical thinking.
ChatGPT increases biorisk. Ok, what if we throughly abt all the biorisk before the internet was created? After all, it gave access to unlimited info to so many ppl. Analysed papers discussing biorisk. Most of which underestimate how hard biology is in practice
The AI doom scenario is concerned about future AI systems with human-level or above intelligence, which Yann LeCun also believes will come.
ChatGPT does not increase biorisk and there is no loss-of-control risk.
Also important to keep in mind that it called risk for reason, i.e. there is a chance of losing control to future smarter-than-human AI systems. We should take that risk seriously and try to mitigate it. Yann LeCun for some reason believes that is guaranteed that we will succeed with this very daunting task of controlling more intelligent (i.e. more powerful) systems on an unlimited time horizon.
Given the enormous progress we have made in recent years and the amount of resources (money, manpower) flowing in, I don't think it makes much sense to make confident predictions even on a 5-year horizon.
Generally, I think Yann LeCun is way too confident with his predictions. Like everybody else, he doesn't have a crystal ball where he can see the future.
That's true. Many experts in the field think it is much closer though. Predictions about the future are inherently speculative. I think the next 5 years will really clear things up, one way or the other.
What makes me a bit uneasy that we are already in the uncanny valley for AI intelligence. We judge AI by human standards. This is IMO just another indication how close we have come.
Plus the enormous amounts of money and manpower that are flowing into AI research.
I'm a software engineer working closely with AI and I'm fully convinced that AI doomerism is a marketing scam to make a toy look far more powerful and capable than it really is.
That's not very plausible, since these ideas have been circulating ever since the emergence of computers. They did not arise or emerge from marketing campaigns. (By the way, which marketing campaigns promulgate the idea that AI pose a threat to the species? Can you produce an example?)
What's more, the mere fact the industry benefits from this position doesn't, by itself, show that they promulgate or encourage it. Not to mention that you attribute to the industry a clearly flawed cost-benefit analysis. It would be totally irrational for them to make these claims. To the contrary, it is far more likely that companies will overstate their ability to develop AI safely, since it is in their financial interest to self-regulate, as is the case for most industries.
It's hard for me to agree or disagree with this unless you are willing to give a probability that that world is destroyed by A.I [p(doom)]. Did you go from 99% to 1%? Or something else?
I'm genuinely sorry to hear that you went through such a stressful time during the release of GPT-4. I do agree that there was something close to a moral panic that happened then. I feel mixed about whether it was net-positive or net-negative. On the one hand, it was nice for AI Safety to finally enter the Overton Window. On the other hand, it didn't feel like the best advocates for AI Safety were getting the most airtime. Yudkowsky, while certainly an important figure in the history of the movement, is--to be blunt--past his prime. He offered little in the way of arguments and much in the way of flailing his arms around and screaming "We are all gonna die!" This did a good job getting the more sclerotic institutions in our society to finally wake up, but it damaged the credibility of AI Safety in the eyes of people who care more about the substantive strength of the arguments and not just vibes.
And a problem is that a lot of people thought that the doomers were talking about LLMs themselves being dangerous. But what AI Safety people were really reacting to was the apparent speed-up in capabilities (their "timelines"). But that wasn't always clear when people were retweeting spicy AI Safety memes. This meant, a year later, when LLMs continue to make the same stupid hallucinatory mistakes, all the sky-is-falling rhetoric looks like fear-mongering.
I would plead with you to give AI Safety another chance. While I'm sure you've done some reading, it's not clear to me just from this short post whether you've found what is (in my opinion) the really good stuff.
Two places I would recommend starting are Paul Christiano's work and Chris Olah's work.
Christiano has a more moderate view compared to Yudkowsky. He also has technical expertise that Yudkowsky lacks: he was iirc one of the creaters of reinforcement learning by human feedback (RLHF). He still thinks that AI constitutes a serious risk, but his scenarios are more interesting and "systemy" than the more super-smart-agent-takes-over-the-world scenarios that Yudkowsky tends to emphasize. He's also a very important person to know going forward. He recently got a bigshot position in the US Government to work on AI Safety.
Chris Olah is a researcher at Anthropic, one of the leading AI companies (their main LLM is Claude 3). He is known for mechanistic interpretability, a subfield of AI Safety that seeks to learn how to decode the neuron weights of the neural network in order to understand what the AI is thinking. My impression is that mech interp is the most respected research direction in AI Safety currently (and I am personally favorable to mechanistic interpretability as someone who has chatted casually about it with acquaintances in a non-technical way).
Well I'm not sure why LLMs becoming better made people react the way they did. We should expect an improvement in one part of the technology to happen fast but that doesn't tell us we're "taking off" towards human intelligence. And many many people absolutely argued LLMs -> AGI
My baseline is the 0.35% chance of doom by 2100 established by a group of superforecasters at the Foresight Institute. Anyone whose risk guess is much higher than that should present a comprehensive argument for why all the superforecasters are wrong. I’ve seen precisely no one engage with that figure seriously. Ergo, doomers are LARPing.
Risk of losing control to smarter-than-human AI systems: 25%
Together: 80%*25% = 20%
It is really not rocket science. To even attempt to control smarter-than-human AI systems feels like hubris. How could anyone suggest that this is an easy task?
Yudkowsky has alot to answer for. Not because he believes something wrong or which I disagree with but because he privleges stories over precisce definitions and -- in a way disturbingly reminiscent of the religious establishment of an earlier day -- not only equates lack of disproof with evidence for his claim but also refuses to admit the weaknesses in his argument.
Look, I don't agree with Bostrom, but he advanced an interesting argument about AGI and actually tried to fill in the holes. What should have happened after he published his arguments is for people to sit down and try and flesh out the concepts he used such as intelligence and tried to clarify the reason why we should expect AGI to act to maximize a simple goal outside of the area it was trained on. Instead, what we got was a pile of steaming crap that amounts to nothing but parables masquerading as arguments.
I mean the whole self-improvement FOOM argument is totally unjustified once you think about it quantatatively or from a mathematical POV. The idea is supposed to be that since we can build an AI smarter than us it will be able to build an even smarter AI and this explodes to infinity. Ok, but the second you try to quantify that it no longer works -- I mean why assume that the time for each unit of intelligence increase is decreasing (much less towards 0) not increasing? And what even are these units? (Correct choice of units can makes it go to infinity or asymptote whatever is happening).
Which brings us to all of those parables that equate intelligence with omniscience -- as if AI somehow wouldn't be bound by the underlying mathematics of computational complexity. Once you realize that it can't, all those little stories about how AI can escape safety measures start to sound like nothing more than modern recastings of stories about genies and lamps. But, unlike the many responsible people in this area, Yudkowsky treats it as the burden of the person who isn't a doomer to disprove this could happen -- effectively saying that lack of knowledge about AI is reason to be confident they'll kill us not reason to say we don't know.
Ultimately, it's still an interesting argument that's worth discussing (even if I don't find it very plausible) but doing so responsibly means demanding arguments be filled in with more precisce language and treating the failure to do so as evidence against them not assuming the things that feel salient when we consider narratives about it are presumptively true and others have the burden of disproof.
I agree that AI is presumably bound by the mathematics of computational complexity, but they might allow a lot more that one would think - simulating protein folding is most definitely one of those computationally intractable problems, but AlphaFold actually does work.
Yes, and I'd welcome work trying to fill in Bostrom's argument here. But Yudkowsky's parables don't help here.
Maybe an advanced AI would be capable of many things but it seems pretty unlikely that it would have the ability to do psychohistory a la foundation and with high probability manipulate it's way out of any set of limitations we might impose.
Indeed, I think Yudkowsky may have done the most out of anyone to increase our risk from unaligned AI because he has simultaneously convinced one group of people that this AI alignment stuff is just non-serious bullshit while convincing another group of people that there is no point in trying to ask for limitations like only allowing AIs to advise people (in contrast to giving it direct control of things in the world) because of his little parables which describe how a computationally unbound agent could manipulate people to achieve whatever it wants anyway.
If there is no significant risk from AI no big deal. But it's people who think we should take that possibility seriously who should be most upset at Yudkowsky. He has effectively made sure people don't pay attention to the kind of practical and perhaps very serious concerns about how to manage AI in the real world by pulling all the attention towards these unrealistic scenarios.
Freaking out is for sure unhelpful. I'm not worried about AI because I don't think our society has the technical juice to get there from here. However, if we could and did, Yudkowsky would be proven right, and Yann wrong. There's no need to argue with his central arguments in order to avoid anxiety.
If astronomers found that a big comet, the size of the one that killed the dinosaurs, was on a direct course to hit Earth, but the collision was going to happen five hundred years from now, would you be concerned and think we should start investing in comet deflection technology right away, or would you just shrug and leave future problems for the future to solve?
There's a general pattern here that's useful to learn from - intellectuals who talk about exponential growth, the rapid rate of change or even the derivative of the rate of change have a very long history of making invalid predictions. I think it's a psychological problem, in which talking about derivatives makes you sound much smarter than other people and therefore there's always a huge temptation to appear more attuned to small changes than everyone else.
A recent example of this was of course covid, in which many people claimed that viruses grow exponentially, even though this is trivially untrue and anyone can see it just by looking at a graph of cases in any previous epidemic. Indeed the founding discovery of epidemiology was that epidemics follow a bell curve like distribution with an early period of rapid growth that then later peters out well before every susceptible person has been infected. It was very apparent to me from reading comments and taking part in discussions at this time that a lot of educated people were in love with the idea that understanding exponential growth made them somehow special. Any attempt to point out basic things, would be met by eye rolling and sneering about how people that don't understand exponentials are dangerous.
The AI risk debate seems to have very similar characteristics, in which people make arguments not based on what is actually possible today but based on an extrapolated rate of growth that they claim will massively speed up in future. We don't necessarily need to engage directly with such arguments, it's often sufficient to merely pattern match it to faulty ways of thinking in the past.
Mmm, my experience has been that they don't like to say this. I guess it would concede the point that they're in dangerous territory. What people say instead is some variant of "You just don't understand the exponential function" or "there's a consensus of experts that you're wrong".
Oh, and of course we must not forget "that other prediction wasn't actually wrong, it just hasn't happened yet!"
Well, in the short run at least, populations of living things - including viruses - do tend to grow exponentially until something stops them. Which is one reason why invasive species can become huge problems.
I was trying to imply that the "short run" period of exponential growth can still last long enough to make a small problem into a really, really big one in what looks like a very short amount of time.
There's an old riddle: a patch of lilypads doubles in size every day. On day 30, it fills the entire lake. How many days does it take to fill half the lake? The answer is, of course, 29 days - it only takes one doubling to go from half the lake to all of the lake. By the time something that grows exponentially-for-now is a big enough problem to notice, it's often too late to stop it from becoming too big to fix.
In theory, yes. In practice this hardly ever happens. Even in cases where there are invasive species due to importation and lack of local predators, you don't get grey goo scenarios and the ecosystem eventually settles down.
And that's the core problem with arguments about exponential growth. The coefficients matter! Exponential growth that starts small and then continues for a short period doesn't matter, the end result is still small. Almost everyone ignores this. They just say - look! Exponential growth! Invariably the growth isn't actually going to be exponential (ignored) and they don't specify the starting values, the exponents, nor any feedback loops that would suppress the growth rate. So it becomes an entirely abstract theoretical argument that leads people astray, as it did so disastrously during COVID.
If there's a threshold number of infections beyond which you can't control an outbreak, and it's less than the point where the exponential growth stops, then in practice your options often come down to "panic at the first sign of anyone getting infected" or "accept that you're not going to control the outbreak very well" - and when it comes to *new* potentially dangerous viruses that nobody knows if they really do need to panic over, if you do panic and stop the spread in time, you still look like an idiot because 1) you panicked and inconvenienced a lot of people and 2) nothing much happened. :(
The issue here is the assumption that "we" control outbreaks that would be "exponential" otherwise. In practice outbreaks always end by themselves. They show logistic growth, not exponential, including when humans do nothing to control them (which is the usual case). Even the bubonic plague did that, despite the huge impact.
In the modern era with better hygiene, food and medicine there basically aren't any pathogens that place serious stress on the health system. The "stress" of COVID on the healthcare system was largely imaginary, hence all the dancing nurse tiktoks and the notable lack of any staffing expansion.
But when you're panicked it's easy to overlook that. Real world experience seems somehow less intellectual than equations, and ruthlessly dishonest academics will never hesitate to use that to seize power.
That seems to me like a weird definition of Doomerism. The concern is that when AI can self-modify, its goals will be incompatible with humans still existing - if only because we are made of matter and matter is useful.
I’m not concerned with LLMs killing us. I’m concerned with their success funneling resources into an “arms race” where no one bothers with safety for when someone DOES invent an AI that can actually kill us.
"I think some form of “AI safety” is necessary... pragmatic, empirically-supported checks embedded in the technology itself and implemented as it is being developed."
Currently, we are nowhere close to what you describe. Instead, it is incredibly easy to make LLMs violate their rules. There are no empirically-supported checks.
I'm not terribly worried about this right now, because I do not believe LLMs have the potential to self-modify and increase their own efficiency and intelligence. They aren't agentic and they don't understand anything.
However, it is the states goal of OpenAI that they are working to develop an AGI, a superintelligence. If one sees this as possible, at all, then it must follow that there are serious risks that follow as well. And right now, no AI company is working on the sort of safety that you describe, which, to me, is an incredibly bad sign.
Funnily enough, I started off the "collapsed timelines era" as e/acc aligned because I thought the initial doom-mongering was overdoing it and there are obviously serious personal (life extension failure) and civilizational (dysgenics/tech stagnation) risks in shutting it all down. However, I think the pendulum has since swung far too much back into e/acc, who tend to be blithe and too dismissive of the idea that AI X risks even exist.
* Yes, there are AI researchers who think it's a nothingburger. But there are expert polls of AI researchers on this, quite a lot of them now. The timelines and p(doom) they project in aggregate are consistently and objectively alarming.
* Ironically, part of the reason I am "doomerish" is that I know some psychometrics, and am triggered when e/acc people compare AI superintelligences to corporations or countries. https://x.com/powerfultakes/status/1661395780027006976
I leave aside LeCun's troll comparisons to cats.
* My main concrete objection to "AI pause" and similar initiatives is that I consider them to be banally unfeasible, and as of the present time likely to do more damage than benefit, not just wrt general tech progress but even as pertains to the narrow domain of AI risk (exemptions are getting universally carved out for military applications of AI, which is like the one VERY obvious thing that all people and all states have a clear and present interest in banning with extreme prejudice, on the notion that militarizing a potential new apex predator life form is insane from any perspective). In short, the institutions that we actually have give no grounds for optimism that they are capable of acting to reduce AI risks, even if they wanted to.
The rest of this post seems to be mostly complaining about Twitter culture. I agree with this. The proper response is to stop using Twitter. I did this in 2020 and have no regrets.
A lot of this piece is about how many doomers make poor arguments and overstate the consensus and risks. I very much agree, and I think a lot of the soldier mindset and focus on unworkable solutions I seen in AI risk discourse reminds me of some folks with another existential anxiety, climate change.
That being said I do think that AI is the most consequential issue of our time and does pose serious existential risks, just on longer timelines than most doomers.
sure, it's not intended as a comprehensive piece against AI risk. I will write one on biorisk specifically
One of the biggest biases of a certain kind of Rationalist is that they think and talk a lot about things they find interesting, then argue that those things are as important as they are interesting. We have bigger, more pressing problems than Paperclip Machines, but they aren't as neato to talk about.
yes, exactly, it's a sort of procrastination from doing things that are boring
I'm not where you were, for one I don't get anxiety over things I can't control so that's helpful. But I am concerned and looking for reasons to be more optimistic.
However I have utterly failed to be convinced by any of LeCun's argument, who's generally eluding points and consistently being proven wrong on specific claims.
Besides "they're full of shit on biorisk!" which is an extremely narrow claim, although it should inform our priors somewhat, what have you found in either LeCun or Recht discourse that you found so compelling?
Because from what you write I understand "well the risk is not imminent so we're good", but capabilities are ratcheting up.
AI Doom is certainly possible but it's still a risk worth taking given the tremendous rewards.
There are no tremendous rewards if we're all dead.
We’re all gonna die, Forrest. One way or another. Nothing ventured and all that. We have yet to see a doomer prediction come true in all of human history. Will unexpected bad shit happen? Yeah, certainly. Will we be OK anyway? Yeah, probably.
If you define "doom" as "human extinction", then obviously no human could ever say "we have yet to see a doomer prediction come true." But is human extinction possible? Yes.
Yup. So? My two favorites are the Jellystone caldera going pop and a GRB that’s not off by just a degree (like the one that microwaved the Earth in the Precambrian). By comparison to real extinction level events we see in the geologic record, I’m terribly unworried about SkyNet. I have faith in John Connor. 🤖✊
Okay. I maintain that it would be a good idea to develop AI safety methods before trying to develop superintelligent machines.
Your solution of "trusting John Connor" doesn't convince me, because you're referring to a movie character that can time travel, which is not apparently possible given the known laws of physics.
There's no evidence that any proposed "safety methods" will actually result in anything safer. They will definitely impede anyone else from entering the game who might raise the bar for the soon-to-be-entrenched incumbents. I'm in favor of simple accountability, but I think that should be applied universally to all software, not just AI. The imposition of standards of care for all software companies, executives, and practitioners is the only reasonable path. Then hammer the accelerator.
Doomers also have a gross misunderstanding of the supply chain required to support the pathetic excuses we have for AGI now and the massive increases required to get closer. The energy suck and fragile hardware supply chain required to keep what little progress we've made functional would be destroyed by anything that's is remotely damaging to humans, let alone any existential impact. I work at chip manufacturer with a primary focus on the AI droplets we produce and I have a passing familiarity other clouds' internals (having worked elsewhere). It's a minor miracle that models get trained and utilized as much as they do today.
We are all John Connor. And John Galt. Also, try playing the game "Everyone is John" it's fun. Doomers seem to be lacking in the fun department.
Geological X risk events happen once every 100M years and X risk scale GRB hits are estimated to happen once every 10B years (longer than Earth will remain habitable). These are if anything events that popular culture vastly overinflates relative to real risk levels, though understandably so (asteroids, eruptions, tsunamis, etc. have something of the sublime about them).
So? You’re saying we’re overdue? OK. Which one is better prepared to mitigate the meteor risk before it becomes a meteorite extinction: humanity alone or humanity with AGI? It’s OK. I’ll wait while you google the math.
No reason to hurry on taking that risk, if we can find ways to reduce the level of risk with time and research.
Hey Ruxandra, I'm a big fan of your blog. Your beef with the doomers makes me sad, because in my mind, it is the biggest thing you're getting wrong.
But first I'm going to say: I'm sorry to hear about your panic period. I don't advocate for panic -- it doesn't sound pleasant, and doesn't seem all that helpful.
Nevertheless, it's important to have a realistic assessment of risks.
Some replies to specific points:
* In terms of expert opinion: "Median respondents put 5% or more on advanced AI leading to human extinction or similar, and a third to a half of participants gave 10% or more." https://blog.aiimpacts.org/p/2023-ai-survey-of-2778-six-things
* That said, your emphasis on expert opinion and "empirical data" seems somewhat misplaced to me. I remember in the early days of the COVID-19 pandemic, here in the US, experts were trying to reassure us by saying there is "no evidence of community spread in the United States". I didn't find that particularly reassuring, because I felt there were strong *theoretical* reasons to expect community spread before too long. I feel similarly regarding AI doom.
* AI experts mostly have expertise in incrementally improving existing AI systems. They aren't very incentivized to be good at *forecasting* the development of AI.
* More importantly, solving alignment-style challenges with existing AI systems is playing on easy mode. If something goes wrong, it's relatively straightforward to identify and fix the problem. With superhuman AI systems, I expect new problems to crop up. They'll be harder to identify, and quite possibly impossible to fix. I'm an ML engineer and I've spent hundreds of hours thinking about this on a fairly technical level, but I'll offer a basic analogy: Imagine you tried to fool your cat, and you were pretty successful. You might conclude from that experience that "it's easy for animals to fool each other". That's not the right conclusion: If your *cat* tried to fool *you*, it's not going to succeed for very long. In some sense, I worry that AI researchers may be developing "anti-expertise": Current AI models are causing them to develop intuitions which may not transfer to the superhuman scenario.
* In terms of biorisk -- most doomers aren't biologists. In the Less Wrong survey, you can see the userbase has around 8x as many people who work in AI as people who work in biology: https://www.lesswrong.com/posts/WRaq4SzxhunLoFKCs/2023-survey-results#III__Work_and_Education Among prolific posters I would guess the ratio is much more skewed. I'm not sure we can conclude that they're wrong about AI based on them being wrong about biology.
The AI doom conversation is well over a decade old at this point. The panic you see on social media is just the tip of the iceberg. The submerged part is many thousands of words: papers, books, blog posts. I've recently been telling the doomers that it might be a mistake to abandon their argumentative standards to stir up panic. I think your blog post is evidence that I'm right.
Michael Huemer is a smart guy. But based on the post you linked, he hasn't engaged with the AI risk literature (the submerged part of the iceberg) in much depth.
E.g. Huemer writes: "If we make a conscious machine, we’ll probably design it to be much more benevolent than us." Doomers agree this is a good idea! They just think it will be really hard. A big portion of the AI risk literature is devoted to discussing and debating this. Here's a recent summary from Eliezer re: why he thinks it's super hard https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/agi-ruin-a-list-of-lethalities
Similarly, in section 4.2 Huemer talks about Ted Kaczynski trying to escape from prison. This is another topic the doomers have discussed amongst themselves in fair depth. Believe it or not, Yudkowsky actually won in a couple of simulations, playing as "Kaczynski" trying to talk his way out of a "prison". Click the "(Read More)" link here for a high-level summary: https://www.lesswrong.com/tag/ai-boxing-containment
As I said, I like your blogging and I'm happy to go back and forth for a bit if you reply to this comment. My overall point is there's more to AI doomerism than meets the eye. Eliezer has some serious character flaws, but having character flaws doesn't mean you're wrong about everything.
Thank you!
I have engaged with the "bottom of the iceberg". This was not a post aimed to comprehensively debunk all the AI arguments. The AI in bio is just something that opened my eyes when I was panicked, not the only thing I took as an argument.
Many, many times through many conversations. I am left unsatisfied. To me it just does not seem convincing. I do not think current LLMs are going to lead us to a superintelligence. I think EY & co are making sort of category errors and their framing is wrong.
I will write a longer separate blog post about my view on this whole thing.
I am not sure I consider it a "beef", btw. I just do not think these are good arguments, which I am not sure should be considered a "beef"
>I have engaged with the "bottom of the iceberg".
Well you could've fooled me, linking to that Huemer post ;-)
>Many, many times through many conversations. I am left unsatisfied. To me it just does not seem convincing. I do not think current LLMs are going to lead us to a superintelligence. I think EY & co are making sort of category errors and their framing is wrong.
I'm an AI alignment researcher and I often find Eliezer's arguments unsatisfying. He seems really overconfident. Many other alignment researchers feel the same way. On Less Wrong a recent upvoted post stated: "The weird thing here is that [Eliezer's] >90% doom disagrees with almost everyone else who thinks seriously about AGI risk." https://www.lesswrong.com/posts/ye78Dip8YNgLBKGcy/seth-herd-s-shortform?commentId=FpdvoZsmmrNLekkz9
The key thing for me is, I think the burden of proof should be on the people building advanced AIs to show they will be safe, in advance of creating them.
EY fails to *prove* that the AI will be unsafe. But attempting to build superhuman systems without much of a safety plan, like OpenAI is trying to do, is inherently a *very* sketchy activity -- *significantly* sketchier than placing the first, gigantic prototype of your new, non-peer-reviewed nuclear power plant design right in the center of Manhattan. You say you're anti-safetyist, but I presume you aren't insane ;-)
If you build a bridge that's gonna take heavy traffic, the burden of proof should be on *you* to show that the bridge will hold up. There are various practices that have arisen in fields like aviation and civil engineering based on decades of failures and post mortems. As an alignment researcher, I spend my time reading about that stuff. That sort of safety culture doesn't exist to nearly the same degree in the world of AI, and the stakes are much higher because it's not just people on the bridge who could die, it's everyone. Plus, we aren't going to get a chance to learn from decades of failure, at least not in the same way -- as I stated, superhuman systems are likely to present qualitatively different failures. So the risk/reward of the current AI capabilities charge looks unfavorable.
Anyway I'll keep an eye out for your post and try to comment. You can also email me a draft at coasterchatter11 on gee-mail and I can offer pre-publication feedback.
I find Huemer's post better than the counterarguments.
I think getting lost into the weeds of this stuff can make one forget about simpler and fundamentally correct arguments.
The other thing is -- if AI Safety ppl do not consider EY to be accurate, they should distance themselves from them. It is not the outsider's responsibility to go through thousands of blog posts to disentangle who says what. As it currently stands, a lot of important figures seem to be influenced by these ideas, so I consider them an integral part of the movement.
I also fundamentally disagree with this approach to technology -- where one needs to prove future iterations of the tech are safe. At the moment I think chatGPT is pretty safe by normal tech standards. If future models become more dangerous, that should be judged at the time, not now.
I don't see much of a "simple and fundamentally correct" argument in this paragraph from Huemer:
>Reply: [Human immorality is] a fair point [for why AIs will be immoral]. So we’d better not model the AI on ourselves—at least not the average human; perhaps we could design a computer to imitate the best people we know. If we make a conscious machine, we’ll probably design it to be much more benevolent than us. E.g., if we figure out the basis for emotions, perhaps we’d make the AI love us.
It really seems like he's handwaving over the core issue, of how difficult it will be to make a benevolent AI? That said, I only skimmed his post; maybe there's something I missed which I addresses that core question elsewhere?
>The other thing is -- if AI Safety ppl do not consider EY to be accurate, they should distance themselves from them.
That's more or less what I'm doing right now :-) It's also what e.g. Paul Christiano did in this post: https://www.lesswrong.com/posts/CoZhXrhpQxpy9xw9y/where-i-agree-and-disagree-with-eliezer It's not uncommon for alignment researchers to express disagreement with MIRI.
The AI alignment community is much bigger than just MIRI. MIRI is unusual since it's focused primarily on public communications and politics. Most alignment researchers have their heads down writing papers and running experiments. See https://www.alignmentforum.org/ for a feed of recent research. It's certainly fair to say that MIRI's ideas have been influential, however.
I understand you don't want to go through a ton of blog posts. I'm happy to answer any questions; I've been following MIRI and friends for over a decade. This survey might be helpful as a quick overview of the alignment community: https://www.lesswrong.com/posts/XTdByFM6cmgB3taEN/key-takeaways-from-our-ea-and-alignment-research-surveys
>I also fundamentally disagree with this approach to technology -- where one needs to prove future iterations of the tech are safe.
I doubt this is your actual position. E.g. if Boeing were to introduce a new jumbo jet, I'll bet you would want someone, somewhere to have good reasons to believe that it won't crash and tragically kill all its passengers. If the main design priority was trying to recreate a child's sketch, I doubt you would want to step on that plane.
Maybe it would be useful to discuss the power plant hypothetical I brought up earlier: a gigantic nuclear power plant prototype right in the middle of Manhattan, which hasn't received any sort of review from regulators or engineers outside of a small design team. We can subdivide the disagreement into two questions: (1) is this power plant hypothetical a reasonable way to deploy nuclear technology, and (2) how does the power plant hypothetical compare with advanced AI development.
You may be correct that people are typically too risk-averse, but it's possible to take things too far in the opposite direction. If your house is too cold in the winter, that doesn't mean you want to set it on fire.
>At the moment I think chatGPT is pretty safe by normal tech standards. If future models become more dangerous, that should be judged at the time, not now.
And then what? Supposing you were to judge a future system as dangerous -- what would an appropriate response be at that point? I suspect by that time, the cat would be out of the bag. We're lucky that nuclear weapons can't be built by the average Joe, for physics-related reasons. Are you confident that it would even be feasible to restrict this hypothetical dangerous AI system, after everyone learns that it's possible to build?
BTW, I recommend reading up on the history of nuclear close calls if you haven't already. See the stories of Stanislav Petrov or Vasily Arkhipov, for example.
You say you're fairly familiar with AI risk discussion. Can you tell me what you remember reading about "hard takeoff" and "soft takeoff"? Presumably you can see how in a world with hard takeoff, your suggestion to wait for clear danger would likely fail. So -- do you have specific reasons why you think hard takeoff can be ruled out? How about AI deception (e.g. the so-called "treacherous turn"), where unfriendly AIs masquerade as helpful and friendly, because they're smart enough to realize that's a better way to achieve their counter-human goals in the longer term? That could also defeat the "wait for clear danger before shutting it off" approach.
Maybe it'd be helpful if you mentioned the specific AI risk intro you read. I feel like most good intros should've covered the topics I'm bringing up.
Thank you for all these links! I hope you understand there is a limit to how much time I can dedicate to debating this...
Briefly, though: I do not think that we should expect a hard take-off based on current evidence, unless a big improvement in architecture happens.
That’s fair. That’s the key question around an AI pause for me. I’m less concerned about scaling per se, and more concerned about preventing that sort of big improvement in architecture which could lead to hard takeoff. But it’s not clear how best to accomplish that from a policy perspective.
what about the next breakthrough in AI field after LLMs ?
my bias is that AI X-risk is a nothin’ burger but bc it flatters high IQ people, they worry about it. The philosophy chooses the apocalypse that nerds can at least take credit for. It’s a fantasy too if all the dumb normies of the earth (AND ones’ more successful peers) will be wiped into valueless statusless nothing or gruesomely purged. More or less don’t want to understand that bc it’s pretty boring
But the refined tastes that make natalism a hot topic, 🤔 hmmm interesting
(not really) The RW journo sphere accidentally caring about something real is pretty cute. But their reasons are probably uninteresting. My sense is that it’s the perfect whatabouttist retort to climate. But infinitely better than trying to legislate climate. Because instead of having to go after the titans of industry it’s the fault of DINKs, ordinary infecund proles (and women, of course).
It's not so much about taking credit for it, but about the idea that intelligence is a super power.
What were your primary concerns as a "doomer"? Was it more "the world is going to end" or more "my work will be obsolete," and therefore what is the point of existence?
And besides following new people on Twitter, like Yann, is there anything else that helped you overcome this fear?
It varied. I mostly oscillated between I will be completely obsolete no matter what I do other than smth super physical to AI will enable super catastrophe.
I mean I also talked to eg Ben in DMs quite a bit. I paid attention to what chatgpt could actually do -- and whether the claims abt its intelligence at the time were warranted.
For every claim I encountered I started to apply more critical thinking.
ChatGPT increases biorisk. Ok, what if we throughly abt all the biorisk before the internet was created? After all, it gave access to unlimited info to so many ppl. Analysed papers discussing biorisk. Most of which underestimate how hard biology is in practice
The AI doom scenario is concerned about future AI systems with human-level or above intelligence, which Yann LeCun also believes will come.
ChatGPT does not increase biorisk and there is no loss-of-control risk.
Also important to keep in mind that it called risk for reason, i.e. there is a chance of losing control to future smarter-than-human AI systems. We should take that risk seriously and try to mitigate it. Yann LeCun for some reason believes that is guaranteed that we will succeed with this very daunting task of controlling more intelligent (i.e. more powerful) systems on an unlimited time horizon.
He also believes we're very far from it
Are you sure? What's his timeline? when pressed on the issue, many "non-doomers" claim "long" timelines like 10 or 15 years.
Or they believe that ASI is coming and will be powerful enough to basically fix everything, but magically not powerful enough to be a risk.
An additional remark:
Given the enormous progress we have made in recent years and the amount of resources (money, manpower) flowing in, I don't think it makes much sense to make confident predictions even on a 5-year horizon.
Generally, I think Yann LeCun is way too confident with his predictions. Like everybody else, he doesn't have a crystal ball where he can see the future.
That's true. Many experts in the field think it is much closer though. Predictions about the future are inherently speculative. I think the next 5 years will really clear things up, one way or the other.
What makes me a bit uneasy that we are already in the uncanny valley for AI intelligence. We judge AI by human standards. This is IMO just another indication how close we have come.
Plus the enormous amounts of money and manpower that are flowing into AI research.
I'm a software engineer working closely with AI and I'm fully convinced that AI doomerism is a marketing scam to make a toy look far more powerful and capable than it really is.
Hah!
That's not very plausible, since these ideas have been circulating ever since the emergence of computers. They did not arise or emerge from marketing campaigns. (By the way, which marketing campaigns promulgate the idea that AI pose a threat to the species? Can you produce an example?)
What's more, the mere fact the industry benefits from this position doesn't, by itself, show that they promulgate or encourage it. Not to mention that you attribute to the industry a clearly flawed cost-benefit analysis. It would be totally irrational for them to make these claims. To the contrary, it is far more likely that companies will overstate their ability to develop AI safely, since it is in their financial interest to self-regulate, as is the case for most industries.
It's hard for me to agree or disagree with this unless you are willing to give a probability that that world is destroyed by A.I [p(doom)]. Did you go from 99% to 1%? Or something else?
I'm genuinely sorry to hear that you went through such a stressful time during the release of GPT-4. I do agree that there was something close to a moral panic that happened then. I feel mixed about whether it was net-positive or net-negative. On the one hand, it was nice for AI Safety to finally enter the Overton Window. On the other hand, it didn't feel like the best advocates for AI Safety were getting the most airtime. Yudkowsky, while certainly an important figure in the history of the movement, is--to be blunt--past his prime. He offered little in the way of arguments and much in the way of flailing his arms around and screaming "We are all gonna die!" This did a good job getting the more sclerotic institutions in our society to finally wake up, but it damaged the credibility of AI Safety in the eyes of people who care more about the substantive strength of the arguments and not just vibes.
And a problem is that a lot of people thought that the doomers were talking about LLMs themselves being dangerous. But what AI Safety people were really reacting to was the apparent speed-up in capabilities (their "timelines"). But that wasn't always clear when people were retweeting spicy AI Safety memes. This meant, a year later, when LLMs continue to make the same stupid hallucinatory mistakes, all the sky-is-falling rhetoric looks like fear-mongering.
I would plead with you to give AI Safety another chance. While I'm sure you've done some reading, it's not clear to me just from this short post whether you've found what is (in my opinion) the really good stuff.
Two places I would recommend starting are Paul Christiano's work and Chris Olah's work.
Christiano has a more moderate view compared to Yudkowsky. He also has technical expertise that Yudkowsky lacks: he was iirc one of the creaters of reinforcement learning by human feedback (RLHF). He still thinks that AI constitutes a serious risk, but his scenarios are more interesting and "systemy" than the more super-smart-agent-takes-over-the-world scenarios that Yudkowsky tends to emphasize. He's also a very important person to know going forward. He recently got a bigshot position in the US Government to work on AI Safety.
Here is a good place to start with Christiano: https://www.lesswrong.com/posts/HBxe6wdjxK239zajf/what-failure-looks-like
Chris Olah is a researcher at Anthropic, one of the leading AI companies (their main LLM is Claude 3). He is known for mechanistic interpretability, a subfield of AI Safety that seeks to learn how to decode the neuron weights of the neural network in order to understand what the AI is thinking. My impression is that mech interp is the most respected research direction in AI Safety currently (and I am personally favorable to mechanistic interpretability as someone who has chatted casually about it with acquaintances in a non-technical way).
To understand mechanistic interpretability, I think this Scott Alexander article would be a good place to start: https://www.astralcodexten.com/p/god-help-us-lets-try-to-understand
Also, obligatory plug to listen to Dwarkesh Patel's podcast: https://www.dwarkeshpatel.com/
Hi, thanks.
Well I'm not sure why LLMs becoming better made people react the way they did. We should expect an improvement in one part of the technology to happen fast but that doesn't tell us we're "taking off" towards human intelligence. And many many people absolutely argued LLMs -> AGI
My baseline is the 0.35% chance of doom by 2100 established by a group of superforecasters at the Foresight Institute. Anyone whose risk guess is much higher than that should present a comprehensive argument for why all the superforecasters are wrong. I’ve seen precisely no one engage with that figure seriously. Ergo, doomers are LARPing.
Rough calculation (with conservative estimates):
AGI/ASI until 2100: 80%
Risk of losing control to smarter-than-human AI systems: 25%
Together: 80%*25% = 20%
It is really not rocket science. To even attempt to control smarter-than-human AI systems feels like hubris. How could anyone suggest that this is an easy task?
Some additional arguments here (part of a discussion with Yann LeCun): https://x.com/StraubHart/status/1795048954859544599
Yudkowsky has alot to answer for. Not because he believes something wrong or which I disagree with but because he privleges stories over precisce definitions and -- in a way disturbingly reminiscent of the religious establishment of an earlier day -- not only equates lack of disproof with evidence for his claim but also refuses to admit the weaknesses in his argument.
Look, I don't agree with Bostrom, but he advanced an interesting argument about AGI and actually tried to fill in the holes. What should have happened after he published his arguments is for people to sit down and try and flesh out the concepts he used such as intelligence and tried to clarify the reason why we should expect AGI to act to maximize a simple goal outside of the area it was trained on. Instead, what we got was a pile of steaming crap that amounts to nothing but parables masquerading as arguments.
I mean the whole self-improvement FOOM argument is totally unjustified once you think about it quantatatively or from a mathematical POV. The idea is supposed to be that since we can build an AI smarter than us it will be able to build an even smarter AI and this explodes to infinity. Ok, but the second you try to quantify that it no longer works -- I mean why assume that the time for each unit of intelligence increase is decreasing (much less towards 0) not increasing? And what even are these units? (Correct choice of units can makes it go to infinity or asymptote whatever is happening).
Which brings us to all of those parables that equate intelligence with omniscience -- as if AI somehow wouldn't be bound by the underlying mathematics of computational complexity. Once you realize that it can't, all those little stories about how AI can escape safety measures start to sound like nothing more than modern recastings of stories about genies and lamps. But, unlike the many responsible people in this area, Yudkowsky treats it as the burden of the person who isn't a doomer to disprove this could happen -- effectively saying that lack of knowledge about AI is reason to be confident they'll kill us not reason to say we don't know.
Ultimately, it's still an interesting argument that's worth discussing (even if I don't find it very plausible) but doing so responsibly means demanding arguments be filled in with more precisce language and treating the failure to do so as evidence against them not assuming the things that feel salient when we consider narratives about it are presumptively true and others have the burden of disproof.
I agree that AI is presumably bound by the mathematics of computational complexity, but they might allow a lot more that one would think - simulating protein folding is most definitely one of those computationally intractable problems, but AlphaFold actually does work.
Yes, and I'd welcome work trying to fill in Bostrom's argument here. But Yudkowsky's parables don't help here.
Maybe an advanced AI would be capable of many things but it seems pretty unlikely that it would have the ability to do psychohistory a la foundation and with high probability manipulate it's way out of any set of limitations we might impose.
Indeed, I think Yudkowsky may have done the most out of anyone to increase our risk from unaligned AI because he has simultaneously convinced one group of people that this AI alignment stuff is just non-serious bullshit while convincing another group of people that there is no point in trying to ask for limitations like only allowing AIs to advise people (in contrast to giving it direct control of things in the world) because of his little parables which describe how a computationally unbound agent could manipulate people to achieve whatever it wants anyway.
If there is no significant risk from AI no big deal. But it's people who think we should take that possibility seriously who should be most upset at Yudkowsky. He has effectively made sure people don't pay attention to the kind of practical and perhaps very serious concerns about how to manage AI in the real world by pulling all the attention towards these unrealistic scenarios.
Freaking out is for sure unhelpful. I'm not worried about AI because I don't think our society has the technical juice to get there from here. However, if we could and did, Yudkowsky would be proven right, and Yann wrong. There's no need to argue with his central arguments in order to avoid anxiety.
Right, I mean Yann also argues we are far away from a "superintelligence", which is mostly the argument I find convincing
If astronomers found that a big comet, the size of the one that killed the dinosaurs, was on a direct course to hit Earth, but the collision was going to happen five hundred years from now, would you be concerned and think we should start investing in comet deflection technology right away, or would you just shrug and leave future problems for the future to solve?
There's a general pattern here that's useful to learn from - intellectuals who talk about exponential growth, the rapid rate of change or even the derivative of the rate of change have a very long history of making invalid predictions. I think it's a psychological problem, in which talking about derivatives makes you sound much smarter than other people and therefore there's always a huge temptation to appear more attuned to small changes than everyone else.
A recent example of this was of course covid, in which many people claimed that viruses grow exponentially, even though this is trivially untrue and anyone can see it just by looking at a graph of cases in any previous epidemic. Indeed the founding discovery of epidemiology was that epidemics follow a bell curve like distribution with an early period of rapid growth that then later peters out well before every susceptible person has been infected. It was very apparent to me from reading comments and taking part in discussions at this time that a lot of educated people were in love with the idea that understanding exponential growth made them somehow special. Any attempt to point out basic things, would be met by eye rolling and sneering about how people that don't understand exponentials are dangerous.
The AI risk debate seems to have very similar characteristics, in which people make arguments not based on what is actually possible today but based on an extrapolated rate of growth that they claim will massively speed up in future. We don't necessarily need to engage directly with such arguments, it's often sufficient to merely pattern match it to faulty ways of thinking in the past.
very good point. Yes, I guess they would say "but this time it's different"...
Mmm, my experience has been that they don't like to say this. I guess it would concede the point that they're in dangerous territory. What people say instead is some variant of "You just don't understand the exponential function" or "there's a consensus of experts that you're wrong".
Oh, and of course we must not forget "that other prediction wasn't actually wrong, it just hasn't happened yet!"
Well, in the short run at least, populations of living things - including viruses - do tend to grow exponentially until something stops them. Which is one reason why invasive species can become huge problems.
Yes but something always does, so that's a kind of trivial observation.
I was trying to imply that the "short run" period of exponential growth can still last long enough to make a small problem into a really, really big one in what looks like a very short amount of time.
There's an old riddle: a patch of lilypads doubles in size every day. On day 30, it fills the entire lake. How many days does it take to fill half the lake? The answer is, of course, 29 days - it only takes one doubling to go from half the lake to all of the lake. By the time something that grows exponentially-for-now is a big enough problem to notice, it's often too late to stop it from becoming too big to fix.
In theory, yes. In practice this hardly ever happens. Even in cases where there are invasive species due to importation and lack of local predators, you don't get grey goo scenarios and the ecosystem eventually settles down.
And that's the core problem with arguments about exponential growth. The coefficients matter! Exponential growth that starts small and then continues for a short period doesn't matter, the end result is still small. Almost everyone ignores this. They just say - look! Exponential growth! Invariably the growth isn't actually going to be exponential (ignored) and they don't specify the starting values, the exponents, nor any feedback loops that would suppress the growth rate. So it becomes an entirely abstract theoretical argument that leads people astray, as it did so disastrously during COVID.
If there's a threshold number of infections beyond which you can't control an outbreak, and it's less than the point where the exponential growth stops, then in practice your options often come down to "panic at the first sign of anyone getting infected" or "accept that you're not going to control the outbreak very well" - and when it comes to *new* potentially dangerous viruses that nobody knows if they really do need to panic over, if you do panic and stop the spread in time, you still look like an idiot because 1) you panicked and inconvenienced a lot of people and 2) nothing much happened. :(
The issue here is the assumption that "we" control outbreaks that would be "exponential" otherwise. In practice outbreaks always end by themselves. They show logistic growth, not exponential, including when humans do nothing to control them (which is the usual case). Even the bubonic plague did that, despite the huge impact.
In the modern era with better hygiene, food and medicine there basically aren't any pathogens that place serious stress on the health system. The "stress" of COVID on the healthcare system was largely imaginary, hence all the dancing nurse tiktoks and the notable lack of any staffing expansion.
But when you're panicked it's easy to overlook that. Real world experience seems somehow less intellectual than equations, and ruthlessly dishonest academics will never hesitate to use that to seize power.
That seems to me like a weird definition of Doomerism. The concern is that when AI can self-modify, its goals will be incompatible with humans still existing - if only because we are made of matter and matter is useful.
I’m not concerned with LLMs killing us. I’m concerned with their success funneling resources into an “arms race” where no one bothers with safety for when someone DOES invent an AI that can actually kill us.
"I think some form of “AI safety” is necessary... pragmatic, empirically-supported checks embedded in the technology itself and implemented as it is being developed."
Currently, we are nowhere close to what you describe. Instead, it is incredibly easy to make LLMs violate their rules. There are no empirically-supported checks.
I'm not terribly worried about this right now, because I do not believe LLMs have the potential to self-modify and increase their own efficiency and intelligence. They aren't agentic and they don't understand anything.
However, it is the states goal of OpenAI that they are working to develop an AGI, a superintelligence. If one sees this as possible, at all, then it must follow that there are serious risks that follow as well. And right now, no AI company is working on the sort of safety that you describe, which, to me, is an incredibly bad sign.
Funnily enough, I started off the "collapsed timelines era" as e/acc aligned because I thought the initial doom-mongering was overdoing it and there are obviously serious personal (life extension failure) and civilizational (dysgenics/tech stagnation) risks in shutting it all down. However, I think the pendulum has since swung far too much back into e/acc, who tend to be blithe and too dismissive of the idea that AI X risks even exist.
* Yes, there are AI researchers who think it's a nothingburger. But there are expert polls of AI researchers on this, quite a lot of them now. The timelines and p(doom) they project in aggregate are consistently and objectively alarming.
* Ironically, part of the reason I am "doomerish" is that I know some psychometrics, and am triggered when e/acc people compare AI superintelligences to corporations or countries. https://x.com/powerfultakes/status/1661395780027006976
I leave aside LeCun's troll comparisons to cats.
* My main concrete objection to "AI pause" and similar initiatives is that I consider them to be banally unfeasible, and as of the present time likely to do more damage than benefit, not just wrt general tech progress but even as pertains to the narrow domain of AI risk (exemptions are getting universally carved out for military applications of AI, which is like the one VERY obvious thing that all people and all states have a clear and present interest in banning with extreme prejudice, on the notion that militarizing a potential new apex predator life form is insane from any perspective). In short, the institutions that we actually have give no grounds for optimism that they are capable of acting to reduce AI risks, even if they wanted to.
The rest of this post seems to be mostly complaining about Twitter culture. I agree with this. The proper response is to stop using Twitter. I did this in 2020 and have no regrets.
I second that advice. I've shunned the X-Parrot since long before it became X.