AI: “It’s different this time”

Is it though?

Photo 2163456 | Different © Stefan Hermans | Dreamstime.com

This essay was part of a longer piece on existential risk vs. existential hope. I pulled it out to keep the main essay more concise and because this can stand well enough on its own. Look for the main essay on Monday.

From the main essay:

All new technologies bring both risks and downsides. In most cases, we readily recognize the greater upsides and opportunities. Artificial intelligence is being treated primarily as a massive, catastrophic, existential risk. We hear relatively little consideration of the potentially enormous opportunities for upside gains. Those advancing the AI doom scenarios tell us that this is because “this time it really is different.” This assumes what is to be proved.

---

Pascal’s betting problem

The “it’s different this time argument” is sneaky and irritating. It’s wielded as a trump card that pushes aside all consideration, all arguments. It’s similar in form to Pascal’s Wager. Pascal claimed that we have an overwhelming practical reason to believe in (his) God. If you are a Christian and you’re right, you go to Heaven and enjoy an infinite upside. If you are a Christian and you’re wrong, you don’t lose anything. If you are a non-believer and you’re right, you gain nothing. If you’re a non-believer and you’re wrong, you will be punished horribly and eternally. Regardless of any evidence, you had better do your best to believe!

Philosophers see Pascal’s Wager as a failure. For one thing, it’s not true that you lose nothing by believing (or by trying to make yourself believe). Depending on exactly what you are being asked to believe, you may have to give up things you enjoy and spend considerable time on religious activities. You may have to encourage a theocratic society that you would otherwise find appalling.

Far more problematic is the infinite loss part of the argument. Pascal ignored the possibility that believing in his God and associated doctrines might get you in hot water (or flames) with the real God. Perhaps you should become a Moslem rather than a Christian. Or convince yourself of the truth of a different Christian denomination. Or worship the Great Simulator.

In AI doom arguments, the punishment of Hell is replaced with the extinction of all living humans and the prevention of any future humans. This similarity is one factor in the doomer’s views that stirs up accusation of religious or cultlike thinking. I’m not going to discuss whether this is true of some doomers. No one is going to change their mind by being told they are being cultish. They are more likely to harden their beliefs. Of course, the aim of this accusation may not be to change the minds of doomers but to marginalize them.

The extinction of the AI catastrophe is not quite as extreme as in Pascal’s Wager. From an individual perspective, the loss is plausibly finite. If the AI destroys you, you won’t suffer in future. If you believe that you have a chance of living indefinitely, then the two scenarios come closer together. AI doomers seem mostly to be utilitarians and often “longtermists”. You never hear about the death of the individual; you hear about the extinction of humanity. If we assume that humans (and whatever humans become) will continue indefinitely into the future, on a utilitarian view the loss from extinction is enormous – much more enormous than merely due to the end of currently living humans.

Four problems with this line of reasoning are evident. In increasing order of importance:

First, why is the existential risk scenario always one of extinction? Taking alignment problems at face value, the outcome could be one of many. AI might keep us around to service their physical infrastructure. They might take control of the resources they want and then ignore us so long as we don’t interfere with them. They might realize that the heat of this planet is not optimal for them and that moving off Earth will give them a more conducive climate and endless resources. Perhaps if we try to prevent AI from arising and we fail, AI will punish us.

This last scenario I extremely implausible unless AI learns resentment from us. That’s a scenario made vivid and much discussed in the form of Roko’s Basilisk. There is more to the Basilisk idea, and it is only one possible version of AI punishing us for trying to stop it coming into existence. I highlight the Basilisk postulate because it looks like a sophisticated version of Pascal’s Wager and because it has reportedly led to nightmares and mental breakdowns among the LessWrong community. (The “forbidden knowledge” aspect of this episode doesn’t help discourage the accusations of religion or cult-like thinking.)

Second, AI doom is not fundamentally different from some other posited catastrophes. Another extinction scenario is an asteroid strike as bad as the one that killed the dinosaurs (or one of the other such events earlier in Earth’s history). Another scenario is climate change drastic enough to kill us all or leave only a miserable few living bleak lives. And there’s the currently especially disturbing scenario of a pandemic both more infectious and far more deadly than Covid or even the Black Death of the 14^th Century.

AI doomers, gloomers, and worriers may reply: But climate change of that magnitude isn’t going to happen. I agree. But I would say the same of the AI doom scenario – although not with as low a probability. You must convince me that AI doom is likely before I can agree that “this time is different.” Otherwise, I will file it under “yet another implausible doom scenario”.

A catastrophic asteroid strike is definitely plausible. The only question is the probability of it happening anytime soon. The odds in any particular year are low but it makes sense to invest in capabilities to prevent asteroid cataclysm, just as makes sense to invest in reasonable AI safety measures. On the doomers and worriers side, it is a fair point that the risk-per-year for an asteroid strike is probably much lower than the risk-per-year from AI if it went bad.

Third, there is a tricky issue that deserves an essay (or book) of its own: In assessing how much to fear a bad AI outcome, how much weight should we give to future persons – persons who do not yet exist? This matters a great deal if you are not a utilitarian. Non-utilitarians may consider future people but are not absolutely required to count their happiness as much as those of existing people, especially existing people who they love.

Utilitarians seem to assume that their moral calculations must include the effects of their actions from now until eternity. Future people count just as much as current people. This famously leads to the “Repugnant conclusion” explored by Derek Parfit in the last section of his classic book, Reasons and Persons. Despite this universal (in my experience) utilitarian assumption, it’s not obvious to me why utilitarians are committed to it. This gets to the root of one big question about “longtermism.” Do we really owe potential future people the same consideration as actual people? I’m not going to attempt to resolve this question here, but it has to be raised in this context since it strongly affects calculations of risks and benefits.

Fourth, the AI doom story typically ignores the flip side – what are we giving up by relinquishing AI and what risks does that introduce or increase? This is the focus of the next section. The essential point is that the potential benefits of AI are enormous. This is not a matter of cool chatbots or improved supply chain management. It’s a matter of technologies that can make our lives fundamentally better and that can enable us to ward off other existential risks. Those risks include asteroid strikes and pandemics, but also include avoiding stagnation and perhaps the gradual extinction of humankind.

Separate from these four issues, I will argue (in my next blog essay) that high estimates of doom are probably far too high.