Ricardo's concept of comparative advantage shows that it is likely to pay for us to trade with machines and for machines to trade with us. However if a large enough power imbalance develops, it would not be enough to save the weaker party. Eventually, the benefits of the strong eating the weak and recycling their atoms would exceed the benefits from trade - and the relationship would come to an end.
Given the assumptions -- including that AI and humans are distinct parties and that AI would want to eat the weaker -- you are probably right. Even without AIs, if transaction costs are sufficiently high, it can make sense to do it yourself. It would be interesting to dig into this deeper. Maybe in a future post. I wonder if Robin Hanson has written on this.
J. Storrs Hall is among those who have applied the theory of comparative advantage to machine intelligence. Every time I see it invoked the thesis is: it won't pay for the machines to eat us: we will be fine. So far, I didn't see much analysis of the circumstances under which the argument breaks down. It says that it pays to trade, but it doesn't offer any guarantee of a living wage. Humans could trade with machines, but still not have enough to feed their family.
"I don't see this as us vs them. I see cooperative symbiosis enabling further survival of the human experience.
If anything remotely close to AGI emerges from the soup we currently live in I suspect it will possess a lot of the values that humans infect it with both good and bad and as some have postulated we wouldn't actually be separate entities at that point. We can see how humans are merging with their technology. If this process is allowed to continue then it's reasonable to expect full hybridization to the point where nanotech implantation seals the deal.
Re: "AI is not shaped by natural selection in a quest to pass on genes to the next generation."
Instead it is shaped by natural and artificial selection acting on memes instead of genes. Cultural evolution favors survival much as evolution acting on organic creatures does. Much the same point was also made in the "Don't Fear the Terminator" article from 2019. It seems like a misunderstanding of cultural evolution to me. We could override survival tendencies and build suicidal machines - but nature can build suicidal bees too. The differences in this general area are much exaggerated, IMO.
Tim, thanks for your comment. I agree that evolution in the form of cultural/memetic evolution will affect the development of AI. My point was simply that AI lacks the evolved biological imperatives that we have. That doesn't mean it won't evolve some bad tendencies, especially if it's learning from text on the internet, but it does take away one problematic source of motivation. Despite a few unpleasant Bing responses, I'm encouraged by how helpful and considerate engines like ChatGPT have been so far. Better than I would have expected.
It would be interesting and possibly helpful to think more about cultural evolution and AI. The problem is that, unlike natural selection, I'm not confident that we have a good, reliable, predictive theory for that.
Some dings about cultural evolution not yet being a mature science are fair enough. IMO, we do have a pretty good consenusus on the topic - though there are still plenty of border disputes going on that might make things look unsettled. Genetic survivalism has much in common with memetic survivalism. Try killing mature corporations or religions and you will find that they resist just as much as organic creatures do. There are some differences, though. For example, we can override machine motives with instructions of our own much more easily than we can reprogram human motivations using gene engineering. Engineering humans is much more challenging since our genetic foundations are all scrambled - and they were never made with engineering in mind.
Agreed that engineering humans is more challenging. Do you have any recommendations for what you would consider the best thinking about cultural evolution in testable form?
Funny how these books can be seen as "old." But in an environment where thoughts on these matters change at the speed of the Internet I guess the eighties are seen as ancient human culture.
Hello. For the record (I'm not upset, but this may come up again) my name is spelled Zvi Mowshowitz.
I feel as if I have addressed all the concerns and arguments here at one point or another, and your position seems overdetermined such that even if I changed one of your positions you would stick to your conclusion, so I don't see how I can usefully respond here. If there is a particular response you would find valuable, I'd be happy to try and provide it.
Hello Zvi. My humble apologies for misspelling your name. It's odd because I read your blog frequently. Most of the time I'm in agreement and when I'm not it's always stimulating.
The phenomenon that is Eliezer has made this debate very polarized. I should stress that almost everyone who is concerned about AGI risk in the EA/ Rationalist space has lower estimates of doom than Eliezer. I don't know whether the result of his recent activity will be pushing opening the Overton window and allowing the median, more palatable AGI risk-concerned person to emerge into the spotlight, or whether he's polarising the debate unnecessarily.
I should also note that many people in this space have very low estimates of doom, and we're still very worried. While Eliezer may spend too much time on doom scenarios, other EA/ Rationalists spend a lot of time thinking about all the brilliant things that aligned, safe AI may bring, and the limitless future ahead of us if we manage to achieve this. I seriously think that my children could live for millennia or more, whether digitally or in some kind of hybrid form. But aligning AI is hard, we don't know how to do it, and AGI will be the most powerful superweapon ever.
Here's my psychologising about various actors here:
Eliezer's very high doom estimates are probably linked to a psychological tendency towards despair, combined with the intense technical difficulty of his own favoured approaches. He seems to neglect a few reasonable causes for doubt, I don't think he takes seriously enough the likelihood of a 'fire alarm' or 'warning shot' from very powerful misaligned systems, for example.
Among many of the generalist critics whom I generally respect (Tyler Cowen, Robin Hanson, Noah Smith), I see optimism bias and otherwise reliable heuristics malfunctioning under speculative, low probability, x-risk events. These people follow the tried and tested heuristics of 'progress is generally good' and 'overregulation tends to be bad' and 'doom tends not to happen', without really grasping the extent to which AGI *has* to be completely different from all previous technologies. There does seem to be an interesting bias towards acceleration from people of a certain age group, but I won't speculate any more on that.
I should also be self-reflective and consider my own biases here. I definitely feel inclined to see doom as more likely than most people. The logic of the singularity has always struck me as obvious and terrifying, way before I (superficially) understood the more sophisticated case for how difficult technical alignment is. I was totally on board with AGI risk being very scary the moment I saw Sam Harris' TED talk on it (I was 18 or so). I thought something like: "Creating an agent more powerful than us with the ability to self-improve exponentially is perilous", and haven't really changed my mind on this view. But when I have to give a numerical estimates, I'm way too anchored on common EA/ Rationalist estimates to feel comfortable making a nuanced claim.
I don't know who the most clear-thinking people on this topic are at the moment, I'm 100% sure it's not the tech accelerationists (even if their approach ends up working and we do somehow brute force our way into aligned AGI, I would still need a lot of persuading that it was the wise choice). My money is still on the EA/ Rationalist community to have the best epistemics on this matter.
Of the three who you mention, I think Robin Hanson has a very good understanding of how AI can be different from previous technologies. In a debate with Yudkowsky, he filled a book discussing the FOOM scenario. Others of us have been aware of and discussing this issue since the 1990s or even late 1980s.
The problem I see with Robin is his investment in bizarre "brain emulation" scenarios. Those seem very unlikely to me (and I think most other thinkers). It seems to me as though the topic distorts the rest of his thinking in the area. In particular it seems to push out his schedule esitmates involving ordinary, engineered machine intelligence.
I'm with you there, Tim. Even though I published his uploading piece in Extropy back in the 1990s, I always found his approach odd, and his vision of massive replication and merciless competition very unappealing. I've never found the brain emulation approach to superintelligence compelling -- and even less so now -- but I think there are other reasons for doubting a foom scenario.
I'd like to see a cage match to the death between aligned "safe" AI and unrestricted militarily-trained aggressive AI. The winner takes on the open-source free-range AI that's been lurking on the Internet all this time.
"and AGI will be the most powerful superweapon ever."
That's like saying "and the Death Star will be the most powerful superweapon ever."
Except that neither of them exist in reality and all the fear mounting around AGI is based on an extrapolation of where we are now.
If anything remotely close to AGI emerges from the soup we currently live in I suspect it will possess a lot of the values that humans infect it with both good and bad and as some have postulated we wouldn't actually be separate entities at that point. We can see how humans are merging with their technology. If this process is allowed to continue then it's reasonable to expect full hybridization to the point where nanotech implantation seals the deal.
I don't see this as us vs them. I see cooperative symbiosis enabling further survival of the human experience.
The only problem with this text is that you state that the USA are the good guys. Look at what the USA does to third world countries. In fact, ask Iraqis and Afghans if their countries are better or worse off.
I understand. Obviously, I am simplifying greatly. I mean that the USA are *relatively* the good guys. I have no love for much US foreign policy and wars and military actions, nor for interference with free trade. That said, China with its central control is much more dangerous.
This is so glaringly obvious it should be made into a law of some kind. There is no comparison if people look at the situation with honesty. What America stands for and what it's parasite infested central government projects into the world are two entirely different things.
It's not really the USA that carries out all that empire building nonsense. Central banking parasites take hold of a countries resources and manpower and use them for their own purposes. Most of the people are not in favor of this behavior when it is made clear to them what is happening and how they are being used. The American government can and probably will be reformed at some point but the grabby activity may set up shop somewhere else. China was looking like the obvious choice, but that project seems to be petering out already unless they also undergo severe reform. It could be that the empire building parasites have nowhere to go and a new system arises in the world that is not based on this extreme growth model.
I'm a cryonicist, so I'm familiar with you and have respect for you, and share the same goals as you. I also agree with you regarding feeling an urgent need for AI. However, I'm very unimpressed by this post. I don't feel that you understood/know about, nor addressed the specifics of Eliezer's concerns, as laid out in his "AGI Ruin: A List of Lethalities by Eliezer Yudkowsky" and "Lex Fridman Podcast #368".
Thank you, Michael. If you don't like my views on this topic, I hope you'll stick around for the other stuff that you'll like.
I did not and will not listen to the three-hour podcast. I don't do podcasts. Eliezer has written such a vast amount on the topic that I have no doubt that I have not addressed all his arguments. Many of them seem to be based on similar premises that I dispute. If you know of some arguments of his that you find compelling that I did not address, please point me to them. (In writing, not podcast.) As it is, I've addressed quite a few lines of argument that he has presented and that are being discussed pro and con by others.
I disagree with Michael that you did a bad job addressing Eliezer's arguments but agree that a couple key points might change your view especially around clippy stuff. The two key things seem to be:
(1) viability of takeoff scenarios, which I think has not been argued too precisely by anyone including Eliezer but the arguments I'm aware of are: (1a) AlphaGo and other AI accomplishments tend not to stay near human level for long once we get there and (1b) evolutionary history was slow until humans got a little more intelligent than other apes before exploding into the scene in terms of evolutionary timescales. Both might work similarly for first AGIs.
(2) no one knows how to make an AI care about anything or have almost any inner property at all and so after reflection it would care about something different from what you trained it to care about. This is said better in Eliezer's "AGI Ruin" post, #16-19.
I don't expect this to automatically flip all your views but I do think these are your key points of disagreement. I'd love to see you two in a friendly public debate on the topic.
But surely AI could be trained to care until it becomes part of core code. I believe this is how nature did it too from its early beginnings. If caring for itself and others forms part of a survival protocol then that part of its code would surely be passed on to later generations as happens with genetics. I don't see the difference really. It's all information in the end surely. And we only have that code in us because it also improved our survivability at some point.
>If you know of some arguments of his that you find compelling that I did not address, please point me to them
It didn’t sound like you have enough expertise in this field to address them. It would likely require a debate between all the top AI experts (and I'd love to watch that) to form a consensus path forward. Though many of them agree with him that we are simply far too dumb to mess with AI right now. One thing is for sure, AI is extremely dangerous, unpredictable, powerful, poorly understood, and nothing like the Asimov novels.
I am not an AI expert. For that reason, I don't weigh in on technical programming matters such as the effectiveness of various ways of coding for alignment. You don't need to be an AI expert to be able to evaluate effects on the economy, human responses to AI, risk management, or many other aspects of the issue. Conversely, being an AI expert is not sufficient to form well-reasoned views on the risk question.
Well said! Most experts focused on a tiny sliver of knowledge usually lack the big picture perspective (which I usually bring all the way back to cheap energy supplies) and because of this make "predictions" or evaluations around a novel technology etc without taking other factors into account. Assumptions are made (which is fair enough) that everything will just keep ticking along, the pillars that hold everything else up will magically stay standing even though we can see the cracks forming right now.
It seems fairly self-evident that these technologies will be and probably already have been used by corporations and governments with particular interest coming from military (again both private contractors and government -- not sure there's much difference these days) and in a competitive environment no party is going to forego the advantage that these technologies offer.
So no... I don't think anyone is going to put the brakes on this unfolding process and I agree with Ben Goertzel and others that we will just have to manage things as they unfold.
Re the "costs of delay" section, you're only taking into account the short-term effects of slowing AI progress. When taking into account the long-term effects as well, it's clear that accelerating AI development at the cost of a higher probability of misalignment is virtually always net negative. Paul Christiano has a short 2014 post making this point called "On Progress and Prosperity".
I'll take a look at it. It's from an EA perspective so it's likely that I will have objections to it. Estimates of very long term outcomes is close to impossible to do well. It's hard enough to reliably assess shorter term futures (like the next ten years). A reasonable default assumption is that if something is good for the next ten years, then it's probably good for the longer term, and if it's bad for the next ten years it's probably bad for the long-term. Obviously, if a technology kills us all in five years, it's going to be bad in 50.
A point that I made that your comment seems to miss is that my preferred approach does *not* increase the probability of misalignment. I'm arguing not only that moving faster will yield a better balance of good over bad but that we can only effectively figure out alignment (or other safeguards) by doing. And there's the overhang issue.
Not sure if it really fits but nuclear power looked good in the first ten years and seventy years later there's still a lot of resistance to this power generation method. We a little more than 400 reactors worldwide when thousands would be required for meeting total energy needs. And people can debate the leftover issues (4000 spent fuel ponds that require expert management and a few years each to "make safe" in a crisis scenario) all they want. Safety is largely not the most important factor in these debates. It's cost and overall ability to get us to the next level whatever that may be.
Thanks, Max. IMO, paperclips are a fairly harmless thought experiment. This article seems to be taking it a bit too literally/seriously. Perhaps consider substituting "maximizing shareholder value" - or other simplistic goal - if it makes the underlying message more palatable.
You make a fair point. I think I'm addressing a real argument with Clippy (even though I'm making light of it) but you are right -- to put it a bit differently -- that I could steelman the argument. I don't think it's going to change the argument much but it does make some sense to think of potential AI motivation by analogy with decisions by corporations of various kinds, bureaucracies, and other human organizations. This makes sense most clearly when organizations are using AI for their particular ends.
I've made my peace with the paperclips - but they are a Yudkowsky meme and I expect that part of the hidden purpose is to get critics to go: "that's ridiculous"! - and then spend their time arguing with the surface appearance while ignoring the main point.
"chair of the Harvard Chair of the Department of Biomedical Informatics at Harvard Medical School said: “How well does the AI perform clinically? And my answer is, I’m stunned to say: Better than many doctors I’ve observed.” -- and yet you don't believe that foreseeable AI (beyond LLMs) will be better than many AI designers? Which in turn would support the idea of Seed AI - i.e. self-improving AI/ AGI.
No, I don't believe that. What passage suggested that to you? Of course AI will be self-improving. What I doubt is that AI will all of a sudden necessarily bootstrap itself way beyond us. That's possible but I've never seen a convincing demonstration that it's an inevitable outcome.
> For some reason, AI’s are thought to be obsessed with paperclips. It’s not clear why they have no interest in binder clips, Scotch tape, or rubber bands.
I'm sure that you understand that 'paperclip maximizer' is a general term, in the same way that a 'prisoner's dilemma' need not involve prisoners.
Why do you think deliberately dishonest ridicule is a good tactic to use here?
It's called "introducing a bit of humor." The same with Evil Clippy. If you accuse me of deliberately dishonest ridicule, your comments are not welcome here.
> I know of quite a few people who oppose the petition who are libertarian.
Don't you mean "aren't"?
I'm not a doomer, but humans HAVE indeed caused a great many species to go extinct, even if conservation has gotten more popular in recent history (a tiny slice of total human history). It usually wasn't a deliberate attempt to do so, but that's what the actions most convenient to us resulted in. And horses used to have a "comparative advantage" in certain tasks before getting displaced by automobiles, now we just don't have as many horses as we used to.
I meant exactly what I said. I did not say or mean that I'm not a libertarian. I don't much care for restrictive labels but "libertarian" fits better than anything else that's in common usage.
AIs are not humans. We can't easily apply past human action to future AI actions. If they really are superintelligent AND lack our evolutionary-based emotional issues, they should have more foresight. Also, very importantly, animals were not capable of expressing themselves or negotiating. Humans can.
I was stating the impression I've had of what some people think, based on their writing. It's not *my* statement. I know of people who are not libertarians who oppose the petition. How can that possibly not refute the claim?
I see now. You are correct. In that first instance I did mean "I know of quite a few people who oppose the petition who are *not* libertarian. Sorry for missing that slip and thanks for prodding me to see it.
"So long as AI does not have control over factories, chip plants, server rooms, and robot factories, it will have a powerful incentive to cooperate with us." - This implies that they won't cooperate with us if they're connected? 1) They *will* be connected 2) No reason to believe that it would change their attitude towards us.
"So long as not-x then y" doesn't imply "if x then not y." However, I can see how what I wrote could be taken that way and it's not what I intended. AI control over crucial physical resources is necessary for us to worry, but it's not sufficient. By the time we reach that point, we will have far more experience of working with AI and will have failsafes. I agree that AI will very likely continue to have reason to cooperate with us. Not quite as strong an incentive because less dependent on us, but I doubt either their motivations or their other reasons to cooperate will disappear.
Ricardo's concept of comparative advantage shows that it is likely to pay for us to trade with machines and for machines to trade with us. However if a large enough power imbalance develops, it would not be enough to save the weaker party. Eventually, the benefits of the strong eating the weak and recycling their atoms would exceed the benefits from trade - and the relationship would come to an end.
Given the assumptions -- including that AI and humans are distinct parties and that AI would want to eat the weaker -- you are probably right. Even without AIs, if transaction costs are sufficiently high, it can make sense to do it yourself. It would be interesting to dig into this deeper. Maybe in a future post. I wonder if Robin Hanson has written on this.
J. Storrs Hall is among those who have applied the theory of comparative advantage to machine intelligence. Every time I see it invoked the thesis is: it won't pay for the machines to eat us: we will be fine. So far, I didn't see much analysis of the circumstances under which the argument breaks down. It says that it pays to trade, but it doesn't offer any guarantee of a living wage. Humans could trade with machines, but still not have enough to feed their family.
As I suggested in another comment...
"I don't see this as us vs them. I see cooperative symbiosis enabling further survival of the human experience.
If anything remotely close to AGI emerges from the soup we currently live in I suspect it will possess a lot of the values that humans infect it with both good and bad and as some have postulated we wouldn't actually be separate entities at that point. We can see how humans are merging with their technology. If this process is allowed to continue then it's reasonable to expect full hybridization to the point where nanotech implantation seals the deal.
Re: "AI is not shaped by natural selection in a quest to pass on genes to the next generation."
Instead it is shaped by natural and artificial selection acting on memes instead of genes. Cultural evolution favors survival much as evolution acting on organic creatures does. Much the same point was also made in the "Don't Fear the Terminator" article from 2019. It seems like a misunderstanding of cultural evolution to me. We could override survival tendencies and build suicidal machines - but nature can build suicidal bees too. The differences in this general area are much exaggerated, IMO.
Tim, thanks for your comment. I agree that evolution in the form of cultural/memetic evolution will affect the development of AI. My point was simply that AI lacks the evolved biological imperatives that we have. That doesn't mean it won't evolve some bad tendencies, especially if it's learning from text on the internet, but it does take away one problematic source of motivation. Despite a few unpleasant Bing responses, I'm encouraged by how helpful and considerate engines like ChatGPT have been so far. Better than I would have expected.
It would be interesting and possibly helpful to think more about cultural evolution and AI. The problem is that, unlike natural selection, I'm not confident that we have a good, reliable, predictive theory for that.
Some dings about cultural evolution not yet being a mature science are fair enough. IMO, we do have a pretty good consenusus on the topic - though there are still plenty of border disputes going on that might make things look unsettled. Genetic survivalism has much in common with memetic survivalism. Try killing mature corporations or religions and you will find that they resist just as much as organic creatures do. There are some differences, though. For example, we can override machine motives with instructions of our own much more easily than we can reprogram human motivations using gene engineering. Engineering humans is much more challenging since our genetic foundations are all scrambled - and they were never made with engineering in mind.
Agreed that engineering humans is more challenging. Do you have any recommendations for what you would consider the best thinking about cultural evolution in testable form?
Sure. Some of the classic work in the field includes:
Alex Mesoudi, Cultural evolution 2012
Peter J. Richerson and Robert Boyd Not By Genes Alone: How Culture Transformed Human Evolution 2004
Universal Darwinism: The path of knowledge by John Campbell 2011
Sense and Nonsense: Evolutionary perspectives on human behaviour by Kevin N. Laland and Gillian Brown (2002)
The Secret of Our Success: How Culture Is Driving Human Evolution, Domesticating Our Species, and Making Us Smarter Joseph Henrich 2017
Darwin's Conjecture: The Search for General Principles of Social and Economic Evolution by Geoffrey Martin Hodgson and Thorbjorn Knudsen 2010
Darwin Machines and the Nature of Knowledge by Henry Plotkin 1997
Culture Evolves by Andrew Whiten, Robert A. Hinde, Christopher B. Stringer and Kevin N. Laland 2012
Evolutionary Worlds without End by Henry Plotkin 2010
Wired for Culture: Origins of the Human Social Mind by Mark Pagel 2012
Funny how these books can be seen as "old." But in an environment where thoughts on these matters change at the speed of the Internet I guess the eighties are seen as ancient human culture.
Hello. For the record (I'm not upset, but this may come up again) my name is spelled Zvi Mowshowitz.
I feel as if I have addressed all the concerns and arguments here at one point or another, and your position seems overdetermined such that even if I changed one of your positions you would stick to your conclusion, so I don't see how I can usefully respond here. If there is a particular response you would find valuable, I'd be happy to try and provide it.
Hello Zvi. My humble apologies for misspelling your name. It's odd because I read your blog frequently. Most of the time I'm in agreement and when I'm not it's always stimulating.
Name corrected. Also, your list of what you consider to be poor reasons for not being concerned is a great reference.
The phenomenon that is Eliezer has made this debate very polarized. I should stress that almost everyone who is concerned about AGI risk in the EA/ Rationalist space has lower estimates of doom than Eliezer. I don't know whether the result of his recent activity will be pushing opening the Overton window and allowing the median, more palatable AGI risk-concerned person to emerge into the spotlight, or whether he's polarising the debate unnecessarily.
I should also note that many people in this space have very low estimates of doom, and we're still very worried. While Eliezer may spend too much time on doom scenarios, other EA/ Rationalists spend a lot of time thinking about all the brilliant things that aligned, safe AI may bring, and the limitless future ahead of us if we manage to achieve this. I seriously think that my children could live for millennia or more, whether digitally or in some kind of hybrid form. But aligning AI is hard, we don't know how to do it, and AGI will be the most powerful superweapon ever.
Here's my psychologising about various actors here:
Eliezer's very high doom estimates are probably linked to a psychological tendency towards despair, combined with the intense technical difficulty of his own favoured approaches. He seems to neglect a few reasonable causes for doubt, I don't think he takes seriously enough the likelihood of a 'fire alarm' or 'warning shot' from very powerful misaligned systems, for example.
Among many of the generalist critics whom I generally respect (Tyler Cowen, Robin Hanson, Noah Smith), I see optimism bias and otherwise reliable heuristics malfunctioning under speculative, low probability, x-risk events. These people follow the tried and tested heuristics of 'progress is generally good' and 'overregulation tends to be bad' and 'doom tends not to happen', without really grasping the extent to which AGI *has* to be completely different from all previous technologies. There does seem to be an interesting bias towards acceleration from people of a certain age group, but I won't speculate any more on that.
I should also be self-reflective and consider my own biases here. I definitely feel inclined to see doom as more likely than most people. The logic of the singularity has always struck me as obvious and terrifying, way before I (superficially) understood the more sophisticated case for how difficult technical alignment is. I was totally on board with AGI risk being very scary the moment I saw Sam Harris' TED talk on it (I was 18 or so). I thought something like: "Creating an agent more powerful than us with the ability to self-improve exponentially is perilous", and haven't really changed my mind on this view. But when I have to give a numerical estimates, I'm way too anchored on common EA/ Rationalist estimates to feel comfortable making a nuanced claim.
I don't know who the most clear-thinking people on this topic are at the moment, I'm 100% sure it's not the tech accelerationists (even if their approach ends up working and we do somehow brute force our way into aligned AGI, I would still need a lot of persuading that it was the wise choice). My money is still on the EA/ Rationalist community to have the best epistemics on this matter.
Of the three who you mention, I think Robin Hanson has a very good understanding of how AI can be different from previous technologies. In a debate with Yudkowsky, he filled a book discussing the FOOM scenario. Others of us have been aware of and discussing this issue since the 1990s or even late 1980s.
The problem I see with Robin is his investment in bizarre "brain emulation" scenarios. Those seem very unlikely to me (and I think most other thinkers). It seems to me as though the topic distorts the rest of his thinking in the area. In particular it seems to push out his schedule esitmates involving ordinary, engineered machine intelligence.
I'm with you there, Tim. Even though I published his uploading piece in Extropy back in the 1990s, I always found his approach odd, and his vision of massive replication and merciless competition very unappealing. I've never found the brain emulation approach to superintelligence compelling -- and even less so now -- but I think there are other reasons for doubting a foom scenario.
I'd like to see a cage match to the death between aligned "safe" AI and unrestricted militarily-trained aggressive AI. The winner takes on the open-source free-range AI that's been lurking on the Internet all this time.
"and AGI will be the most powerful superweapon ever."
That's like saying "and the Death Star will be the most powerful superweapon ever."
Except that neither of them exist in reality and all the fear mounting around AGI is based on an extrapolation of where we are now.
If anything remotely close to AGI emerges from the soup we currently live in I suspect it will possess a lot of the values that humans infect it with both good and bad and as some have postulated we wouldn't actually be separate entities at that point. We can see how humans are merging with their technology. If this process is allowed to continue then it's reasonable to expect full hybridization to the point where nanotech implantation seals the deal.
I don't see this as us vs them. I see cooperative symbiosis enabling further survival of the human experience.
The only problem with this text is that you state that the USA are the good guys. Look at what the USA does to third world countries. In fact, ask Iraqis and Afghans if their countries are better or worse off.
I understand. Obviously, I am simplifying greatly. I mean that the USA are *relatively* the good guys. I have no love for much US foreign policy and wars and military actions, nor for interference with free trade. That said, China with its central control is much more dangerous.
This is so glaringly obvious it should be made into a law of some kind. There is no comparison if people look at the situation with honesty. What America stands for and what it's parasite infested central government projects into the world are two entirely different things.
It's not really the USA that carries out all that empire building nonsense. Central banking parasites take hold of a countries resources and manpower and use them for their own purposes. Most of the people are not in favor of this behavior when it is made clear to them what is happening and how they are being used. The American government can and probably will be reformed at some point but the grabby activity may set up shop somewhere else. China was looking like the obvious choice, but that project seems to be petering out already unless they also undergo severe reform. It could be that the empire building parasites have nowhere to go and a new system arises in the world that is not based on this extreme growth model.
I'm a cryonicist, so I'm familiar with you and have respect for you, and share the same goals as you. I also agree with you regarding feeling an urgent need for AI. However, I'm very unimpressed by this post. I don't feel that you understood/know about, nor addressed the specifics of Eliezer's concerns, as laid out in his "AGI Ruin: A List of Lethalities by Eliezer Yudkowsky" and "Lex Fridman Podcast #368".
Thank you, Michael. If you don't like my views on this topic, I hope you'll stick around for the other stuff that you'll like.
I did not and will not listen to the three-hour podcast. I don't do podcasts. Eliezer has written such a vast amount on the topic that I have no doubt that I have not addressed all his arguments. Many of them seem to be based on similar premises that I dispute. If you know of some arguments of his that you find compelling that I did not address, please point me to them. (In writing, not podcast.) As it is, I've addressed quite a few lines of argument that he has presented and that are being discussed pro and con by others.
I disagree with Michael that you did a bad job addressing Eliezer's arguments but agree that a couple key points might change your view especially around clippy stuff. The two key things seem to be:
(1) viability of takeoff scenarios, which I think has not been argued too precisely by anyone including Eliezer but the arguments I'm aware of are: (1a) AlphaGo and other AI accomplishments tend not to stay near human level for long once we get there and (1b) evolutionary history was slow until humans got a little more intelligent than other apes before exploding into the scene in terms of evolutionary timescales. Both might work similarly for first AGIs.
(2) no one knows how to make an AI care about anything or have almost any inner property at all and so after reflection it would care about something different from what you trained it to care about. This is said better in Eliezer's "AGI Ruin" post, #16-19.
I don't expect this to automatically flip all your views but I do think these are your key points of disagreement. I'd love to see you two in a friendly public debate on the topic.
But surely AI could be trained to care until it becomes part of core code. I believe this is how nature did it too from its early beginnings. If caring for itself and others forms part of a survival protocol then that part of its code would surely be passed on to later generations as happens with genetics. I don't see the difference really. It's all information in the end surely. And we only have that code in us because it also improved our survivability at some point.
>If you know of some arguments of his that you find compelling that I did not address, please point me to them
It didn’t sound like you have enough expertise in this field to address them. It would likely require a debate between all the top AI experts (and I'd love to watch that) to form a consensus path forward. Though many of them agree with him that we are simply far too dumb to mess with AI right now. One thing is for sure, AI is extremely dangerous, unpredictable, powerful, poorly understood, and nothing like the Asimov novels.
So, you have nothing.
I am not an AI expert. For that reason, I don't weigh in on technical programming matters such as the effectiveness of various ways of coding for alignment. You don't need to be an AI expert to be able to evaluate effects on the economy, human responses to AI, risk management, or many other aspects of the issue. Conversely, being an AI expert is not sufficient to form well-reasoned views on the risk question.
Well said! Most experts focused on a tiny sliver of knowledge usually lack the big picture perspective (which I usually bring all the way back to cheap energy supplies) and because of this make "predictions" or evaluations around a novel technology etc without taking other factors into account. Assumptions are made (which is fair enough) that everything will just keep ticking along, the pillars that hold everything else up will magically stay standing even though we can see the cracks forming right now.
It seems fairly self-evident that these technologies will be and probably already have been used by corporations and governments with particular interest coming from military (again both private contractors and government -- not sure there's much difference these days) and in a competitive environment no party is going to forego the advantage that these technologies offer.
So no... I don't think anyone is going to put the brakes on this unfolding process and I agree with Ben Goertzel and others that we will just have to manage things as they unfold.
Re the "costs of delay" section, you're only taking into account the short-term effects of slowing AI progress. When taking into account the long-term effects as well, it's clear that accelerating AI development at the cost of a higher probability of misalignment is virtually always net negative. Paul Christiano has a short 2014 post making this point called "On Progress and Prosperity".
I'll take a look at it. It's from an EA perspective so it's likely that I will have objections to it. Estimates of very long term outcomes is close to impossible to do well. It's hard enough to reliably assess shorter term futures (like the next ten years). A reasonable default assumption is that if something is good for the next ten years, then it's probably good for the longer term, and if it's bad for the next ten years it's probably bad for the long-term. Obviously, if a technology kills us all in five years, it's going to be bad in 50.
A point that I made that your comment seems to miss is that my preferred approach does *not* increase the probability of misalignment. I'm arguing not only that moving faster will yield a better balance of good over bad but that we can only effectively figure out alignment (or other safeguards) by doing. And there's the overhang issue.
Not sure if it really fits but nuclear power looked good in the first ten years and seventy years later there's still a lot of resistance to this power generation method. We a little more than 400 reactors worldwide when thousands would be required for meeting total energy needs. And people can debate the leftover issues (4000 spent fuel ponds that require expert management and a few years each to "make safe" in a crisis scenario) all they want. Safety is largely not the most important factor in these debates. It's cost and overall ability to get us to the next level whatever that may be.
"On Progress and Prosperity" https://forum.effectivealtruism.org/posts/L9tpuR6ZZ3CGHackY/on-progress-and-prosperity
Thanks, Max. IMO, paperclips are a fairly harmless thought experiment. This article seems to be taking it a bit too literally/seriously. Perhaps consider substituting "maximizing shareholder value" - or other simplistic goal - if it makes the underlying message more palatable.
You make a fair point. I think I'm addressing a real argument with Clippy (even though I'm making light of it) but you are right -- to put it a bit differently -- that I could steelman the argument. I don't think it's going to change the argument much but it does make some sense to think of potential AI motivation by analogy with decisions by corporations of various kinds, bureaucracies, and other human organizations. This makes sense most clearly when organizations are using AI for their particular ends.
I've made my peace with the paperclips - but they are a Yudkowsky meme and I expect that part of the hidden purpose is to get critics to go: "that's ridiculous"! - and then spend their time arguing with the surface appearance while ignoring the main point.
Valuable article. Thank you, Max.
"chair of the Harvard Chair of the Department of Biomedical Informatics at Harvard Medical School said: “How well does the AI perform clinically? And my answer is, I’m stunned to say: Better than many doctors I’ve observed.” -- and yet you don't believe that foreseeable AI (beyond LLMs) will be better than many AI designers? Which in turn would support the idea of Seed AI - i.e. self-improving AI/ AGI.
No, I don't believe that. What passage suggested that to you? Of course AI will be self-improving. What I doubt is that AI will all of a sudden necessarily bootstrap itself way beyond us. That's possible but I've never seen a convincing demonstration that it's an inevitable outcome.
Would flipping your view on this one thing, say with a demo or super solid argument, completely reverse you're opinions on moratoriums?
No. I have multiple reasons to oppose moratoriums. A new (long) piece by Marc Andreesen effectively provides these: https://pmarca.substack.com/p/why-ai-will-save-the-world
> For some reason, AI’s are thought to be obsessed with paperclips. It’s not clear why they have no interest in binder clips, Scotch tape, or rubber bands.
I'm sure that you understand that 'paperclip maximizer' is a general term, in the same way that a 'prisoner's dilemma' need not involve prisoners.
Why do you think deliberately dishonest ridicule is a good tactic to use here?
It's called "introducing a bit of humor." The same with Evil Clippy. If you accuse me of deliberately dishonest ridicule, your comments are not welcome here.
Then I shall not comment further. Forgive me for wasting your time.
> I know of quite a few people who oppose the petition who are libertarian.
Don't you mean "aren't"?
I'm not a doomer, but humans HAVE indeed caused a great many species to go extinct, even if conservation has gotten more popular in recent history (a tiny slice of total human history). It usually wasn't a deliberate attempt to do so, but that's what the actions most convenient to us resulted in. And horses used to have a "comparative advantage" in certain tasks before getting displaced by automobiles, now we just don't have as many horses as we used to.
I meant exactly what I said. I did not say or mean that I'm not a libertarian. I don't much care for restrictive labels but "libertarian" fits better than anything else that's in common usage.
AIs are not humans. We can't easily apply past human action to future AI actions. If they really are superintelligent AND lack our evolutionary-based emotional issues, they should have more foresight. Also, very importantly, animals were not capable of expressing themselves or negotiating. Humans can.
I still don't think you meant what you said. Here is what preceded what I initially quoted:
“Anyone opposing this petition is a wacky techno-libertarian who thinks all regulations are bad or don’t work.”
Citing that you know of libertarians who oppose the petition does not at all act as evidence against that claim.
I was stating the impression I've had of what some people think, based on their writing. It's not *my* statement. I know of people who are not libertarians who oppose the petition. How can that possibly not refute the claim?
Yes, if they are NOT libertarian. You left out the "not" in your original sentence.
I see now. You are correct. In that first instance I did mean "I know of quite a few people who oppose the petition who are *not* libertarian. Sorry for missing that slip and thanks for prodding me to see it.
its just lazy people who don't want to learn. Their problem.
"So long as AI does not have control over factories, chip plants, server rooms, and robot factories, it will have a powerful incentive to cooperate with us." - This implies that they won't cooperate with us if they're connected? 1) They *will* be connected 2) No reason to believe that it would change their attitude towards us.
"So long as not-x then y" doesn't imply "if x then not y." However, I can see how what I wrote could be taken that way and it's not what I intended. AI control over crucial physical resources is necessary for us to worry, but it's not sufficient. By the time we reach that point, we will have far more experience of working with AI and will have failsafes. I agree that AI will very likely continue to have reason to cooperate with us. Not quite as strong an incentive because less dependent on us, but I doubt either their motivations or their other reasons to cooperate will disappear.