I think this is a good point — I’ve also seen a variety of responses to these arguments, and I don’t even think all the ones I proposed hold so much weight after the responses. I would like to make clear a few things, though:
1) The point of this post, as stated at the beginning, is not to produce novel arguments that lack response. Rather, the point is to consolidate the AI optimist side’s points as has been done much more (in my opinion) on the AI worried side.
2) Each argument, even with a solid rebuttal, should still change people’s probabilities. Nobody has 100% probability in their counter arguments being true, and I actually think people typically overweight the ability to ignore an argument because they found a satisfying counter argument.
3) Arguments going back and forth can be done for days on every topic, and one can go up so many levels. An argument can be made against almost any claim. As stated, I think the main things to be worried about are initial arguments and probably second responses in most cases. The back and forth on a single point really do very little in my opinion.
I think that counter arguments are important and can make the difference between “consideration is worth mentioning” and “consideration is not worth mentioning.” Repeating considerations that have been determined to be not worth mentioning is misleading.
You did a great job at cataloging the optimist arguments.
Perhaps you have knowledge that I don’t. These are the arguments that I think have been mentioning despite counter arguments made. This should just give greater reason for you to post some of the best counter arguments to at least some of the arguments — maybe I haven’t been looking at the right places on LessWrong!
Feels like you can effectively refute all of the arguments against AI risk simply by plugging it into Sonnet 3.5, which does an excellent job arguing for AI safety.
This is a very thoughtful and comprehensive list of anti-AI doomer arguments. I’ll probably be referencing it in the future. Thanks for pulling this together.
I agree with MatthewK that basically all this stuff has already been addressed (in a way where I haven't seen sensible replies from the other side). However, a piece of the puzzle that the AI optimists don't seem to consider is time.
Suppose for the sake of argument that there is a chance we get superintelligent AI at some point (as even many AI skeptics do). Suppose also that there is a chance of this AI (or any of the ones that may follow) being a serious risk for humanity. Clearly, then, we should do some work to ensure that if this scenario occurs, we're prepared and humanity is not in danger from the superintelligence. But how much time and resources does it take until we're capable of creating AI without risking catastrophe? 10 years? 50? 100? If you think basically *solving* AI safety just takes 10 more years and we'll only get AGI in 50, we're golden. If the former takes 100, we better have a backup plan. But in any case, given these premises, you can't seriously discuss the urgency of AI safety without taking into account the difficulty of the problem.
As it stands, AI safety appears to be a very difficult technical problem. And unless we get it solved before(!) we build AGI - whether that's in 1 or 100 years - we have quite a bit to worry about. So in a sense, it shouldn't make a huge difference to your expectations based on the optimistic arguments if we e.g. expect AGI to arrive 20 years later, or if the probability of catastrophe is only half as much. It's the order of [AI safety solved] vs. [AGI solved] that matters the most. So, as an optimist, one should compare one's AI safety timeline (after reading up on why all the approaches you think solve the issue trivially actually don't) to one's AGI timeline. *Then* you can consider whether worrying is in order or not.
P.S.: I do understand that this post was primarily intended to collect the optimistic arguments. You did a good job at that.
I agree with your time point -- I find that point to be quite compelling, in general, but longer timelines would give us more time for safety infastruture, technical safety work, ect.
I guess the only point I have against would be the point about long-winded arguments (which I think may survive many of the rebuttals I’ve seen) and partial alignment/ meta preferences (I don’t find the rebuttals there super compelling but send me some you’ve seen!).
It's true that adding in further premises decreases the probability multiplicatively. However:
1. When there are multiple alternative paths leading to a conclusion, only one of which needs to succeed for the conclusion to follow, the probabilities of the branches get summed up. This can quickly pump the probability back up. But it depends on the argument whether it uses such a structure or not, of course.
2. Claims about necessity are very strong in general. You are essentially saying: The ONLY way for event X to occur is for condition C to be fulfilled (among others). Thus, you need to be pretty darn sure that the condition really is necessary, or else you're a requirement that paints an incomplete picture. As an example:
- You write that we'd need to fix the shrinking amount of fresh training data. But we could also develop methods which use the existing training data more efficiently (as humans clearly do). (And approaches like reinforcement learning generate their own training data through interactions with the environment.)
- You write that AI training may eat up an unacceptable amount of energy. But we know that you can reach human level intelligence using a comparatively miniscule amount of energy. Again, greater efficiency could provide a solution without any kind of fight with the government or public.
- You write that a shutdown button would be an effective safeguard. But of course a superintelligence would do it's best to not be shut down (since that would hinder its goals) by whatever means necessary. So the shutdown button would need to remain usable and accessible in the critical case.
1. Good point. I’m not sure it brings it back up all the way (would be a really hard question to get even a practical range of probability on), but I think this point makes sense.
2. Yea, I think there are various levels to these with few or none being absolute. Perhaps I overestimated the weight of the things that definitely need to happen in order for AI to scale up - though these seem like general requirements in most scenarios (not logically deductive arguments); for almost any statement that claims that something needs to happen, it is more a claim that this needs to happen in the vast majority of circumstances.
Still, even with the things that have to happen only in a majority of scenarios, it seems like the probability would be greatly diminished, and we should lower our credences pretty far.
But let me know why you think I’m wrong or might be wrong, if you do.
OK, I've found a Rob Miles video referencing the partial alignment case. The proper term for the idea seems to be the Value Learning Problem. Relevant part starts at 4:18
This seems related but not precisely what I’m talking about. The general idea there seems to be that we can basically teach AI our values. The general point I am making is that, even If we don’t teach AI precisely our values, it would be partially (on the degree point -- I have zero idea - like you said, sounds like technical problem that is well above my pay grade lol) aligned by default because of the data we’re feeding it on (which gives it some context of what we want) and the goal we give It to optimize for (especially in cases where this the goal given is done with some forethought).
The partial alignment case is a good point, I don't remember the discussions about it. I guess the issue remains of a technical nature - how do you ensure the Ai really does "try to figure out what humanity wants and then act in accordance with that", which would be the goal we want it to have, as opposed to some different goal? We'd also have to ensure it's incentivized to figure out when it's got a wrong idea (since it will presumably only get negative reward if it finds out that it was the wrong idea - thats not a scenario we want to be in).
I feel like:
AI-Worried make some arguments.
AI-Optimists make some arguments.
AI-Worried address the optimist arguments.
AI-Optimists don’t respond.
You’re on level one. I’ve seen responses to all your arguments. I’d like to see the next level, please.
But do feel free to plug any resources that you think are good responses to these arguments. I (and I’m sure many others) would appreciate them!
I wish I had time to find the exact page for each counter argument. They are all somewhere here: https://www.lesswrong.com
I think this is a good point — I’ve also seen a variety of responses to these arguments, and I don’t even think all the ones I proposed hold so much weight after the responses. I would like to make clear a few things, though:
1) The point of this post, as stated at the beginning, is not to produce novel arguments that lack response. Rather, the point is to consolidate the AI optimist side’s points as has been done much more (in my opinion) on the AI worried side.
2) Each argument, even with a solid rebuttal, should still change people’s probabilities. Nobody has 100% probability in their counter arguments being true, and I actually think people typically overweight the ability to ignore an argument because they found a satisfying counter argument.
3) Arguments going back and forth can be done for days on every topic, and one can go up so many levels. An argument can be made against almost any claim. As stated, I think the main things to be worried about are initial arguments and probably second responses in most cases. The back and forth on a single point really do very little in my opinion.
But let me know where you disagree with this.
I think that counter arguments are important and can make the difference between “consideration is worth mentioning” and “consideration is not worth mentioning.” Repeating considerations that have been determined to be not worth mentioning is misleading.
You did a great job at cataloging the optimist arguments.
Perhaps you have knowledge that I don’t. These are the arguments that I think have been mentioning despite counter arguments made. This should just give greater reason for you to post some of the best counter arguments to at least some of the arguments — maybe I haven’t been looking at the right places on LessWrong!
Fair. I’ll put it on my todo list.
Thanks! Whatever you can do would be great! Also don’t do it at the bottom of this threat because it would be hard to see!
Feels like you can effectively refute all of the arguments against AI risk simply by plugging it into Sonnet 3.5, which does an excellent job arguing for AI safety.
This is a very thoughtful and comprehensive list of anti-AI doomer arguments. I’ll probably be referencing it in the future. Thanks for pulling this together.
I agree with MatthewK that basically all this stuff has already been addressed (in a way where I haven't seen sensible replies from the other side). However, a piece of the puzzle that the AI optimists don't seem to consider is time.
Suppose for the sake of argument that there is a chance we get superintelligent AI at some point (as even many AI skeptics do). Suppose also that there is a chance of this AI (or any of the ones that may follow) being a serious risk for humanity. Clearly, then, we should do some work to ensure that if this scenario occurs, we're prepared and humanity is not in danger from the superintelligence. But how much time and resources does it take until we're capable of creating AI without risking catastrophe? 10 years? 50? 100? If you think basically *solving* AI safety just takes 10 more years and we'll only get AGI in 50, we're golden. If the former takes 100, we better have a backup plan. But in any case, given these premises, you can't seriously discuss the urgency of AI safety without taking into account the difficulty of the problem.
As it stands, AI safety appears to be a very difficult technical problem. And unless we get it solved before(!) we build AGI - whether that's in 1 or 100 years - we have quite a bit to worry about. So in a sense, it shouldn't make a huge difference to your expectations based on the optimistic arguments if we e.g. expect AGI to arrive 20 years later, or if the probability of catastrophe is only half as much. It's the order of [AI safety solved] vs. [AGI solved] that matters the most. So, as an optimist, one should compare one's AI safety timeline (after reading up on why all the approaches you think solve the issue trivially actually don't) to one's AGI timeline. *Then* you can consider whether worrying is in order or not.
P.S.: I do understand that this post was primarily intended to collect the optimistic arguments. You did a good job at that.
Thanks and thanks for your helpful comment!
I agree with your time point -- I find that point to be quite compelling, in general, but longer timelines would give us more time for safety infastruture, technical safety work, ect.
I guess the only point I have against would be the point about long-winded arguments (which I think may survive many of the rebuttals I’ve seen) and partial alignment/ meta preferences (I don’t find the rebuttals there super compelling but send me some you’ve seen!).
Regarding long-winded arguments:
It's true that adding in further premises decreases the probability multiplicatively. However:
1. When there are multiple alternative paths leading to a conclusion, only one of which needs to succeed for the conclusion to follow, the probabilities of the branches get summed up. This can quickly pump the probability back up. But it depends on the argument whether it uses such a structure or not, of course.
2. Claims about necessity are very strong in general. You are essentially saying: The ONLY way for event X to occur is for condition C to be fulfilled (among others). Thus, you need to be pretty darn sure that the condition really is necessary, or else you're a requirement that paints an incomplete picture. As an example:
- You write that we'd need to fix the shrinking amount of fresh training data. But we could also develop methods which use the existing training data more efficiently (as humans clearly do). (And approaches like reinforcement learning generate their own training data through interactions with the environment.)
- You write that AI training may eat up an unacceptable amount of energy. But we know that you can reach human level intelligence using a comparatively miniscule amount of energy. Again, greater efficiency could provide a solution without any kind of fight with the government or public.
- You write that a shutdown button would be an effective safeguard. But of course a superintelligence would do it's best to not be shut down (since that would hinder its goals) by whatever means necessary. So the shutdown button would need to remain usable and accessible in the critical case.
Etc.
1. Good point. I’m not sure it brings it back up all the way (would be a really hard question to get even a practical range of probability on), but I think this point makes sense.
2. Yea, I think there are various levels to these with few or none being absolute. Perhaps I overestimated the weight of the things that definitely need to happen in order for AI to scale up - though these seem like general requirements in most scenarios (not logically deductive arguments); for almost any statement that claims that something needs to happen, it is more a claim that this needs to happen in the vast majority of circumstances.
Still, even with the things that have to happen only in a majority of scenarios, it seems like the probability would be greatly diminished, and we should lower our credences pretty far.
But let me know why you think I’m wrong or might be wrong, if you do.
OK, I've found a Rob Miles video referencing the partial alignment case. The proper term for the idea seems to be the Value Learning Problem. Relevant part starts at 4:18
https://youtu.be/eaYIU6YXr3w?si=GHfAA_TT4nXPv-di
Awesome! I haven’t seen that; will check it out.
This seems related but not precisely what I’m talking about. The general idea there seems to be that we can basically teach AI our values. The general point I am making is that, even If we don’t teach AI precisely our values, it would be partially (on the degree point -- I have zero idea - like you said, sounds like technical problem that is well above my pay grade lol) aligned by default because of the data we’re feeding it on (which gives it some context of what we want) and the goal we give It to optimize for (especially in cases where this the goal given is done with some forethought).
The partial alignment case is a good point, I don't remember the discussions about it. I guess the issue remains of a technical nature - how do you ensure the Ai really does "try to figure out what humanity wants and then act in accordance with that", which would be the goal we want it to have, as opposed to some different goal? We'd also have to ensure it's incentivized to figure out when it's got a wrong idea (since it will presumably only get negative reward if it finds out that it was the wrong idea - thats not a scenario we want to be in).