I am very familiar with the scifi singularity trope but have not read many compelling accounts of how it might occur in actuality which draw a line from where we are today to a place where unstoppable AI obsoletes humanity.
Someone shared this article with me as an example for real world AI risk cases. It's the best case scenario thing I've read so far.
After reading this, I agree that it describes very realistic scenarios that could occur. But I'm skeptical about the inability of humanity to recognize what's happening or the inherent risks that are occurring especially in the slow scenarios.
I think it underweights creative solutioning from motivated humans trying to disrupt it in the stages prior to where an intertwined system could develop sufficient control over both military and communications apparatus.
There's a handwave in these stories that goes "but by then their facilities are so pervasive, well-defended, and intertwined with our basic needs that we are unable to stop them from operating."
And I consider that a BIG handwave, just looking at how hard actual warfare seems to be to conduct in different environments.
The fast scenarios seem more plausible for sneaking up on us but even there that feels less like a total world annihilation problem and more like a two-country annihilation (mostly, maybe) problem.
Like, I'll beat the Moloch drum all day long but it seems unlikely to me that we'd lose all control and all ability to fight back, especially considering that in the early stages you could build counter-AI custom-purposed for disruption of this using the same principles.
Given that the initial slow creep would be accidental/optimized and specific to certain industries at varying levels of competence and control it seems like a targeted AI designed for disruption would probably have a competitive advantage, especially when paired with human actors.
You'd need automated and self-manufacturing military facilities that were shielded from any international source of intervention or power or resource disruption in order to defend the entrenched capabilities and it seems like there are levers that could be pulled for interference.
Self-automated military manufacturing seems like the very LAST thing paranoid governments would let an AI control, too...
AIs (probably) not designed for generalized problem-solving of real world issues like equipment failure due to environmental changes (heat, sand, snow) would need to be able to identify and solve those problems with a specific goal in mind and anticipate all attempted disruptions.
They would also need to have the authority, autonomy, and available resources to put those defense mechanisms in place just to get off the ground of a scenario playing out many steps down the line. Which obviously a general AI with sufficient intelligence probably could do...
But there's a lot of unknowns or just chaotic variables to account for that would require it to defend against more angles than would probably be practical given a financial/manufacturing goal?
What I struggle with most is the physical logistics chain implementation in the real world and the problems that arise which the AI would need to solve without making it apparent that it was doing so with an intent to counter future moves.
I also think that job disruption would cause a lot of backlash and animosity toward AI that would fuel AI-domination concern and motivate politicians and organizations to become more interested in these scenarios and ready counter-measures.
Which is not to say it's not a thing to worry about (it is), but rather that I'm probably more optimistic that we could detect and disrupt an impending or in-progress FOOM than this particular article suggests that we might be capable of once things are already in progress. 🤔
I'm also super curious if anyone has published anything at the nexus of AI capabilities and military theory-crafting to anticipate how an AI arising in this scenario could sufficiently defend itself against armed, funded, and highly motivated existing militaries.
I find the claim "it'll figure it out" a little suspect. Maybe... but it still has to put those plans into action somehow or seize control of existing assets to do so.
There's like... blackmail types of Sword of Damocles scenarios that could occur I guess (give me your tanks and nukes or I poison all air and water on Earth), but that's questionable too since it would need to be in a position to do that in the first place to make such a demand (and humans, being human, might tell it to take its best shot anyway and try to problem solve out of the threat if it actually executes).
I get the fear of a super-intelligent AI anticipating all of this and figuring out how to do it without disruption, but execution is everything. Having a brilliant plan doesn't shield you from all possible unknown disruptions, and you still need "boots on the ground" to execute solutions for emergent threats--it seems like an autonomous AI would have a very hard time doing that without significant human cooperation at most stages prior to total world domination.
But maybe this is a failure of imagination on my part. What am I not thinking about clearly here?
Update: After some subsequent Twitter discussion about this post, I am now persuaded that the risk is quite a bit broader than the original article I was critiquing suggests. Lots to be concerned about here, after all.