“The problem is not the systems that exist today. The systems that exist today are already bad enough to create Black October, but it can get only worse from here.” Our US Executive Director @NPCollapse on the growing threat posed by ever more autonomous AI systems, on CBS. “The most important thing for citizens to do is to contact their lawmakers and tell them that this is not acceptable.”

It’s clear that we must put an end to the dangerous race to superintelligent AI. This should be accomplished by banning its development through an international trust-but-verify regime.

The inability of developers to understand or control the behavior of the AIs they’re building, combined with their rapid pursuit of superintelligent AI, is why top AI scientists are warning that humanity is faced with a risk of human extinction.

This has some effect, but as we’ve seen, they can still be misused. In the case of open-weight AIs, developers have essentially no ability to apply such mitigations, as they can be, and are, trivially removed.

AI developers don’t want cybercriminals to take their models and use them to cause damage, and in fact, they take measures specifically to try to prevent this: putting AIs through further training in an effort to make them less likely to assist with harmful requests.

One would be that of AIs escaping control, the other of bad guys controlling AIs. But these issues actually have the same root, which is that developers can’t control their systems.

The second piece here is that AI developers don’t know how to control or ensure the safety of the systems they’re building. It is tempting to understand the problem of rogue AIs and the use of powerful AI agents by cybercriminals as two distinct issues.

As these AIs become more capable, new abilities, like the ability to find flaws in computer systems, fall out in unpredictable ways. The AI companies aren’t especially interested in developing nation-state-level cyberweapons; it’s happening as a side effect of the AIs improving.

There are two main things to understand here. The first is that AI companies are in a race to develop ever more powerful AIs, with the goal being to create superintelligent AI, fully autonomous AI systems smarter than any human and acting without human direction.

Importantly, the flaws found in PHP aren’t an isolated case. Just eyeball the graph: the number of such flaws being found in software is staggering. This is increasingly being driven by AI.