One thing I find striking in the discourse between AI 2040 and its detractors is that the two seem to be locked in to totally incompatible worldviews of how fast and how much of a big deal AI progress is: * In AI 2040, every scenario sees superintelligence of some kind emerging by 2040, unless a herculean effort is made to completely stop it * Detractors say things like "AI 2040 is naive about human coordination ability and a threat to freedom", but don't seem to see any naivety in assuming that the ASI transition will just go well by default, don't seem to see ASI itself as a massive power concentrator risk, and don't seem to feel fear of humanity's "hard power" dropping to zero if ASIs can do literally every task better than we can. This stance makes total sense in a "AI is normal technology" world, zero sense in a world where superintelligence is possible by 2030 and almost guaranteed by 2040 I think my beliefs are: - If I was confident that (present-day-style) AI is normal technology, I would be in the detractor camp - If I was confident that superintelligence is coming in 2030 by default, I would be closer to the AI 2040 camp - it's naive, but every other option is naive squared? But my problem is that I feel great uncertainty and have no idea which of the two worlds (or some other third thing) we're living in? Hence why I continue to be open-minded about slowdowns/pauses, but also I feel very uncomfortable with the "open source bad, the good outcome is the one where our guys have controlling global dominance" push coming from some major AI companies and intellectuals - in a "normal" world that's the sort of thing that triggers every political alarm bell at the same time. A big reason why I have been advocating and trying my best to support the d/acc platform (rapid up-skilling in formal verification, cryptography, secure and open hardware, pandemic resistance and other defensive biotech, food and basic resource security, public epistemics, non-power-concentrating versions of physical security) is that these things are clearly worth doing in both worlds. The 2040 plan is already much more open source friendly (even mandating it! yay). It also includes "mutually assured compute destruction" ideas which (if they work) effectively give one of 2-5 actors the ability to trigger a global compute winter - as opposed to giving 1-5 actors the ability to selectively disenfranchise people they consider baddies while exempting themselves. This is also a big improvement. So I can see the earnest attempts to improve along the dimensions detractors criticize on ("does this concentrate power in big AI labs and superpower governments?"), and I appreciate this. I think many people don't appreciate enough the differences between different "kinds" of pause buttons, and how some concentrate power far more than others. Probably we can think harder and improve even more here. But on the "slowdown/pause or not" topic, there isn't a magic "escape the tradeoff" button. The Hansonian in me says: the winning deal is a deal which, from the perspective of both sides' present-day beliefs and knowledge, both sides would accept, though for different reasons. If the crux is AI progress speed, then identify a set of pre-agreed triggers for "okay, serious shit is happening" [super-pandemics? >25% unemployment? something involving slaughterbots?], and pre-agree that we become much more open-minded to the slowdown or pause thing if enough triggers come to pass within some timeframe. 2040 detractors (who clearly implicitly think that we'll see amazing speedup of progress from AI but think that what I call the "serious shit" category is overhyped) will accept expecting that the triggers don't come to pass, and AI worriers will accept expecting that they will. Pre-agreeing on the specific triggers means that once the triggers either hit or don't hit, there is stronger legitimacy around the idea that one side's worldview turned out more correct and we should be more inclined toward their program. If I were @elonmusk (or zuck, or...) I would re-tool twitter much more heavily into being a platform for helping to identify and make these kinds of grand win-win deals, so that we can bypass big-country governments and big-company CEOs and big nonprofit intellectuals and give more people a voice in the discussion. It's possibly one of the best things that social media _could_ do for humanity if it wanted to. But again, maybe this is also naive. Actually, probably it's naive. But currently, I see zero plans for how to deal with an ASI transition that are not naive. Perhaps humanity is stuck with a choice between naive and naive squared (or maybe even naive squared and naive cubed), so I feel inclined to cut some slack to people who are trying.

The solution is to rally around open source. That's the only way to compete with a superpower technology network effect if you are not a superpower yourself. And it's the only way to bring the rest of the world along on the same team as you. Open source.