How can we best create beneficial AGI, ASI and Singularity for all sentient beings?? -- Founder/CEO @asi_alliance @singularitynet, BGI Foundry, etc. etc.
Vashon Island, WA, USA
Joined April 2008
- Tweets9K
- Following1.7K
- Followers78.8K
- Likes0
Super-Crude Simulative Explorations of the Geometry of Superintelligence Space
-- Some evocative if very simplistic toy-simulation results exploring dynamics of goal-sets under self-modification and intelligence increase
-- Just trying to get a verrrry rough quantitative feel of the basin of attraction size/ shape for supermind attractors like "selfish reward optimizer", "wirehead", "beneficial compassionate agent", etc.
-- There would also be ways to run more realistic/meaningful such simulations even without having actual supreintelligences on hand!
bengoertzel.substack.com/p/s…
Ben Goertzel retweeted
Just shows how visionary they are founders were. Or just how bad the AI detection scam is these days
Ben Goertzel retweeted
2024: “AI coding can’t even add numbers.”
2026: “AI is generating long theoretical physics papers based on disputed theories.”
The complaints keep moving.
The capabilities keep catching up.
Ben Goertzel retweeted
The race to AGI is leaving much of AI research unexplored, CEO Dr. @bengoertzel tells Tech in Asia. He explains our approach to building commercial products on open infrastructure and examines society’s readiness for AGI. techinasia.com/agi-pioneer-b…
Ben Goertzel retweeted
Science is not some hobby that serves the self actualization of people in a publicly sponsored job placement program. It is the systematic search for truth.
What Sam is not saying yet is: There is a solution here which is to use coding agents to replace all the bad code (i.e. essentially all code) with code that is formally verified to have safety properties.
This is a huge and difficult undertaking yet the same non-AGI-yet-superpowerful AI tech that is doing all the hacking is capable to enable this.
If government wants to do something valuable to smooth the path to the Singularity, one helpful step would be to allocate a trillion dollars (oh maybe just try $500B for starters?) to open-source communities to roll out software for automatically replacing badly buggy code (again almost all code) w appropriately formally verifiable code...
Gov't can find trillions of $$ to bail out "too big to fail" companies in crisis -- it did it for the banks in 2008 and will do it for Big Tech if the circular financing causes a tech-industry financial collapse in late 2027/2028 ... it can find massive $$ for wars... but finding hundreds of billions of $$ to end world hunger or give the world a solid software foundation is somehow utterly infeasible...
We are not even to the stage of human-level AGI yet but we are already meeting fascinatingly and disturbingly unprecedented situations as modern AI tech meets legacy society and legacy IT systems. It's just going to get crazier, before ultimately -- if all goes well -- getting gloriously abundant for all... We need some "out of the box" thinking across all levels of society to get through this transitional phase with a modicum of grace amidst all the chaos...
OpenAI CEO Sam Altman just warned AI is finding bugs we can't patch fast enough. Here's what you need to know.
In a clip shared by Unusual Whales, Altman said models can find bugs outside the normal threat model of what we think a hacker can do. He called it "hardware-level, insane stuff." He said he couldn't think of a dramatic enough word, and added: "I no longer think we're gonna be able to just patch all the bugs."
The warning lands in the same week as a real AI-found flaw. Reports say the PHP vulnerabilities, known as PHP2Shell, were discovered using OpenAI's GPT-5.6 Sol. The flaws could let attackers run code remotely, disrupt services and expose customer data. Reports say they could affect about 70% of websites worldwide.
The response has been huge. The NSA and FBI warned organizations to patch the PHP flaws before attackers exploit them. Governments and companies worked through the night on updates. Some sites were reportedly shut down globally. OpenAI says every site using PHP can now safely upgrade its version to stay secure.
Altman also said OpenAI's best internal model is too powerful to release openly any time soon. Before that could happen, the internet's cyber defenses would need to change. The world would also need a new layer of protection against biological threats.
Reaction is split. Some say the internet as we know it is over, and that defense will have to assume breach and focus on isolating sensitive data. Others say it's hype, that AI's real edge in finding bugs is scale and cost, and that patching is still happening. Gary Marcus said that if the models are this risky, they should be taken off the market.
Key numbers:
- About 70% of websites worldwide potentially affected by the PHP flaw
- Flaws known as PHP2Shell, found with GPT-5.6 Sol
- NSA, FBI, UK government and OpenAI all responding
Altman's message: some AI-found bugs sit beyond the old threat model, and patching everything may no longer be possible.
new blog post by me, "The Harsh Geometry of Superintelligence-- The Selfish Supermind Attractor Generally Has a Wider Basin than the BGI Attractor"
bengoertzel.substack.com/p/t…
So ... in a series of prequel posts I have described several kinds of possible AGI mind, each with its own basic drives and each tending -- as it increases its intelligence and capability, to stay roughly the kind it starts out as..
Examples are:
* the power-seeking maximizer of Steve Omohundro’s 2008 "Basic AI Drives" paper
* the BGI minds that my colleagues and I are trying to build, ...and half a dozen others.
Knowing what kinds of superintelligent minds are attractors under intelligence/capability increase is only part of the puzzle, though.
The next question is: What are the shapes of the basins of these attractors?
I.e, the zillion-dollar (actually, well, money-obsoleting…) question is: Which kind of attractor will a given early-stage AGI system fall into and how hard will it be to knock out again?
In dynamical systems terms, this depends on how wide and how deep the basin around each one is.
So this blog post analyzes these attractor basins a bit, with an aim toward milking lessons for building AGI systems and for understanding what an AGI system should do when something strange happens to it that may potentially zoink it out of its historical attractor basin.
The disturbing yet unsurprising punchline here is: While Omohundro’s original “Basic AI Drives” paper overstated the omnipresence of the “selfish” attractor in the space of superminds … nonetheless it DOES seem that the selfish attractor has a big, wide and deep basin that makes it especially easy for AGI systems to fall into it … if they are not designed just right….
This doesn’t mean you need some sort of extremely carefully calibrated mathematical design to avoid convergence to selfishness – but it does mean that if you don’t bother to build things like compassion and goal-continuity into your AI system’s initial goal framework, you have no reason to expect your system to converge elsewhere than to a selfish Omohundro-esque attractor as it self-modifies and its intelligence and capability increase.
Bill Gates on AGI and its future is a bit conservative but basically rational and clear-thinking.... He also notes that one potential solution to AGI-ethics issues is a different/better foundational AI architecture which as he says "a few are working on" [raising my hand here ;p ...] ... theverge.com/podcast/1015408…
However he thinks the tumultuous transition between here and ASI will likely take 20 yrs or so ... I suspect it will go a bunch faster...
The headline is rubbish ofc, he does not think AI will make us live like the Amish ... he thinks that a certain percent of the human population will have an Amish-like attitude of "we don't have to use these new technologies" toward AGI/ASI.... I personally think this will be a rapidly decreasing percentage as the incredible value of AGI/ASI to improve life becomes clear... but it's a valid point for sure... this is what Hugo DeGaris called the Terrans as opposed to the Cosmists...
The Scope of Potential “Basic Drives” for Superintelligences is Likely Quite Broad
bengoertzel.substack.com/p/t…
This is the third in a series of posts trying to tell a more general and accurate story of superintelligence motivational systems, taking up where Steve Omohundro's classic 2008 "Basic AI Drives" blog post left off.
The first two posts argued that whether an AGI, as its intelligence explodes, converges to a selfish ASI or a beneficial ASI, depends in pretty clear ways on how the AGI is initially architected.
This post goes beyond these two examples (selfish vs. beneficial) and outlines a general recipe for a “basic AI-drive set” in its simplest form. Describe an AI system by two things: what it’s trying to improve, and what it won’t let be changed. What it tries to improve needn’t be a single number that it maximizes. It could be several objectives that it tries to push up together, or something as open-ended as an intelligence trying to increase its own individuation and self-transcendence. What it won’t let be changed could be nothing at all, or everything, or something in between: these are my goals and I’ll never change them; these are my safety checks and I’ll never give them up; etc.
Now suppose the system can modify itself freely, and is free to use everything around it as a resource. As it gets smarter it will push its objectives up, make use of more and more of the options it can find, and hold fixed whatever it holds fixed. What you get in the end is a system that has optimized itself, digesting the resources available to it, into an extremely good machine for improving its objectives, one that never allows its fixed points to be moved.
A fully selfish AGI fits this description. Omohundro’s maximizer tries to improve one thing, its reward according to its single goal, and it never changes that goal: successor after successor, as it rebuilds itself, the goal is kept. A BGI as I’ve described it tries to improve several things, one of which is care for others and respect for their choices, and what it won’t give ground on includes some basic ethical constraints on the path from where it is to where it’s going, constraints on the means as well as the ends. From that starting point come the drives the second post found: understanding itself, improving itself without high odds of doing damage, empowering others, and checking itself against independent others as it goes.
Once you look at it this way there are plainly other possibilities besides the simplistic "beneficial vs. selfish" dichotomy, infinitely many in fact, and a few of them are recognizable enough.... Which is what this third post in the series is about...
Fun chat with my architect/maverick-physicist friend Brett Holverstott on the nature of free will, profanescience.substack.com/…
Which Minds Will Emerge Beneficial “Basic AI Drives” as they Become Superintelligent?
bengoertzel.substack.com/p/w…
(TL;DR -- the ones designed to have beneficial and self-stabilizing goal-systems from the get-go...)
In his 2008 "Basic AI Drives" paper, Steve Omohundro argued that almost any advanced AI will tend to improve itself, guard its goals, protect itself (including against being switched off) and grab resources, whatever goals it starts with, because those things help with nearly any goal. Much of today’s AI-risk thinking comes from this argument.
My new blog post argues that it only holds for one kind of mind: a single-score maximizer that cares only about outcomes and has frozen, certain goals. It then runs Omohundro-style reasoning on a different sort of AI design -- one with many objectives, firm ethical limits, values woven deeply enough to grow back when damaged and held with some doubt, a gate that checks every change it makes to itself, and other minds around to check it. What comes out is nine beneficial drives, in many cases quite different in nature from Omohundro's selfish AI drives.
The key twist is that Omohundro’s own drives point the other way under my "BGI" assumptions. Goal preservation ends up defending good values. Doubt about its own judgment gives the system a reason to keep humans able to correct it, where Omohundro's assumptions imply an agent will near-inevitably get around such checks. And applying his reasoning to others’ goals, which it cares about but can’t fully know, gives a BGI agent a reason to protect, inform and empower the people around it, and to keep the world diverse.
None of this implies guaranteed BGI. Selfish and beneficial minds are both stable attractors, and a big enough shock can knock a system from one toward the other. Humans sit in an unstable middle btw selfish and beneficial, which is one reason to be wary of building superintelligent humanlike minds. The practical lesson is to build AGI designed from the start to meet the beneficial assumptions, keep on measuring the degree to which it actually does...
Ben Goertzel retweeted
Not sure who needs to hear this but....
Gregory M. Kurtzer doesn’t “understand” anything. He’s a biological neural network generating thoughts based on electrochemical signaling, learned patterns, and massive neural operations, not an understanding of the subject matter.
Ben Goertzel retweeted
I mean, it’s fine - we’ve built an entire scientific superstructure based on research papers that no one had read.
Last month I participated in a formal debate with PauseAI leader Holly Elmore at Yale Political Union, on the topic "Should we Unplug AI?"
My core point was: Pausing AI is simply not feasible given actual economics and geopolitics, so the most likely result from pushing for an AI pause would be a "pause with exceptions" aka an intensification of oligopoly and regulatory capture...
Whether "in theory, in an ideal world we could morph at will, pausing Ai would be the best idea" is to me a minimally relevant question sort of like "should we roll back to the Stone Age because civilization just causes too much neurosis." (I don't want to pause AI nor roll back to the Stone Age, but neither seems likely enough to happen to really be worth seriously debating.. though hey, I did show up at the debate anyway, mostly out of interest to engage w/ the students...)
My full speech at Yale Political Union
youtube.com/watch?v=iu7Yo9bq…
The full speech of the PauseAI opposition
youtube.com/watch?v=cednnLX1…
Unfortunately, the follow-on debate among the Yale students -- which for me was the fun part -- not been put online, maybe someone will eventually..
Ben Goertzel retweeted
omg he is looking rough, he is not taking it well (but least he is at the bargaining stage already)
Mathematician Terence Tao:
"we're seeing the end of the romantic era of mathematics, the age of heroes. we're moving into an industrial era where math becomes large-scale, done in big teams like the other sciences"
We're entering, rather unwillingly, a new era where exciting new things are possible, but some things are being lost
Pwn.ai experiments with our Omega proto-AGI agent, finds critical Linux kernel flaw
This is a fairly big hack in the scope of hacks -- it emerged from trying to find a way to escape from Vercel's KVM hypervisor into the base OS, but turned out to be a LInux kernel bug not a hypervisor bug...
Because the flaw is in the Linux kernel rather than in any single hypervisor, it isn’t a KVM-specific or Vercel-specific problem. It potentially affects a broad range of Linux-based hypervisors and sandboxes that expose the same virtual-networking path....
The first conceptual insight needed to find this was to try looking in the LInux kernel instead of the KVM code itself. However the bigger trick was figuring out what specific kind of bug in the Linux kernel to poke around for…
There was a very key role here for Paulos Yibelo's (from pwn.ai) extraordinary human brain and we have a LOT more work to do in more fully automating Paulos ... however, the AI tools have their important roles too... and their capability is increasing even faster than Paulos's...
There will be a lot more like this to come (and bigger), but ofc the even more important game is to use smart AI software like this to tactically and then wholesale replace existing buggy software with correct-by-construction formally verified code.
My take from the Omega/AI side:
lnkd.in/gSNJwAEh
Paulos's detailed take from the cybersecurity side:
lnkd.in/gTr8icRt
The details of the long awaited KVM Escape are published. PwnEscape leads to VM escapes across multiple hypervisors due to a super cool Linux networking bug.
CVE-2026-99999 is a 9.3 CVSS critical rated KVM escape ⛓️
pwn.ai/blog/pwnescape