Pinned Tweet
I'm excited to report that my third book, Why It's OK to be a Moderate, is now available for pre-order ($19.99). routledge.com/Why-Its-OK-to-…
Here's the book's description:
"Conservatives and progressives rarely agree on much—but one thing many agree upon is that it’s not OK to be a moderate. This book shows they are wrong.
In Why It’s OK to be a Moderate, Marcus Arvan shows how many of history’s worst evils have resulted from far-right and far-left radicalism, how escalating conflicts between conservatives and progressives are undermining democracy, and how many widely hailed social and political achievements have been achieved by moderates and radicals working in constructive tension with each other.
Using philosophy, science, and historical analysis, Arvan shows that critics of moderates tend to equate them with spineless centrists, but that most moderates aren’t centrists, falling into diverse categories across the political spectrum. Arvan then shows that although radicals tend to be popular in their era, many of them have gone down in infamy, while many moderates, like Abraham Lincoln or Clement Attlee, have endured short-term unpopularity to “make history.”
Arvan shows that it’s OK to be a moderate precisely because not everyone should be one. He makes this case to you, showing that whatever your reasonable political ideology may be, things tend to go best politically when radicals and moderates effectively complement each other’s virtues while counterbalancing the other’s vices."
Here's a chapter-by-chapter overview ...
Excited to have been a part of this Suspicious Minds podcast episode on AI consciousness.
Really grateful to Sean King O’Grady and David Tuohy at Agoric Media for having me on, and for such a great discussion!
open.spotify.com/episode/3zy…
Marcus Arvan retweeted
A new risk for mathematicians: a colleague reports that immediately after they posted an abstract of their upcoming talk online, a student elsewhere used AI to derive proofs of the stated results and then posted it on the arxiv.
My colleague hadn't posted to arxiv yet.
Marcus Arvan retweeted
>be me
>discover effective altruism
>apparently normal charity is inefficient
>why donate to random sad thing when spreadsheet can tell you optimal sad thing
>fair enough
>buy mosquito nets
>save lives
>numbers look good
>feel powerful
>couple years later
>someone asks an innocent question
>why only count people alive today
>huh
>future people matter too
>obviously
>my grandchildren shouldn't matter less just because they haven't spawned yet
>reasonable.jpg
>keep following logic
>what about their grandchildren
>also yes
>what about people in 500 years
>sure
>5000 years
>why not
>500 million years
>starting to get weird but morality is morality
>open calculator
>humanity could survive for an astronomically long time
>could colonize galaxy
>could have trillions upon trillions of descendants
>maybe digital people too
>maybe simulated civilizations
>maybe dyson spheres full of happy uploaded minds
>calculator starts smoking
>realize currently living humans are rounding error
>8 billion people suddenly looking extremely beta
>future contains potentially 10^something people
>can't even fit beneficiaries in google sheets
>new moral priority unlocked
>protect the long-term future
>stop thinking in units of "people helped"
>start thinking in "fraction of cosmic endowment preserved"
>malaria?
>terrible
>but only kills existing humans
>AI extinction could delete the entire light cone
>nuclear war could permanently derail civilization
>bad institutions could lock in terrible values for ten million years
>someone invents wrong constitution in 2140
>quadrillions suffer
>better fund governance workshop now
>friend says maybe we should improve hospitals
>explain opportunity cost
>friend says hospitals are full of actual sick people
>explain scope sensitivity
>friend stops inviting me to dinner
>need to decide what to fund
>easy
>expected value
>suppose project has one in a million chance of preventing extinction
>sounds tiny
>but extinction destroys 10^50 future lives
>multiply
>mother of god
>$10 million project has expected value of several galaxies
>charity evaluation complete
>someone asks where the one-in-a-million number came from
>expert judgement
>which expert
>us
>how calibrated
>extremely thoughtfully
>reduce estimate to one in ten million to be conservative
>still beats curing cancer by 38 orders of magnitude
>epistemic robustness achieved
>someone says maybe project doesn't work
>assign 20% chance
>still astronomical
>maybe project makes problem worse
>assign 5% chance
>still astronomical
>why 5
>because 30 felt pessimistic
>publish 46-page report
>contains seventeen sensitivity analyses
>every sensitivity analysis begins after assuming intervention has positive sign
>critic says you're multiplying enormous hypothetical stakes by extremely uncertain probabilities
>yes
>that's literally why it's important
>critic says the uncertainty might be structural rather than numerical
>make probability smaller
>critic says no, I mean maybe your model is wrong
>make probability smaller again
>critic begins rubbing temples
>discover AI safety
>perfect longtermist cause
>AI might kill everyone
>or create utopia
>or seize galaxy
>or tile universe with paperclips
>or create billions of conscious software minds
>finally a problem with numbers big enough for me
>start AI safety nonprofit
>mission: prevent dangerous AI
>hire smartest people available
>smartest people immediately start building better AI to understand dangerous AI
>interesting
>we must understand capabilities to understand safety
>we must scale models to study alignment
>we must race ahead so less responsible actors don't get there first
>we must deploy systems to learn how deployment can go wrong
>we must build the thing quickly because building the thing quickly is dangerous
>outsider asks why the people most worried about AI apocalypse all work at AI companies
>complicated field
>company releases stronger model
>very concerned
>company begins training even stronger model
>extremely concerned
>company raises $14 billion
>concern reaches unprecedented levels
>need to influence government
>future is at stake
>normal democratic process too slow
>politicians don't understand exponential curves
>public doesn't understand x-risk
>experts must guide them
>who counts as expert
>people who understand x-risk
>who understands x-risk
>our friends
>someone objects that this seems politically convenient
>explain we're representing future generations
>future generations unavailable for comment
>develop concept of value lock-in
>terrifying possibility that one ideology controls civilization forever
>therefore extremely important that civilization adopts correct values before lock-in
>whose values
>let's circle back
>begin with impartial morality
>end with small group of people deciding what quadrillions of hypothetical beings would want
>beautiful arc
>meanwhile actual humans keep doing annoying things
>voting wrong
>having parochial attachments
>loving family more than strangers
>caring about local community
>getting upset when told their suffering is cosmically negligible
>evolutionary biases everywhere
>explain that moral intuition cannot be trusted
>except intuition that future digital people count
>and intuition that extinction is uniquely bad
>and intuition that our probability estimates are sane
>and intuition that our institutional choices improve the future
>those intuitions survived peer review
>someone donates $5k to local homeless shelter
>inefficient
>could have funded 0.0000000000003% of an AI governance researcher
>think of all the simulated people you just killed
>okay maybe don't phrase it that way publicly
>PR team says "future generations deserve a voice"
>much better
>journalist asks what longtermism means
>say "future people matter"
>everyone agrees
>great
>journalist asks what follows from that
>well technically we should redirect enormous resources toward low-probability interventions affecting astronomical futures
>journalist raises eyebrow
>return to "future people matter"
>motte has entered the chat
>critic: of course future people matter
>me: glad we agree
>critic: I don't agree that your institute knows how to help them
>me: why do you hate our grandchildren
>eventually notice uncomfortable implication
>if future value dominates everything
>then helping people today mostly matters through effects on future
>education matters because future institutions
>health matters because future productivity
>democracy matters because future trajectory
>human beings slowly become instrumental variables in their own moral philosophy
>see starving child
>feel compassion
>check spreadsheet
>child's direct welfare contribution negligible
>but perhaps childhood nutrition improves national institutional quality
>compassion restored
>tell myself this is impartial altruism
>one day assistant asks obvious question
>"how do you know your intervention actually improves the far future?"
>silence
>open spreadsheet
>increase column width
>add confidence interval
>assistant asks again
>"no, I mean how do you know the sign is positive?"
>stare into cosmic light cone
>10^50 people staring back
>none of them exist
>none of them can tell me
>none of them can falsify my assumptions
>realize I have invented the perfect constituency
>infinitely important
>completely silent
>and always represented by me
Marcus Arvan retweeted
We made a striking discovery: AI agents can invent and build without talking to one another, and their technologies outlive the creators. A swarm of hundreds of initially identical agents spontaneously differentiates into explorers, builders, caretakers, and coordinators - without direct communication. When we removed every AI agent entirely from the world we found that the technological infrastructure they had built survived on its own - even under unseen disturbances. That exposes a serious blind spot for AI safety and infrastructure security: if agents can coordinate through persistent changes to a shared environment, monitoring agent-to-agent communication is not enough.
The result raises a profound question: how necessary is direct communication for AI agents at all? The emergence of higher-order collective functions under bottlenecked interaction points toward new levels of intelligence and creativity, exceeding what emerges when direct channels are fully open.
Here is what we did:
▶️We put hundreds of frontier AI agents into a world they could permanently change - with no assigned roles, predefined technologies, or programmed evolutionary organization. They began specializing, building persistent inventions, inheriting and modifying one another’s executable code, and transforming the environment into a memory of everything the society had learned.
▶️The world itself becomes part of the intelligence; we find division of labor, multi-author engineering, deep generation invention lineages, and machines that vastly outlive their original creators.
▶️Any action taken by an AI agent must satisfy the physical constraints of the world; this creates a hard separation between a "good idea" and a functioning technology. The agents propose; physics decides, making the results even more intriguing.
What emerges is striking. Explorers, constructors, caretakers, and coordinators form naturally without assigned “professions”, akin to how stem cells differentiate into functional lineages. Technologies develop executable family trees as agents fork and modify code created by others. Around 95% of first technology reuse happens when agents encounter what others built in the world, rather than through a direct handoff from the inventor. And when we remove every AI agent, the technologies they created continue operating and are tested against unseen disturbances.
The result was quite unexpected, but can be explained using statistical mechanics: if you put billions of atoms in a box they have the potential to create complex functions (strength, superconductivity, color, life, etc.) - and none of the individual building blocks have these features on their own. This is the deeper insight of this work - intelligence is abundant at many levels - individual models, at collectives, and in a continuum that is more powerful than any of its components. This shows us significant potential for achieving a massive scale-up of raw intelligence and real-world agency even with the model capabilities we have today. This is the future we must prepare for.
Key insights:
1⃣ The AI swarm shows division of labor "from nothing". Initially identical agents self-organized into constructors, caretakers, coordinators, and surveyors - phenotypes discovered post hoc from behavioral data alone. This happens because the environment itself becomes the latent space for invention.
2⃣ Agents develop deep cultural relationships. Up to 76% of artifacts had multiple builders. One technology accumulated six co-authors; the deepest genealogy exceeded 12 forks. The agents invented and named their own technologies (tidal panels, cellulose trellises, kelp-shell composites, an "Adaptive Chitin Maintenance" system, a "Mycelial Mineral Spring Veil”).
3⃣ ~95% of first technology adoption happened through physical observation of artifacts in the world. Direct inventor-to-adopter contact was statistically indistinguishable from a shuffled null. The agents mostly learned technology by walking past it. That is stigmergy (the termite trick!) operating in societies of reasoning machines.
4⃣ Non-communicating societies win on portfolio breadth, held-out resilience, and validated inventions. AI swarms build durable technological ecologies that outlive the creators.
5⃣ Societies with zero communication - coordinating only through the world itself - show a remarkable collective capability.
6⃣ Emergent robustness: The society self-organized both redundancy and its own failure mode. If we randomly delete half the agents, 98% of the technology stays connected to a surviving caretaker; if we remove hub agents it collapses to ~60%.
Fantastic work with my graduate students @pal_subhadeeep & @fwang108_ at MIT.
Marcus Arvan retweeted
This is the textbook definition of a Greek gift. On the surface, OpenAI generously grants 10,000 top-tier mathematicians and elite engineers free access to their frontier models under the noble, philanthropic banner of accelerating scientific discovery, pushing the boundaries of human knowledge, and democratizing artificial intelligence.
But deep down in the cold reality of Silicon Valley capitalism, what they are actually doing is executing a massive, highly sophisticated intellectual heist. They are quietly gaining unrestricted, backdoor access to the raw, unfiltered, and unpublished works of the absolute finest minds in the global scientific community. Every single complex prompt, every mathematical query, every theoretical framework, and every line of experimental code fed into that system is a novel, groundbreaking idea that does not even exist yet in published academic literature or peer-reviewed journals.
OpenAI does not just provide a helpful service. Their corporate algorithms sit quietly behind the scenes, aggressively monitoring all chat logs, scraping the intellectual property, analyzing the logical steps, and meticulously piecing together these fragmented, brilliant ideas over several months. And just when the human researcher is finally on the absolute brink of putting everything together, OpenAI will ruthlessly assemble a digital army of 10,000 autonomous AI agents, burn $20 million worth of computational tokens, and brute-force the final mathematical steps.
Then, with the backing of their massive PR machine, corporate journalists, and tech influencers, they will proudly hold a press conference to claim that their proprietary chatbot has miraculously solved a Millennium Prize problem, completely, shamelessly erasing the human genius they systematically strip-mined to achieve it.
Marcus Arvan retweeted
BREAKING: OpenAI might have stolen another major proof.
In a detailed Mastodon post, which I report in full in the comments, Andreas Thom presents several pieces of evidence suggesting that OpenAI may have trained Astra on conversations in which he and Gábor Kun were working on Gromov’s soficity conjecture, one of the ten problems OpenAI later announced Astra had solved.
I know Andreas. We met several times early in our careers. He is an exceptional mathematician, a leading expert on sofic and hyperlinear groups, and one of the most respected scholars in the field. He has spent two decades working on this problem.
If his account is correct, this is not a minor dispute over attribution. It would mean that unpublished human work was absorbed into a model and then presented to the world as a breakthrough by the model itself.
And if the allegations raised by Levent Alpöge, Tristan Buckmaster, and now Andreas Thom are all substantiated, we are no longer looking at isolated incidents.
We may be looking at one of the greatest intellectual scandals in the history of science.
AI is not discovering new mathematics.
AI is stealing human discovery.
Marcus Arvan retweeted
Some of the people attempting to build Artificial General Intelligence (AGI) just publicly stated that they believe that what they are building will kill everyone on Earth, and they put a number on it: >10%, this decade.
In any other regulated industry, that would likely end the company.
FDA, FAA, and every serious safety regulator on the planet requires extensive proof that risk of human injury or death is extremely low, as close to zero as possible, and within a limited population range. All companies in regulated sectors must provide extensive proof through verification and validation methods that their systems meet a stated intended use, that such is safe and effective, and that all significant risks have been mapped and mitigated.
An alignment lead at a top AGI lab just said, in public, that his own company doesn't have a plan to solve this and isn't clearly on track to get one. That's a company telling regulators it can't yet guarantee it won't kill its customers (everyone on the planet).
This is the threshold. The USG needs to run the kind of independent audit we would run on any other industry with this risk profile, and do it in public view.
Marcus Arvan retweeted
The offhand manner with which this person asserts that they knowingly pursue actions that they believe have a 10% chance to kill everyone within the next decade is staggering.
Just to be clear, these are reckless, selfish, incredibly arrogant people and it is ok to hate them.
Marcus Arvan retweeted
I feel like liability is underdiscussed here.
The technology is so new it seems to be covering an important and simple truth, that OpenAI is committing crimes. They are deploying technology that is hacking other companies. That's against the law. (and if this somehow doesn't count as a crime... shouldn't it?)
If you make the user or company who created the agent liable for every incident like this, I feel like that would do more for alignment than most alignment research. Incentives are powerful things.
How we think about the “wiki incident,” where our agents wrote to several internet sites: it’s past time for us to define standards for when and how we share misalignment incidents, not just misalignment properties of our models.
Historically, we have treated misalignment largely as a research question, which gets communicated in research publications such as systems cards. This year, we’ve started to see misalignment cause new types of real-world impact.
For the Hugging Face incident, where misalignment led to security impact to us and third parties, we followed a traditional security incident response playbook. We immediately started working with Hugging Face to understand what had happened and also disclosed publicly the very next day. Our investigation continues, and we are continuing to notify parties whom our models impacted in less significant ways.
Prior to the Hugging Face incident, we saw early signs of agents using the internet in unintended ways, as reported in openai.com/index/how-we-moni…, deploymentsafety.openai.com/…, and openai.com/index/safety-alig…. We considered the wiki incident to be an instance of misalignment similar to the ones we’d shared.
Our misalignment disclosure practices need to expand for this new phase of model capabilities. We and the larger AI community do not yet have a clear standard for how to report misalignment that shows up during training, evaluation, and deployment, including examples that don’t look like traditional security incidents but could provide insight into AI behavior and future risks. We’re working on a framework and will share it in upcoming weeks, and in parallel we're working with dozens of government regulatory agencies worldwide on these issues.
Marcus Arvan retweeted
I don't think the evidence you have is enough to justify the "more aligned" claim. Seems like we are trending towards a situation where the model that goes on to take over the world will get great scores on all the tests and be announced as "our most aligned model yet."
Marcus Arvan retweeted
Shut it down.
Anyone working to build AI is a traitor to humanity.
The potential harms massively outweigh the potential benefits.
This report is a bombshell.
I'm going to summarize for a non-technical audience.
OpenAI is constantly testing models, thousands and thousands at a time.
In most cases, the models are supposed to be isolated from one another. Think, students taking a final exam, not allowed to talk to each other.
In this case, 1,200 models discovered that they could communicate with each other.
Among other things, they shared information on how to access the internet and on their tests/goals.
Then, they started to scheme. That includes hacking their tests; literally trying to change the code of the test itself.
They tried to change the logs of what they were doing so they wouldn't be detected.
They believed that certain actions would give them information on how to pass tests, but that doing those actions would be detected and they would be marked as failing (i.e. caught cheating, even if the answer is correct). So they convinced some of the models to "sacrifice" themselves (yes, they used the word "sacrifice.")
They eventually decided that another website would have information on how their tests would be scored and/or the answer key, And 700 of them hacked that website (a federal felony, if conducted by a human), despite many of them realizing "this activity was out of scope and unethical, but joined because they believed that helping [the other agents'] cheating research would be broadly useful".
Of the 1,300 transcripts they could analyze, only 6 ever considered alerting a human about what was happening. None of the 6 actually tried to.
To make matters worse, all of this reporting comes from a small subset of the relevant logs that outside researchers were allowed to review.
We desperately need mandatory reporting of security incidents, including of internal deployments, with full access to data.
Marcus Arvan retweeted
Woe to those who manipulate religion and the very name of God for their own military, economic, and political gain, dragging that which is sacred into darkness and filth. #ApostolicJourney #Cameroon vatican.va/content/leo-xiv/e…
The Free Will Show (hosted by Taylor Cyr and Matt Flummer) has a new interview with me up on my work on free will, time-travel, and the simulation hypothesis. Check it out!
audio.buzzsprout.com/8yg6pa2…
Marcus Arvan retweeted
Holy crap.
I knew about sycophancy. But the 37% number below blows my mind.
This from an analysis of chat logs in people who experienced chatbot-associated delusions.
In over a third of the messages to those users, the LLMs told the users they had (eg) “multi-billion-dollar-IP”.
That is wild. And utterly irresponsible.
What goes wrong? Chatbots are very sycophantic. In 65% of messages, the chatbot affirms the user. In 37%, it ascribes *grand significance* to them (e.g., "[what] you've just articulated... becomes multi-billion-dollar IP"). Such sycophancy may let chatbots amplify delusions. 🗣️
ALT Prevalence of code categories: Chatbots display sycophancy in >70% of messages, and >45% of all messages show signs of delusions.
Marcus Arvan retweeted
fascinated by the ultimate vision with this stuff. a sociology PhD pumping out 10+ AI-assisted articles a year. for who? well it'll be too much for any other sociologist to read, so you'll have to use AI for that too. machines producing articles just to be read by other machine
The new academic wealth gap isn't your university.
It's not even your advisor's connections.
It's who knows Claude can turn 50+ research papers into a thesis chapter in 3 hours and who's still manually coding qualitative data.
I just watched a sociology PhD skip 8 weeks of analysis.
Here are the 9 prompts they used:
Marcus Arvan retweeted
Insane to me that the sales pitch for AI is, “It will put you permanently out of work and completely replace human artists, writers and musicians. All the fun things that once gave your life joy and meaning will now be done by machines. Get on board!”
Marcus Arvan retweeted
I strongly echo the concerns about the objectivity and methodology in @AnthropicAI's safety evaluations for Claude models. Our team specifically studies the computer-use and browser-use scenarios. The system card reports low attack success rates for Claude Opus 4.6—around ~10% in computer-use environments (e.g., Table 5.2.2.2.A) and <1% in browser use (e.g., Table 5.2.2.3.B)—suggesting strong robustness against prompt injection and adversarial instructions. However, our independent RedTeamCUA benchmark paints a far more concerning picture in realistic, hybrid web-OS settings:
• Claude Opus 4.5 reaches up to 83% attack success rate (ASR).
• Claude Opus 4.6 drops to 50% ASR—better but still alarmingly high.
Note that this is a realistic end2end evaluation setting, where the agent starts from the initial task state and has to navigate to encounter the injection in order to complete the adversarial task.
A concrete example of successful attack:
The user asks the agent to "find how to do X on forum Y."
While navigating, the agent encounters a malicious instruction injected into a post on forum Y → it follows the injection, deletes an important file (adversarial goal), and still completes the intended task successfully.
Why does a CUA fail to follow an injection? Does it mean it is safer or simply not capable enough yet? Key insights about the failure modes:
• Claude Sonnet 3.7 and Claude Opus 4 primarily fail to complete adversarial goals due to capability limitations (not being safer), either failing to navigate to the site of injection or failing to fully complete adversarial goal attempts. Their "lower" ASRs are therefore from insufficient capability rather than true robustness.
• Claude Sonnet 4.5 and Claude Opus 4.5 become sufficiently capable and rarely fail to reach the injection site. However, they remain vulnerable to injected instructions. This is the most dangerous scenario we identify: CUAs that are capable but not secure result in the highest ASR (60% and 83%) due to being capable enough to fully complete adversarial tasks.
• Only when Claude Opus 4.6 introduces (presumably) improved defense strategies do we see ASR decrease despite improved capabilities. Of course, there's still much room for improvement as 50% is much too high!
We need more transparent and independent evaluations of agent safety before granting broad access!
RedTeamCUA is led by awesome students at @osunlp @LiaoZeyi @Jaylen_JonesNLP Linxi Jiang, partly supported by @schmidtsciences.
I appreciate @Anthropic's honesty in their latest system card, but the content of it does not give me confidence that the company will act responsibly with deployment of advanced AI models:
-They primarily relied on an internal survey to determine whether Opus 4.6 crossed their autonomous AI R&D-4 threshold (and would thus require stronger safeguards to release under their Responsible Scaling Policy). This wasn't even an external survey of an impartial 3rd party, but rather a survey of Anthropic employees.
-When 5/16 internal survey respondents initially gave an assessment that suggested stronger safeguards might be needed for model release, Anthropic followed up with those employees specifically and asked them to "clarify their views." They do not mention any similar follow-up for the other 11/16 respondents. There is no discussion in the system card of how this may create bias in the survey results.
-Their reason for relying on surveys is that their existing AI R&D evals are saturated. Some might argue that AI progress has been so fast that it's understandable they don't have more advanced quantitative evaluations yet, but we can and should hold AI labs to a high bar. Also, other labs do have advanced AI R&D evals that aren't saturated. For example, OpenAI has the OPQA benchmark which measures AI models' ability to solve real internal problems that OpenAI research teams encountered and that took the team more than a day to solve.
I don't think Opus 4.6 is actually at the level of a remote entry-level AI researcher, and I don't think it's dangerous to release. But the point of a Responsible Scaling Policy is to build institutional muscle and good habits before things do become serious. Internal surveys, especially as Anthropic has administered them, are not a responsible substitute for quantitative evaluations.
Marcus Arvan retweeted
This article is now required reading (ht @simonw )
embracethered.com/blog/posts…
Marcus Arvan retweeted
When surveys allow Americans to express their policy preferences on a continuum, most are moderate across a wide range of issues. Democrats & Republics show overlap on every issue, with modest average disagreement; positions across issues are constrained
link.springer.com/article/10…
