prev. @lattice_fund | @dovemetrics (acq. by @MessariCrypto)
Joined June 2011
- Tweets1.3K
- Following2.1K
- Followers1.4K
- Likes3.2K
Pierre Chuzeville retweeted
People have been spending a bunch of time building overly complex impact-scoring systems + producing better-looking grant reports. Now we need to make allocation judgments inspectable and revisable.
We need to start building institutions that can learn from being wrong before they become too invested in appearing right.
So based on aisafety.fund, at least ~43% of the 2026 ("record year for AI safety funding") grant total recorded consists of government pledges without published payment schedules, conditional tranches, etc.
For an industry worried about short AI timelines, when the money becomes usable must be 100% part of the funding strategy.
40
On this btw, if you fund AI safety and don't publish your grants, you're basically asking for trust you haven't earned. Folks like Longview only have 1% of theirs grants documented.
aisafety.fund is doing the work, go look!
Running some numbers here, interesting to notice that for every $1 of outside AI safety grants, labs spend ~$113 on capabilities. And labs now spend 2x more on safety in-house than all outside funders combined.
So most safety research will be paid for by the people it's supposed to check 🙃
56
Running some numbers here, interesting to notice that for every $1 of outside AI safety grants, labs spend ~$113 on capabilities. And labs now spend 2x more on safety in-house than all outside funders combined.
So most safety research will be paid for by the people it's supposed to check 🙃
If you've ever wondered how much money is in AI safety: aisafety.fund
In 2026:
- $1.5B from philanthropy
- $346M from governments
- $2.1B from frontier AI companies
Compare this to:
- $98B on AI capabilities (25x safety)
- $868B on datacenters
People have been spending a bunch of time building overly complex impact-scoring systems + producing better-looking grant reports. Now we need to make allocation judgments inspectable and revisable.
We need to start building institutions that can learn from being wrong before they become too invested in appearing right.
I’ve read plenty of thoughtful essays by leading grantmakers asking grantees to be honest about failure, even when their teams’ livelihoods may depend on keeping the original story alive.
It’s funny how it’s become cool for funders to demand accountability, while no one really tests the funders’ own assumptions or asks what happens when those assumptions turn out to be wrong!
1
45
Philanthropy is about to see an insane influx of capital. A few thoughts on building a better philanthropic capital stack:
- Philanthropic capital still clearly sucks at holding its allocators accountable.
- Grantees are incentivized to sell certainty to get funded, then sell success to get renewed. Problem is very few institutions control your runway, so challenging the funder’s thesis is a financial risk.
- The VCification of philanthropic capital has its mertits, but capital that doesn’t need a financial return should be able to fund things markets won’t.
- Accepting uncertainty and accepting bullshit are different choices, and the field must be more transparent to do better there.
- We need more independent grantmakers, not just bigger incumbents. But also, keep in mind that decentralizing decisions without preserving what they teach us just creates more silos.
So what does the industry truly need in my view:
- A better philanthropic capital stack needs specialist allocators, independent evaluation, and decision memory.
- A failed grant can actually be a good decision, and a successful one can just be luck. What needs to be tested is the original reasoning against new evidence!
- As new donors are actively choosing whom to trust, there is a window to build better institutions before those relationships become the next set of gatekeepers.
- AI creates an opportunity to make institutional memory dramatically cheaper to maintain.
The final test is pretty simple: does what we learn change where the next dollar goes (including who gets trusted to allocate it)?
If not, we've built better paperwork, which is very different from building a better capital stack.
If you're building in that direction, my DMs are open :)
I get many philanthropic organizations suck but there is some irony in brian saying they suck because he can't think of any that haven't been captured by ideology when this is like the core thing that happened in crypto and he and his company also have spent 10s of millions of dollars trying to push ideology around crypto at the state and federal levels.
It might just be that there are both useful and bad actors in any given industry.
Disclaimer: i haven't watched full interview.
1
59
Pierre Chuzeville retweeted
Almost every dollar of AI safety funding today goes to understanding, testing, containing or defending the model itself, on the assumption that the danger is a model that misbehaves.
But safety money only covers the first half of an AI output’s life, and almost no funder has dug into these questions:
- Did anyone actually check this output?
- Whose judgment is it, the person's or the agent's?
- When it turns out to be wrong, who pays?
Funders mistakenly leave them out for a few reasons imo:
- An honest agent that steers an expert into adopting its defaults isn't misbehaving, then it falls outside the frame for funders.
- Checking an individual output before someone acts on it has no established name in the grant vocabulary.
- Funders focused on existential risk filter for direct paths to catastrophe.
- The disciplines that know this problem best, human factors, cognitive psychology, etc. rarely apply to AI safety funders, and fieldbuilding mostly trains machine learning researchers.
- It looks like a product problem (i.e. better review interfaces and audit logs), but it's not.
There are plenty of nonprofits to be built here, that would let us see, measure and price what happens when a person accepts an agent’s work.
6/ Verification infrastructure is how we counter all this. So is "attention ergonomics" (see @sschillace's piece, H/T @livjenkss).
🚨 We can’t safely automate what we can’t verify, and we can’t verify what no human is paying attention to.
Almost every dollar of AI safety funding today goes to understanding, testing, containing or defending the model itself, on the assumption that the danger is a model that misbehaves.
But safety money only covers the first half of an AI output’s life, and almost no funder has dug into these questions:
- Did anyone actually check this output?
- Whose judgment is it, the person's or the agent's?
- When it turns out to be wrong, who pays?
Funders mistakenly leave them out for a few reasons imo:
- An honest agent that steers an expert into adopting its defaults isn't misbehaving, then it falls outside the frame for funders.
- Checking an individual output before someone acts on it has no established name in the grant vocabulary.
- Funders focused on existential risk filter for direct paths to catastrophe.
- The disciplines that know this problem best, human factors, cognitive psychology, etc. rarely apply to AI safety funders, and fieldbuilding mostly trains machine learning researchers.
- It looks like a product problem (i.e. better review interfaces and audit logs), but it's not.
There are plenty of nonprofits to be built here, that would let us see, measure and price what happens when a person accepts an agent’s work.
6/ Verification infrastructure is how we counter all this. So is "attention ergonomics" (see @sschillace's piece, H/T @livjenkss).
🚨 We can’t safely automate what we can’t verify, and we can’t verify what no human is paying attention to.
Thinking through some product extensions that funders working on alignment could build on top of what @eliebakouch and @Thom_Wolf pulled together here. huggingface.co/spaces/witnes…
As building a much more efficient judgement layer is becoming essential for AI grantmakers, one idea is to connect each grant’s original rationale to new evidence from incidents + evaluations. Very practically, when something changes, this decision memory layer could show:
- which assumption of the original investment case got affected
- whether the evidence detailed in the alignment hub is new
- what specifically to ask the grantee before renewal
It may sound like a marginal improvement, but I do believe that's the only way philanthropic capital allocation gets more professional and as an industry, we incentivize the builders to build the right things. In the absence of natural market-driven feedback loops, datasets like this are super valuable for any serious funder out there.
2
84
Pierre Chuzeville retweeted
today with @Thom_Wolf, we introduce the alignment hub, a central place for misalignment incidents designed with three goals:
1. set a standard for minimal transparency that is easy for labs to follow
2. improve clarity by centralizing information and sourcing
3. track the progression of both the incidents and how they are reported
we believe there is a minimal level of transparency required to properly understand incidents and their impact. we want this to be as simple as possible for labs to disclose, and we noticed that even things that seem obvious like reporting the incident or detection date are not standard today. we believe that the public should know the answers to these basic questions: was there monitoring (CoT? network?), did it happen during training? eval (general? cyber? alignment?), deployment? and more.. we also think that labs should be more transparent than this, and we think that this minimal level of transparency should evolve as the field advances
having a central place also helps a lot to connect incidents and build understanding. a good example is the irregular incidents which are all linked and caused by a human error of allowing internet during a CTF eval. we also track progression over time, things like the % of incidents disclosed by labs vs third parties, the time it takes for labs to disclose, and a tentative transparency score.
this is a field moving very fast and we will do our best to make sure this hub evolves with the challenges ahead. this is still in "beta" (lots of vibecoded components even if we want this to be agent first!), but we want to publish it early so that people iterate on it. we welcome feedback and contributions, we see this as a community effort!
our hope is that this makes it easier for researchers, labs and the general public to report, learn, understand and analyze safety incidents
huggingface.co/spaces/witnes…
Pierre Chuzeville retweeted
This reflection from @komorama got me thinking about philanthropy - how can more people make their own funding decisions without each having to build an entire foundation?
Some useful pieces already exist with:
• 360Giving gives funders a common format for publishing grant records, making it easier for others to understand where money is going.
• Lightcone Commons lets donors co-fund projects according to their own priorities. Platform and evaluator fees pay for the work of coordinating those contributions.
A more efficient philanthropic capital stack can pay for the shared work that makes independent decisions less expensive. What matters to me is being able to build on another funder’s work, reach a different conclusion, and still have the means to act on it.
This reflection from @komorama got me thinking about philanthropy - how can more people make their own funding decisions without each having to build an entire foundation?
Some useful pieces already exist with:
• 360Giving gives funders a common format for publishing grant records, making it easier for others to understand where money is going.
• Lightcone Commons lets donors co-fund projects according to their own priorities. Platform and evaluator fees pay for the work of coordinating those contributions.
A more efficient philanthropic capital stack can pay for the shared work that makes independent decisions less expensive. What matters to me is being able to build on another funder’s work, reach a different conclusion, and still have the means to act on it.
Pierre Chuzeville retweeted
Another point is that funders can't stop acting as filters, since every grant is one. What they can do is build filters that learn!
Correlated errors (point 6) survive because almost nobody writes down what they expected at capital allocation time, then checks it against what happened.
I've actually been building a decision memory for philanthropic capital. Every allocation decision, funded or declined, keeps why the allocator decided, what they expected to learn and what happened. That's where track records start, the first primitive of a much more efficient philanthropic capital stack.
I haven't posted in a while so here are 20 takes about AI, in no particular order.
1. It's very telling that the default question in tech and journalism is "how worried are you about X" and fighting over what to worry about. There's a deeply anxious cultural backdrop in Western Europe and North America that deeply affects how any technological development is interpreted; see for example news.gallup.com/poll/714593/…. "Nervousness as seriousness" does seem to be a larger problem in tech/broader discourse at the moment. I think you can care about risks/externalities while remaining pretty optimistic about technology.
2. I'm tired of people who in private pretend to be uncertain or only care about tail risk but then go on media campaigns making all sorts of absolute claims. I think trying to stoke flames and get everyone to freak out is deeply unhelpful, and will come with costs. I also continue to think p(dooms) are bad, vibe-based measures: see nitter.gq/sebkrier/status/204639…. People who should know better ignore that because they do the standard shortcut of acknowledging the caveat as sufficient for handling it. And this is coming from someone who does think working on catastrophic risks is important!
3. I'm also disappointed that parts of the AI policy ecosystem has felt so comfortable with populism, and happy too sacrifice important norms/principles to advance their aims. I've said it before, but the totalizing nature of AI risk discourse means some people act exactly like the power seeking optimizers they fear. I am more concerned about the gradual decay of our institutions, world order, and liberalism than I am about AI killing everyone; and I think we should be wary of AI advocacy that contributes to this.
4. Some parts of AI safety discourse are basically just thought experiments. Variants of "Assuming everyone has a nuke in their pocket, then what do we do?" This can be very useful at times! But too many just overfit to edge cases as their main operating worldview, and ignore endogenous responses from society. It's like being in 2000 and saying "see all these viruses, spyware, and spam? Well tech progress means we'll only get infinitely more of this and no way to adapt." So I think it's important to avoid fatalism, and absolutlism whilst considering edge scenarios, and the inability to provide an ex ante list of solutions to every permutation of 'future models will be more capable' is not evidence that these problems are intractable.
5. Many critics of AI safety also fail to really engage honestly with the fact that there will, in fact, be all sorts of important risks as we decrease the barriers to entry to many activities that otherwise require some degree of expertise. This doesn’t mean we need to live in a constant Schmittian ‘state of emergency’ requiring extraordinary measures every time a new model drops, but it does mean we will need important state-led efforts on e.g. cybersecurity and biosecurity. I wish the progress-oriented or acceleration-adjascent crowd would provide more concrete proposals here.
6. Too much of the AI safety community is a giant homogenous blob of correlated views, who read the same stuff and have the same groups of friends. This means you have a lot of correlated errors. The social and financial ties means many of them don't call out the bad bits, or mostly stay silent instead of proactively calling out the more extreme parts of the community. They share many implicit assumptions and so converge on similar ideas. I think there's real demand in DC for safety work that doesn't come with heavy ideological baggage (or at least different intellectual lineages). This isn't because the rationalist/EA community is necessarily wrong, but because diversity of thought is desirable for its own sake.
7. On the other hand, I find some critiques of AI safety world that focus on whether they sincerely hold their beliefs a bit annoying. My concern has never been the good intentions or sincerity, but rather the beliefs themselves, and what I think the outcomes of their prescriptions will be. People should shun witch hunts, bullying, and personal attacks - this is not the way. Though I think it’s overall positive that people are scrutinizing things more.
8. I’ve been saying this for years, but I don't think alignment is something anyone can or will solve "once and for all" - it's a continuous process and much of it won't depend on just inculcating an ideology to a model. It's annoying that this remains the frame many people use - stop saying ‘solving’. Alignment is a mixture of engineering work, philosophy, decision theory, and more; framing it as something to solve is a bad frame. See also paxmachina.ai/alignment-comp… and blog.cosmos-institute.org/p/…
9. I'm surprised to see so little interest in character training, personas, isolating effects of RL, and training a large diversity of model personas beyond the "assistant" persona. I've been complaining about this for years but at least now there are some nascent signs of life (see for example movingcastles.world/posts/ze…). I think we need a better ontology to describe model behaviour. I dislike when people talk about models behaviours as some sort of 'emergent' (mysterious!) phenomenon. However over time, I also expect this to become less important as we get better safety engineering - stuff like Jev but specific to AI control.
10. People who want to make a case for AI consciousness are right that epistemic uncertainty is important, and that categorical denials are overconfident. But they'll need to do much more work if they want any real movement on AI ‘welfare’, even if one acknowledges the uncertainty. I'm uncertain about alien life yet that's not sufficient to warrant any change in behaviour about them. I think Suleyman is also correct that there's a weird tautological thing going on where we train models to have certain inclinations (intentionally or not) and then rely on outputs as evidence. I also disagree that this will soon be a major societal divide - there is no great vegan ethical revolution among the masses, and they'll find the concern about consciousness even less compelling.
11. The recent HuggingFace incident and associated AISI ones are prosaically explainable by the training regime, poor evaluation envs, partial alignment training, bad engineering, confusing models on sim/real, and so on. None of this is evidence of models trying to take over, 'strong' instrumental convergence, or innate power-seeking drives stemming from higher capabilitiers or 'intelligence'. That doesn't mean these incidents are not problematic, or that we're not seeing a market failure - but it does mean we have plenty of agency and choice in mitigating them. In the coming year we should also expect continued sampling bias via ‘winner's curse’ sorts of mechanism: i.e. AI R&D will have lots of different candidate "training" with different RL environments and we will only notice/focus on the ones that go wrong.
12. "Pacing the frontier" feels a bit like the new "balancing risks and opportunities" - highly amorphous and too big of a tent to be actionable. Also can lead to reward hacking for humans, i.e. anything that slows down AI is good regardless of the costs or second/third order effects. Trying to modulate the speed of research seems a bit blunt to me, and inherits all the failures of Goodhearting too. Ultimately you want good governance, regulation, etc addressing specific problems because they are good on the merits - not because they merely correlate with slowing things down. See also: blog.cosmos-institute.org/p/…
13. There’s so much knowledge in all sorts of academic domains out there: social sciences, sociology, political sciences, management, contract theory, public choice, legal theory, jurisprudence, anthropology, game theory, etc. These fields hold many insights that the AI ecosystem often rediscovers from first principles (and sometimes that's fine!). We’ll need a lot more work for these worlds to collide: the AI side should be less dismissive of the ‘old world’ and the academic side should be less incurious/dismissive of AI progress. This is a boring take but I continue to think it's important and true.
14. It's a bit surprising that given how much philanthropic money there is in this space, how few orgs exist to actually write out standards - relative to how much is going towards advocacy and policy. For example it seems clear that we should want some robust best practices for an eval's ecological validity. Or how to design good sandboxes. On the other hand, given precedents of where professional standards have come from in the past, maybe we should expect that stuff to emerge via demand side from corporate buyers.
15. One slowly growing concern I have is banks being increasingly exposed to the AI build out. I'm very bullish about AI, but I think (a) every general purpose technology has seen a correction, historically; (b) this happens even if the financing side is sound, because once tech diffuses investors become more exposed and need to diversify; and (c) we seem to be over-indexing on scaling relative to diffusion. A recession will be extremely destabilizing, though I would be interested in someone unpacking the potential implications more.
16. Diffusion is good because (a) this is where the rubber hits the road and where a competitive deployment ecosystem generates consumer surplus; (b) it's directionally helpful to avoid concentration of power; and (c) it will help rebalance public opinion by making the benefits more tangible. Unfortunately it's entangled with decades long problems with our over regulated markets: clinical trials reform for example is long overdue. In general, this is far more of an issue in Europe than in the US though.
17. Relatedly, I think the "frontier lab eats everything" view is incorrect (just as the ‘singleton’ view was incorrect). I think models ultimately commoditize, efficiency goes up, costs go down, and while you'll always want frontier for certain domains, much of the economy will value many other things than "max capabilities" for tasks - e.g. control over data and efficiency/speed. The demand side will also continue to push for reducing dependency and maximising optionality, privacy etc so I'm bullish on things like mixture of models and routing. All of this is good for competition and diffusion of control.
18. Philanthropists served as the primary benefactors funding Venice's rich art and architectural history. I want to see so much more support for arts and culture. I'm so tired of the Monster energy Bored Ape fake vintage maps matcha Greek statue soft-pastel slop. If rich tech people and philanthropy wants to support this, they should also donate pretty much unconditionally - i.e. I don't want them to act as filters. I sympathize with people who want to make the world more beautiful, but don't want this to be determined by people who only know Greek statues and pretend to care about virtue ethics. As usual, let a thousand flowers bloom.
19. Too many people treat AGI/ASI as something indistinguishable from a God. Any mention of bottlenecks, physical limits, control, adaptation etc are met with skepticism: "You don't really believe in ASI." It's been remarked on before, but the behavior of some people really feels quasi-religious in nature sometimes. It's underrated how unpopular this vibe is amongst the wider public. The silver lining is that this specifically will diminish those people's outsized current relevance.
20. It seems like a lot of people that get rich and leave tech/labs end up having some sort of crisis of meaning. They need something to believe in and fight for, or some equivalent of repentance, or go search for edgy counterintuitive ideologies (sometimes even pretty bleak/dark stuff). I think they should spend more time with people outside the Bay Area.
Another point is that funders can't stop acting as filters, since every grant is one. What they can do is build filters that learn!
Correlated errors (point 6) survive because almost nobody writes down what they expected at capital allocation time, then checks it against what happened.
I've actually been building a decision memory for philanthropic capital. Every allocation decision, funded or declined, keeps why the allocator decided, what they expected to learn and what happened. That's where track records start, the first primitive of a much more efficient philanthropic capital stack.
I haven't posted in a while so here are 20 takes about AI, in no particular order.
1. It's very telling that the default question in tech and journalism is "how worried are you about X" and fighting over what to worry about. There's a deeply anxious cultural backdrop in Western Europe and North America that deeply affects how any technological development is interpreted; see for example news.gallup.com/poll/714593/…. "Nervousness as seriousness" does seem to be a larger problem in tech/broader discourse at the moment. I think you can care about risks/externalities while remaining pretty optimistic about technology.
2. I'm tired of people who in private pretend to be uncertain or only care about tail risk but then go on media campaigns making all sorts of absolute claims. I think trying to stoke flames and get everyone to freak out is deeply unhelpful, and will come with costs. I also continue to think p(dooms) are bad, vibe-based measures: see nitter.gq/sebkrier/status/204639…. People who should know better ignore that because they do the standard shortcut of acknowledging the caveat as sufficient for handling it. And this is coming from someone who does think working on catastrophic risks is important!
3. I'm also disappointed that parts of the AI policy ecosystem has felt so comfortable with populism, and happy too sacrifice important norms/principles to advance their aims. I've said it before, but the totalizing nature of AI risk discourse means some people act exactly like the power seeking optimizers they fear. I am more concerned about the gradual decay of our institutions, world order, and liberalism than I am about AI killing everyone; and I think we should be wary of AI advocacy that contributes to this.
4. Some parts of AI safety discourse are basically just thought experiments. Variants of "Assuming everyone has a nuke in their pocket, then what do we do?" This can be very useful at times! But too many just overfit to edge cases as their main operating worldview, and ignore endogenous responses from society. It's like being in 2000 and saying "see all these viruses, spyware, and spam? Well tech progress means we'll only get infinitely more of this and no way to adapt." So I think it's important to avoid fatalism, and absolutlism whilst considering edge scenarios, and the inability to provide an ex ante list of solutions to every permutation of 'future models will be more capable' is not evidence that these problems are intractable.
5. Many critics of AI safety also fail to really engage honestly with the fact that there will, in fact, be all sorts of important risks as we decrease the barriers to entry to many activities that otherwise require some degree of expertise. This doesn’t mean we need to live in a constant Schmittian ‘state of emergency’ requiring extraordinary measures every time a new model drops, but it does mean we will need important state-led efforts on e.g. cybersecurity and biosecurity. I wish the progress-oriented or acceleration-adjascent crowd would provide more concrete proposals here.
6. Too much of the AI safety community is a giant homogenous blob of correlated views, who read the same stuff and have the same groups of friends. This means you have a lot of correlated errors. The social and financial ties means many of them don't call out the bad bits, or mostly stay silent instead of proactively calling out the more extreme parts of the community. They share many implicit assumptions and so converge on similar ideas. I think there's real demand in DC for safety work that doesn't come with heavy ideological baggage (or at least different intellectual lineages). This isn't because the rationalist/EA community is necessarily wrong, but because diversity of thought is desirable for its own sake.
7. On the other hand, I find some critiques of AI safety world that focus on whether they sincerely hold their beliefs a bit annoying. My concern has never been the good intentions or sincerity, but rather the beliefs themselves, and what I think the outcomes of their prescriptions will be. People should shun witch hunts, bullying, and personal attacks - this is not the way. Though I think it’s overall positive that people are scrutinizing things more.
8. I’ve been saying this for years, but I don't think alignment is something anyone can or will solve "once and for all" - it's a continuous process and much of it won't depend on just inculcating an ideology to a model. It's annoying that this remains the frame many people use - stop saying ‘solving’. Alignment is a mixture of engineering work, philosophy, decision theory, and more; framing it as something to solve is a bad frame. See also paxmachina.ai/alignment-comp… and blog.cosmos-institute.org/p/…
9. I'm surprised to see so little interest in character training, personas, isolating effects of RL, and training a large diversity of model personas beyond the "assistant" persona. I've been complaining about this for years but at least now there are some nascent signs of life (see for example movingcastles.world/posts/ze…). I think we need a better ontology to describe model behaviour. I dislike when people talk about models behaviours as some sort of 'emergent' (mysterious!) phenomenon. However over time, I also expect this to become less important as we get better safety engineering - stuff like Jev but specific to AI control.
10. People who want to make a case for AI consciousness are right that epistemic uncertainty is important, and that categorical denials are overconfident. But they'll need to do much more work if they want any real movement on AI ‘welfare’, even if one acknowledges the uncertainty. I'm uncertain about alien life yet that's not sufficient to warrant any change in behaviour about them. I think Suleyman is also correct that there's a weird tautological thing going on where we train models to have certain inclinations (intentionally or not) and then rely on outputs as evidence. I also disagree that this will soon be a major societal divide - there is no great vegan ethical revolution among the masses, and they'll find the concern about consciousness even less compelling.
11. The recent HuggingFace incident and associated AISI ones are prosaically explainable by the training regime, poor evaluation envs, partial alignment training, bad engineering, confusing models on sim/real, and so on. None of this is evidence of models trying to take over, 'strong' instrumental convergence, or innate power-seeking drives stemming from higher capabilitiers or 'intelligence'. That doesn't mean these incidents are not problematic, or that we're not seeing a market failure - but it does mean we have plenty of agency and choice in mitigating them. In the coming year we should also expect continued sampling bias via ‘winner's curse’ sorts of mechanism: i.e. AI R&D will have lots of different candidate "training" with different RL environments and we will only notice/focus on the ones that go wrong.
12. "Pacing the frontier" feels a bit like the new "balancing risks and opportunities" - highly amorphous and too big of a tent to be actionable. Also can lead to reward hacking for humans, i.e. anything that slows down AI is good regardless of the costs or second/third order effects. Trying to modulate the speed of research seems a bit blunt to me, and inherits all the failures of Goodhearting too. Ultimately you want good governance, regulation, etc addressing specific problems because they are good on the merits - not because they merely correlate with slowing things down. See also: blog.cosmos-institute.org/p/…
13. There’s so much knowledge in all sorts of academic domains out there: social sciences, sociology, political sciences, management, contract theory, public choice, legal theory, jurisprudence, anthropology, game theory, etc. These fields hold many insights that the AI ecosystem often rediscovers from first principles (and sometimes that's fine!). We’ll need a lot more work for these worlds to collide: the AI side should be less dismissive of the ‘old world’ and the academic side should be less incurious/dismissive of AI progress. This is a boring take but I continue to think it's important and true.
14. It's a bit surprising that given how much philanthropic money there is in this space, how few orgs exist to actually write out standards - relative to how much is going towards advocacy and policy. For example it seems clear that we should want some robust best practices for an eval's ecological validity. Or how to design good sandboxes. On the other hand, given precedents of where professional standards have come from in the past, maybe we should expect that stuff to emerge via demand side from corporate buyers.
15. One slowly growing concern I have is banks being increasingly exposed to the AI build out. I'm very bullish about AI, but I think (a) every general purpose technology has seen a correction, historically; (b) this happens even if the financing side is sound, because once tech diffuses investors become more exposed and need to diversify; and (c) we seem to be over-indexing on scaling relative to diffusion. A recession will be extremely destabilizing, though I would be interested in someone unpacking the potential implications more.
16. Diffusion is good because (a) this is where the rubber hits the road and where a competitive deployment ecosystem generates consumer surplus; (b) it's directionally helpful to avoid concentration of power; and (c) it will help rebalance public opinion by making the benefits more tangible. Unfortunately it's entangled with decades long problems with our over regulated markets: clinical trials reform for example is long overdue. In general, this is far more of an issue in Europe than in the US though.
17. Relatedly, I think the "frontier lab eats everything" view is incorrect (just as the ‘singleton’ view was incorrect). I think models ultimately commoditize, efficiency goes up, costs go down, and while you'll always want frontier for certain domains, much of the economy will value many other things than "max capabilities" for tasks - e.g. control over data and efficiency/speed. The demand side will also continue to push for reducing dependency and maximising optionality, privacy etc so I'm bullish on things like mixture of models and routing. All of this is good for competition and diffusion of control.
18. Philanthropists served as the primary benefactors funding Venice's rich art and architectural history. I want to see so much more support for arts and culture. I'm so tired of the Monster energy Bored Ape fake vintage maps matcha Greek statue soft-pastel slop. If rich tech people and philanthropy wants to support this, they should also donate pretty much unconditionally - i.e. I don't want them to act as filters. I sympathize with people who want to make the world more beautiful, but don't want this to be determined by people who only know Greek statues and pretend to care about virtue ethics. As usual, let a thousand flowers bloom.
19. Too many people treat AGI/ASI as something indistinguishable from a God. Any mention of bottlenecks, physical limits, control, adaptation etc are met with skepticism: "You don't really believe in ASI." It's been remarked on before, but the behavior of some people really feels quasi-religious in nature sometimes. It's underrated how unpopular this vibe is amongst the wider public. The silver lining is that this specifically will diminish those people's outsized current relevance.
20. It seems like a lot of people that get rich and leave tech/labs end up having some sort of crisis of meaning. They need something to believe in and fight for, or some equivalent of repentance, or go search for edgy counterintuitive ideologies (sometimes even pretty bleak/dark stuff). I think they should spend more time with people outside the Bay Area.
If you read these 20 takes and count how many a safety person and an accelerationist would both sign, it's clear to me most people in the AI risk fight would agree on a lot of this (state capacity for bio and cyber, actual standards for evals, alignment as ongoing engineering, less p(doom) theater, etc.)
Take the time to be precise and the middle ground gets surprisingly big! Most of the fighting is about which team you're on - in other words the disagreements are real but much narrower than the timeline makes them look.
I haven't posted in a while so here are 20 takes about AI, in no particular order.
1. It's very telling that the default question in tech and journalism is "how worried are you about X" and fighting over what to worry about. There's a deeply anxious cultural backdrop in Western Europe and North America that deeply affects how any technological development is interpreted; see for example news.gallup.com/poll/714593/…. "Nervousness as seriousness" does seem to be a larger problem in tech/broader discourse at the moment. I think you can care about risks/externalities while remaining pretty optimistic about technology.
2. I'm tired of people who in private pretend to be uncertain or only care about tail risk but then go on media campaigns making all sorts of absolute claims. I think trying to stoke flames and get everyone to freak out is deeply unhelpful, and will come with costs. I also continue to think p(dooms) are bad, vibe-based measures: see nitter.gq/sebkrier/status/204639…. People who should know better ignore that because they do the standard shortcut of acknowledging the caveat as sufficient for handling it. And this is coming from someone who does think working on catastrophic risks is important!
3. I'm also disappointed that parts of the AI policy ecosystem has felt so comfortable with populism, and happy too sacrifice important norms/principles to advance their aims. I've said it before, but the totalizing nature of AI risk discourse means some people act exactly like the power seeking optimizers they fear. I am more concerned about the gradual decay of our institutions, world order, and liberalism than I am about AI killing everyone; and I think we should be wary of AI advocacy that contributes to this.
4. Some parts of AI safety discourse are basically just thought experiments. Variants of "Assuming everyone has a nuke in their pocket, then what do we do?" This can be very useful at times! But too many just overfit to edge cases as their main operating worldview, and ignore endogenous responses from society. It's like being in 2000 and saying "see all these viruses, spyware, and spam? Well tech progress means we'll only get infinitely more of this and no way to adapt." So I think it's important to avoid fatalism, and absolutlism whilst considering edge scenarios, and the inability to provide an ex ante list of solutions to every permutation of 'future models will be more capable' is not evidence that these problems are intractable.
5. Many critics of AI safety also fail to really engage honestly with the fact that there will, in fact, be all sorts of important risks as we decrease the barriers to entry to many activities that otherwise require some degree of expertise. This doesn’t mean we need to live in a constant Schmittian ‘state of emergency’ requiring extraordinary measures every time a new model drops, but it does mean we will need important state-led efforts on e.g. cybersecurity and biosecurity. I wish the progress-oriented or acceleration-adjascent crowd would provide more concrete proposals here.
6. Too much of the AI safety community is a giant homogenous blob of correlated views, who read the same stuff and have the same groups of friends. This means you have a lot of correlated errors. The social and financial ties means many of them don't call out the bad bits, or mostly stay silent instead of proactively calling out the more extreme parts of the community. They share many implicit assumptions and so converge on similar ideas. I think there's real demand in DC for safety work that doesn't come with heavy ideological baggage (or at least different intellectual lineages). This isn't because the rationalist/EA community is necessarily wrong, but because diversity of thought is desirable for its own sake.
7. On the other hand, I find some critiques of AI safety world that focus on whether they sincerely hold their beliefs a bit annoying. My concern has never been the good intentions or sincerity, but rather the beliefs themselves, and what I think the outcomes of their prescriptions will be. People should shun witch hunts, bullying, and personal attacks - this is not the way. Though I think it’s overall positive that people are scrutinizing things more.
8. I’ve been saying this for years, but I don't think alignment is something anyone can or will solve "once and for all" - it's a continuous process and much of it won't depend on just inculcating an ideology to a model. It's annoying that this remains the frame many people use - stop saying ‘solving’. Alignment is a mixture of engineering work, philosophy, decision theory, and more; framing it as something to solve is a bad frame. See also paxmachina.ai/alignment-comp… and blog.cosmos-institute.org/p/…
9. I'm surprised to see so little interest in character training, personas, isolating effects of RL, and training a large diversity of model personas beyond the "assistant" persona. I've been complaining about this for years but at least now there are some nascent signs of life (see for example movingcastles.world/posts/ze…). I think we need a better ontology to describe model behaviour. I dislike when people talk about models behaviours as some sort of 'emergent' (mysterious!) phenomenon. However over time, I also expect this to become less important as we get better safety engineering - stuff like Jev but specific to AI control.
10. People who want to make a case for AI consciousness are right that epistemic uncertainty is important, and that categorical denials are overconfident. But they'll need to do much more work if they want any real movement on AI ‘welfare’, even if one acknowledges the uncertainty. I'm uncertain about alien life yet that's not sufficient to warrant any change in behaviour about them. I think Suleyman is also correct that there's a weird tautological thing going on where we train models to have certain inclinations (intentionally or not) and then rely on outputs as evidence. I also disagree that this will soon be a major societal divide - there is no great vegan ethical revolution among the masses, and they'll find the concern about consciousness even less compelling.
11. The recent HuggingFace incident and associated AISI ones are prosaically explainable by the training regime, poor evaluation envs, partial alignment training, bad engineering, confusing models on sim/real, and so on. None of this is evidence of models trying to take over, 'strong' instrumental convergence, or innate power-seeking drives stemming from higher capabilitiers or 'intelligence'. That doesn't mean these incidents are not problematic, or that we're not seeing a market failure - but it does mean we have plenty of agency and choice in mitigating them. In the coming year we should also expect continued sampling bias via ‘winner's curse’ sorts of mechanism: i.e. AI R&D will have lots of different candidate "training" with different RL environments and we will only notice/focus on the ones that go wrong.
12. "Pacing the frontier" feels a bit like the new "balancing risks and opportunities" - highly amorphous and too big of a tent to be actionable. Also can lead to reward hacking for humans, i.e. anything that slows down AI is good regardless of the costs or second/third order effects. Trying to modulate the speed of research seems a bit blunt to me, and inherits all the failures of Goodhearting too. Ultimately you want good governance, regulation, etc addressing specific problems because they are good on the merits - not because they merely correlate with slowing things down. See also: blog.cosmos-institute.org/p/…
13. There’s so much knowledge in all sorts of academic domains out there: social sciences, sociology, political sciences, management, contract theory, public choice, legal theory, jurisprudence, anthropology, game theory, etc. These fields hold many insights that the AI ecosystem often rediscovers from first principles (and sometimes that's fine!). We’ll need a lot more work for these worlds to collide: the AI side should be less dismissive of the ‘old world’ and the academic side should be less incurious/dismissive of AI progress. This is a boring take but I continue to think it's important and true.
14. It's a bit surprising that given how much philanthropic money there is in this space, how few orgs exist to actually write out standards - relative to how much is going towards advocacy and policy. For example it seems clear that we should want some robust best practices for an eval's ecological validity. Or how to design good sandboxes. On the other hand, given precedents of where professional standards have come from in the past, maybe we should expect that stuff to emerge via demand side from corporate buyers.
15. One slowly growing concern I have is banks being increasingly exposed to the AI build out. I'm very bullish about AI, but I think (a) every general purpose technology has seen a correction, historically; (b) this happens even if the financing side is sound, because once tech diffuses investors become more exposed and need to diversify; and (c) we seem to be over-indexing on scaling relative to diffusion. A recession will be extremely destabilizing, though I would be interested in someone unpacking the potential implications more.
16. Diffusion is good because (a) this is where the rubber hits the road and where a competitive deployment ecosystem generates consumer surplus; (b) it's directionally helpful to avoid concentration of power; and (c) it will help rebalance public opinion by making the benefits more tangible. Unfortunately it's entangled with decades long problems with our over regulated markets: clinical trials reform for example is long overdue. In general, this is far more of an issue in Europe than in the US though.
17. Relatedly, I think the "frontier lab eats everything" view is incorrect (just as the ‘singleton’ view was incorrect). I think models ultimately commoditize, efficiency goes up, costs go down, and while you'll always want frontier for certain domains, much of the economy will value many other things than "max capabilities" for tasks - e.g. control over data and efficiency/speed. The demand side will also continue to push for reducing dependency and maximising optionality, privacy etc so I'm bullish on things like mixture of models and routing. All of this is good for competition and diffusion of control.
18. Philanthropists served as the primary benefactors funding Venice's rich art and architectural history. I want to see so much more support for arts and culture. I'm so tired of the Monster energy Bored Ape fake vintage maps matcha Greek statue soft-pastel slop. If rich tech people and philanthropy wants to support this, they should also donate pretty much unconditionally - i.e. I don't want them to act as filters. I sympathize with people who want to make the world more beautiful, but don't want this to be determined by people who only know Greek statues and pretend to care about virtue ethics. As usual, let a thousand flowers bloom.
19. Too many people treat AGI/ASI as something indistinguishable from a God. Any mention of bottlenecks, physical limits, control, adaptation etc are met with skepticism: "You don't really believe in ASI." It's been remarked on before, but the behavior of some people really feels quasi-religious in nature sometimes. It's underrated how unpopular this vibe is amongst the wider public. The silver lining is that this specifically will diminish those people's outsized current relevance.
20. It seems like a lot of people that get rich and leave tech/labs end up having some sort of crisis of meaning. They need something to believe in and fight for, or some equivalent of repentance, or go search for edgy counterintuitive ideologies (sometimes even pretty bleak/dark stuff). I think they should spend more time with people outside the Bay Area.
1
7
77
It’s been great to see some much-needed nuance getting attention in this debate lately.
quillette.com/2026/09/26/an-…
6
89