Theoretical computer scientist working on the mathematical foundations of AI alignment.
Toronto, Ontario
Joined July 2015
- Tweets848
- Following800
- Followers131
- Likes3.4K
Now that math has been solved, let's talk about something that hasn't: AI alignment.
Vanessa Kosoy and I uploaded a new paper to arxiv: arxiv.org/abs/2610.08740
We argue that for solving alignment it is important to design computationally tractable learners in the unrealizable setting, and then design such learners for a special class of unrealizable bandits. Here is a summary in three parts of why we want computationally tractable learners for unrealizable learning.
Why learning?
The short answer is that we want to automate stuff but we don't have the specs.
First, we aren't able to provide the equations that govern the universe, because we just don't have them.
And worse, we aren't even able to specify exactly what *we* want and exactly what *we* are thinking about. We have concepts that seem well-defined in our heads but we aren't able to write them down precisely. We can see a picture of a cat and know that it's a cat, but we can't write a program from scratch that does the same. We sort of know what's good and what's bad for us but, again, we aren't able to define it.
Thus any agent that functions well in this world and is aligned with our goals and desires must learn these ideas itself.
Why computationally tractable?
This could be contentious in some circles, but it does seem like it is an empirical fact about our physical universe that large, worst-case instances of NP-complete problems aren't solvable by a physical process in a reasonable amount of time. An agent that develops superhuman capabilities is not going to do that by solving worst-case instances of NP-complete problems. If something appears to be solving NP-complete problems (such as AlphaFold), it's likely doing that because it has learned something about the distribution from which the universe is generating input instances to the problem, and that just goes back to the previous point.
Why unrealizable?
The short answer is that learning the laws of a universe while being a part of it is likely an unrealizable learning problem.
Somewhat relatedly, learning in an environment that contains other agents like you is also likely unrealizable. To quote the paper:
"Suppose Alice and Bob are playing a known game. Neither knows the other’s strategy but assumes that they lie in a class H of strategies. Can each player simultaneously achieve a low regret? The issue is that the strategy that achieves a low regret against all strategies in H may not itself be a member of H. In that case, if Alice’s strategy is in H, then Bob’s strategy will have to be outside of H and at least one of the players needs to solve an unrealizable learning problem.
More generally, realizability works only when the agent is more expressive than its environment. However, if the environment contains other agents that are similar or more powerful than the agent doing the learning then realizability fails. This can happen, for example, when Alice and Bob are running similar source codes, or when Bob is able to predict Alice’s actions."
5
149
Hey, I also uploaded a paper to arxiv today
We’re releasing a broad range of new mathematical results produced by an internal frontier model.
We’ve been consulting with the independent Advisory Group on Mathematics and Artificial Intelligence at the Institute for Advanced Study, and we have drawn on their advice and public recommendations to inform how we release these results.
github.com/openai/math
1
16
1,160
If an AI proves a theorem but no human understands the proof, is the proof still useful?
I don't think the answer to this is always no.
I think math has played three roles broadly speaking:
1. as a sport (who can solve the most difficult problems?)
2. as an art (creating beautiful and elegant theories)
3. as social good (proving things that are actually useful for society)
While the rise of superhuman AI mathematicians is bad news for 1 and 2, it could be good news for 3.
In particular, I think AI alignment itself is a math question and falls under 3.
2
9
342
Vinayak Pathak retweeted
I'm excited to co-found this new AI safety nonprofit research institute in Cambridge, MA, with an amazing team. Join us in making AI safe by design.
resi.org
The Institute for Responsible Superintelligence (RESI): a nonprofit founded by Shafi Goldwasser, Adam Kalai, Vinod Vaikuntanathan. @ShafiGoldwasser @Vinod_MIT @AdamFungi
resi.org
Our goal: build scientific foundations needed to make superintelligence safe by design
Vinayak Pathak retweeted
I don’t think it is rational for anyone to be doing capabilities research at a frontier lab right now. We are not in a Prisoners Dilemma: the situation is very dangerous, and if one person or lab stops it makes it easier and more peer-compatible for other people or labs to stop.
lol strong agree.
People seem a lot more upset about AI slop than the large amounts of human slop we have had to deal with all these years.
A modest proposal: authors who submit math papers to arxiv containing obvious human-written errors that Opus 4.6 or GPT 5.5 would flag should receive a 1-year arxiv ban.
Why should I read your paper if you haven't even taken basic steps to ensure its accuracy?
1
198
Oh come on, having a few hallucinated citations in the paper isn't going to make NeurIPS paper quality that much worse than it already is.
Okay so, we just found that over 50 papers published at @Neurips 2025 have AI hallucinations
I don't think people realize how bad the slop is right now
It's not just that researchers from @GoogleDeepMind, @Meta, @MIT, @Cambridge_Uni are using AI - they allowed LLMs to generate hallucinations in their papers and didn't notice at all.
It's insane that these made it through peer review👇
1
7
302
This paragraph from a recent NYT article makes it sound like renting isn't throwing money away just because owning has maintenance costs. But as I explain in my article firstprinciples.substack.com…, renting is *never* throwing money away, even if owning has 0 maintenance costs.
1
102
In fact, whether you rent or own, you are *always* paying rent. If you live in the place you own, you are paying rent by missing out on the money you would have received if you had rented it out to a tenant instead.
1
86
Link to the nyt article in question: nytimes.com/2024/05/19/busin…
61
This should be pretty uncontroversial but somehow it isn't
open.substack.com/pub/firstp…
1
89
Same message as the @UWaterloo math building
The specific style chosen is less important than the message underlying it. Look at the FBI building in Washington — its huge concrete overhang makes you, the citizen, feel small as you approach.
1
346
What a nice poll! I was really hoping to see close to 0% yes. The current number is depressing. 😔
If this is ML, god knows what’s going on in fields like health sciences.
This question is for everyone who has published at NeurIPS/ICML/ICLR/CVPR/AAAI/ACL/etc. with empirical results/experiments.
Have you personally published something that, in your own opinion, "doesn't actually work," but the experiments paint the picture that it does?
10%Yes
22%No
67%Show me
1,529 votes • Final results 1
2
2,693
Apologies for the ridiculous number of uhs and ahs in my 20 min long presentation. For best results watch at 1.5X the speed
Thanks to our local organizers @abhatt2 and @j_m_scarlett, and to our student volunteers, the recordings of all the sessions of ALT'23 are now online: tinyurl.com/alt2023videos
Enjoy!
5
820
Didn’t know that one way to make cilantro chutney was to simply leave it in the fridge for three weeks. No messy blenders, no need to wash any dishes
1
141
