Dev Nag

Another key reason people are spooked: around 2016ish we started seeing the *insane* power of purely self-improving Reinforcement Learning (RL) (think AlphaZero going from no knowledge to superhuman at chess in hours), and it was formative for a lot of folks, in terms of their expectations about the progression of AI systems. Frankly, a lot of us have been waiting for the RL self-play shoe to drop with language models, but for some reason it has been surprisingly stubborn and hasn't happened much before. There seems to be at least somewhat of a breakthrough here on that front, which is spooky. The thing that makes RL scary is the ability to keep self-improving by inventing ever harder tasks for yourself that you can practice against, to keep getting higher and higher quality training data. The traditional sort of "train on the internet + some finetuning at the end" picture doesn't seem likely to crazily spiral out of control in terms of skill, while "invent new programming problems for yourself, then solve them" has much more of the flavor of something that could keep self-improving. To paint an extremely over-simplified picture of how these self-improving RL systems work: They have some part for making decisions, which we'll call the "policy" (it's just the standard term), and then some sort of mechanism for turning this policy into a better policy. In the case of chess in AlphaZero, the policy picks out a move, and the mechanism for turning the policy into a better one is to do tree search. In other words, if you have a policy that finds 1500 elo level moves, you can use that policy to find 1700 elo moves by simply searching over many possible game trees using your policy, and taking the best one. The clever trick of AlphaZero is now this: You now *distill* those 1700 elo moves back into your policy by training on them! This maybe increases your policy to being one that finds 1502 elo moves, but that's okay, because now with tree search it finds 1702 elo moves, which you in turn distill back into your policy, getting out 1504 elo moves, and so on. In a sentence: "we keep distilling what we conclude after a lot of thinking into what we conclude intuitively in a single step of thinking, which in turn improves what we conclude with a lot of thinking, and so on". Note that the above core trick is part of what we've all been waiting to see if someone will figure out for LLMs. The analogy is quite strong. You make up ever harder problems, and your "policy" is just what answers your LLM gives in a single step. The analogy to tree search is letting your LLM do chain-of-thought (CoT) reasoning. The hope is that a model that produces "1500 elo thoughts" shooting from the hip will, via CoT reasoning, produce "1600 elo thoughts" or something, and you can distill those back into the model to get a model that thinks 1501 elo thoughts to start with, and then you can iterate this over and over. Frankly, a lot of us were pretty comforted to see that major labs were not seeming to be having much success with this sort of self-improving RL, so it's a big update to see something like that work so well now. (Why you should pay any attention to what I'm saying: I worked on commercializing LLMs professionally 2019-2021, and then worked on more researchy projects on LLMs after that (e.g. paper in NeurIPS). I've implemented AlphaZero from scratch, and gotten strong models at new games, like my duck chess engine here: peter.website/duck-chess/ana…)