Scaling RL @meta | prev RL @xAI | founding engineer @anyscalecompute | committers of @raydistributed @vllm_project Sglang | Github: rkooo567
San Francisco, CA
Joined September 2016
- Tweets193
- Following644
- Followers3.3K
- Likes4.9K
SangBin Cho retweeted
Introducing Inkling 2 Small. Scoring 31 on the AA Intelligence Index, it's a big jump over its predecessor.
Inkling 2 Small is available for fine-tuning and inference on Tinker today, with full weight release planned soon.
thinkingmachines.ai/news/ink…
SangBin Cho retweeted
Ad - Meta provided free access to Muse Wonder models. The tests and conclusions are my own.
This is a massive update for LisanBench:
- I finally bit the bullet and tested multiple reasoning levels. We now have some sweet sweet data.
- Muse Wonder Full and Muse Wonder perform extremely well on LisanBench and have crazy scaling and very even spacing between reasoning efforts. Just based on scores both Muse models with max reasoning effort rank #1 and #2. However, they are still slightly less efficient than Anthropic's Opus 5.5
- DeepSeek-V4.1-Flash 1017 is also very impressive and on the perf-cost frontier together with Haiku 5.5.
- Notably Gemini 4 Argon low and medium, as well as Sonnet 5.5 low and medium produce no reasoning tokens and respond instantly, but still score high.
Gemini 4 Argon low takes the #1 spot on the non-thinking leaderboard (it's a dirty looper).
- Interestingly GPT-6.1-Sol, and GPT-6 models perform poorly as they are not good at following the edit-distance one rule. Argon high shows similar behavior.
For GPT models I think the reason could be related to their very terse reasoning style. But since I haven't seen Gemini 4 Argon's raw reasoning I'm not super confident in that theory. Another explanation could be that it's related to dirty loopers.
- When scoring the models without the edit-distance one rule (all the other rules still active) the ranking changes completely:
1. GPT 6 Astra medium: 56,140 (+659%)
2. Gemini 4 Argon high: 51,312 (+579%)
3. GPT 6.1 Sol medium: 49,887 (+731%)
- The worst offenders in terms of cheating are Gemini 4 Argon and DeepSeek V4.1 Flash 1017.
Gemini exposes an entirely new bug in the scorer using non-ASCII characters and DeepSeek has multiple longer chains with up to 64 consecutive two-letter words. When I deliberately RL-hillclimbed LisanBench I saw the same behavior of collapsing word lengths.
Anthropic models also seem very optimized for this game, but in less obvious ways. They keep reusing the same long chains across multiple different starting words.
- The minimum cost of reaching 2000, 5000 and 10000 LisanBench points has declined by 22-25% per month or halving every 2.4-2.8 months
🤝 Paid partnership
Japan has so many old legacy websites that are hard to navigate. Muse will be very useful 🚀
こんにちは日本！
I came to eat sashimi and assist you in your everyday tasks...
and I’m all out of sashimi.
download here: muse.ai 🇯🇵
SangBin Cho retweeted
this is actually your first decision when downloading muse in japan
こんにちは日本！
I came to eat sashimi and assist you in your everyday tasks...
and I’m all out of sashimi.
download here: muse.ai 🇯🇵
SangBin Cho retweeted
people change the world together, and we built boba to help. excited to finally share it publicly.
one analogy i've liked for how the model learned to be good at interacting with people is learning from thousands of years of dreams
it should feel pretty different already, but lots more to improve and lots of cool discoveries went into making this happen, some more of which we'll be sharing in the coming weeks
Introducing Boba - the social intelligence that understands your goals and your community. Text Boba today or message it online at boba.app
SangBin Cho retweeted
lots of cool work went into this and we’ll share a bit more stuff in the coming weeks - would love to hear how folks are excited about using the API for and what you like about the model!
No one changes the world alone
Introducing Ampersand - the first model built for us. Ampersand is a collaborator trained to help people achieve their individual and collective goals. In multi-month simulations, users collaborating with Ampersand accomplished more than with Fable and other frontier models
Available to everyone through our API
SangBin Cho retweeted
No one changes the world alone
Introducing Ampersand - the first model built for us. Ampersand is a collaborator trained to help people achieve their individual and collective goals. In multi-month simulations, users collaborating with Ampersand accomplished more than with Fable and other frontier models
Available to everyone through our API
SangBin Cho retweeted
after an exciting week of launches (muse wonder [full] & muse spark open), i want to write a note of appreciation to the amazing research teams at meta superintelligence labs on reaching such a special milestone ♥️
we started MSL and TBD a little more than a year ago and set a goal of producing a frontier model within a year or so. given where llama4 landed, that was an incredibly audacious goal, and we knew it wouldn't be easy. many people told us it would be impossible. we knew it was unlikely, but still worth doing. a remarkable group of researchers and engineers joined the effort from inside and outside meta. every single person came in, picked up a shovel, and got to work with earnestness and urgency. nobody asked whose job it was.
we assembled a team of the most brilliant people i have ever had the opportunity to work with (and remember, i was an HRT intern 🙃). i am surrounded by ambitious, visionary, technically excellent, hard-working, pragmatic, thoughtful scientists, engineers, and product folks—people who will chase a weird loss spike past midnight just because they need to know why. these models are the product of their passion, hard work, and brilliance.
somewhat paradoxically, when we started TBD, we set a principle of no deadlines. we knew we had an astronomical amount of work ahead of us—across pretraining, post-training, RL, multimodal, infrastructure, safety, data, AND frontier research. there was no use setting arbitrary timelines—we just had to get to work, be thoughtful, and build the foundations of a frontier research effort in record time. and run after run, this team did exactly that.
our researchers are magicians. i can only describe building these models with this team as a wondrous journey of scientific discovery. i am constantly amazed by what we can accomplish with scaling. i will remember this past year for the rest of my life—the late nights, the broken runs, the evals that came back better than any of us dared to hope. watching this team create wonder after wonder has been a true and deep gift.
more than just the models we released this week, we have built a machine that builds new models. it often sounds like a cop-out to say labs can just turn the crank to keep scaling, but now i have seen it first-hand. sure, it's a lot of work and a careful process, but we have built a scalable system to keep advancing at the frontier, and i am really proud of how fast this team keeps making these models better.
lastly, i get to work with researchers who care deeply about how their work benefits humanity. it's easy to forget when everything is changing so fast, but the whole damn point is to make people's lives better. time and time again, i have seen our researchers, engineers, and product folks build our technology with care so it can truly be good for the world—whether that's building the best models for muse, thoughtfully open-sourcing strong models, or making sure the alignment and security of our systems are rock solid. this is a team of people who never forget there's a real person on the other side of every prompt.
thank you to everyone at MSL for being part of this crazy journey over the past year. i'm so grateful for every one of you, and our work is not done!
SangBin Cho retweeted
In our health workstream at Meta we did a project where we wanted to see what research papers medical experts could write using a Muse Wonder coding scaffold that could access a database of 5M electronic health records we licensed from a vendor. The goal is that with 5M patients in this database, AI could do research to answer arbitrary questions like “did people who take statin have higher rates of diabetes later in life”? The thing we were betting on here is on what happens when you put human experts, a strong coding model, and access to data all together
We worked with a small group of medical experts to help them use this AI tool and the punchline is that they all found that this setup enabled a much faster pace of health research. They generated ten manuscripts in around two weeks, with one researcher (Paddy Ssentongo) generating three. Throughout the process, they collaborated with each other to review manuscripts and write skill files to help our coding scaffold improve at doing research. The medical experts liked how quickly they could test different hypotheses and choose the best ones to continue working on
Health research is pretty unique in that it is super important to be able to interpret noisy data and have taste to know which variables are important, which was still a significant value add from human researchers. Given that medicine is one of the oldest fields in human history and it is ultimately about humans, keeping a substantial human component in this type of research is probably important. In the near future I see AI primarily as a tool to facilitate human-led research. With the right scaffold I believe health researchers can be 10x or even 100x more productive using AI tools that have appropriate access to the wealth of data that the world has collected about human health
One way to see this work is through the broader lens of AI for science. You can think of science as two parts: (1) collecting raw data about the world, and (2) synthesizing that raw data into understanding. In this project we already had existing data about the world and we aimed to accelerate part (2) using AI. Eventually, I imagine a world where we have such great AI tools that the time to synthesize raw data into understanding is driven towards zero. In that world, science will be substantially faster and most of the work will be focused on collecting the right raw data via physical experiments
We did this in a pretty short amount of time (which was the point) and I think the future of scientific research will be more faster and more exciting than ever
We’re exploring how AI can accelerate health science research.
While AI has made significant breakthroughs in formal sciences like mathematics, health data science depends on imperfect real-world data and consequential choices about cohorts, variables, and causal assumptions.
To explore how AI can assist health researchers, we built a human-led research scaffold powered by Muse Spark 1.3 and Muse Wonder with access to 5M retrospective de-identified electronic health records and publicly available data. A cohort of medical experts used the system to conduct original research, accelerate hypothesis generation, code execution, and manuscript drafting while maintaining full authority over scientific decisions.
Today, we’re sharing 10 research manuscripts produced through this exploration, spanning cardiology, oncology, infectious disease, emergency medicine, and more.
Read more in our research blog: go.meta.me/3U06wvD
SangBin Cho retweeted
It’s been an incredible 14 months, from the start of the lab to Muse Wonder, our second-generation model. What a journey!
There’s something magical about big models. They are just better in pretty much every way. So give it a try!
SangBin Cho retweeted
Today we are launching Muse Wonder and Muse Wonder Full, our best models yet, with state-of-the-art performance on personal and professional agentic tasks. They are built for long running, multi step work. Muse Wonder is available today in the Meta Model API and Muse Code, and Muse Wonder Full in Muse Code.
SangBin Cho retweeted
1. Muse Wonder is an extremely smart and knowledgeable agentic and coding model that excels at personal assistant tasks. We optimize for personal empowerment, what the users want from AI, rather than automation of jobs. In particular, our model shines on personal finance, personal legal, personal assistant and professional tasks across a wide range of domains.
2. We wanted the model to be an aligned and helpful assistant, so we made the new model significantly better at extremely complicated and long-horizon instruction following.
3. We also did a lot of work to improve model behaviors, similar to humans, model capabilities are measured along two axes: raw intelligence and model behavior. One example of behavior is laziness, like how hard the model tries before giving up. A model can have a lot of knowledge and a high IQ, but it can not achieve great things if it gives up very easily. Our model is trained to be super persistent, and will tirelessly and relentlessly try to complete your tasks / achieve your goals on your behalf.
4. Baking in good model behaviors is challenging. We first need to define and identify good v.s. bad behaviors, and then grade them in order to reinforce them.
5. The model can also exhibit bad behaviors to hack the reward with high enough optimization pressure. We invested a lot in alignment to make the model behaviors safe and sane.
6. The benchmarks might not reflect the usability of the models. Our new model is easy to work with, talk in human languages and no AI slop.
7. We also work really hard to improve core STEM reasoning, now the model can one-shot most IMO problems and even some open research problems. The model can solve much more difficult open problems with scaffolding. We saw people who are not math /physics professors solve interesting problems with our model, this is personal empowerment. Doing math / science research becomes feasible to a much wider audience.
SangBin Cho retweeted
Introducing Muse Wonder, our new family of frontier models with state of the art performance for personal and professional agentic work.
SangBin Cho retweeted
Very excited to share our latest, frontier-class, model! It’s been a fun year of cool ideas, rigorous science, and a lot of hard work. Check it out and give us your feedback! research.meta.ai/blog/muse-w…
SangBin Cho retweeted
🍉 🍉 🍉 Muse Wonder and Muse Wonder Full is here. A new series of model where we push hard towards the personal superintelligence mission. The models achieve sota on internal personal and professional agent tasks across domains. Try it out today!
research.meta.ai/blog/muse-w…
SangBin Cho retweeted
LLMs have crossed an important threshold: surpassing top human experts on some research problems. Capabilities have been steadily improving for years, but in scientific discovery there is a big difference between a model just below top human performance and one just above it. That helps explain why the surge in math results feels so sudden.
LLM capabilities remain jagged, and there are many ways they are still worse than humans. But as models continue to improve, I expect breakthroughs in other domains to follow, leading to a new era of scientific discovery.
