co-founded @parsedlabs before we started hill sprinting @baselabs @baseten, half eng/cs phd @rhodes_trust @UniofOxford
SF
Joined January 2018
- Tweets171
- Following426
- Followers816
- Likes1.4K
Pinned Tweet
It has been easy to back the open source narrative recently with so much exciting progress, but the cyclical swings in energy we have seen seem inevitable. Inspired by the great scientists (including LeBron) pictured in video, @baselabs is a very direct effort to pursue a long-term research agenda that aims to support the WHOLE ecosystem (that we see being a mix between closed and open source models) for the long run. It is backed by @baseten ripping (which has a little more $$ than academia), but has a clear mandate to publish without exception and encourage skepticism. Our manifesto mentions 'Science done behind closed doors is worse science. It cuts off the one mechanism that has ever made knowledge reliable: correction from the outside'. Here's to democratising intelligence!
Today we're announcing Base Labs, a dedicated research organization focused on advancing open-source AI.
We believe in a healthy, open frontier model ecosystem. To enable this, we are working on:
- Blue-sky research on continual learning, the science of RL, and how models learn across their full lifecycle, with every experiment and recipe shared openly.
- The BaseHub Data Foundry: the highest-quality open RL environments, training data, and real-world benchmarks, built for anyone to train and benchmark on.
- Post-post training: taking open-source models and making them better, safer, and more aligned through continual post-training, built on our research, and deployed with our frontier safety stack so organizations can use open-source models with confidence.
- Making models cheaper and more performant through our model performance research.
This is a mission-driven research effort, not a commercial product. We believe the health of the open-source AI ecosystem matters and that the best way to advance it is to do science in the open.
We’re hiring engineers, researchers, and research fellows to advance this mission.
labs.baseten.co/
Mudith Jayasekara retweeted
We trained the #1 decision models in the world, mammoth-1 and elephant-1!
This is our first Base Labs model release and we're excited to open-source all weights and training details. We really focused on scaling high-quality data in these training runs – not only synthetically generating new decision data but stacking how we validated our data. This follows something called our 'swiss cheese' approach to looping over and improving training data.
There are also a bunch more low hanging fruit when it comes to training these models, and we expect that we'll improve quickly over the coming weeks.
Today we release mammoth-1 and elephant-1!
We post-trained mammoth-1, the #1 ranked best decision model in the world, and elephant-1, which is the best model under 15B (and beats Jev).
We're open-sourcing all model weights and provide a technical report detailing all our training and data methods.
Mudith Jayasekara retweeted
@baselabs have the best decision model in the world (Mammoth-1, 27b), and also the best under 15b (Elephant-1 9b, which also beats Jev!) The most exciting thing about this model work is that it crystallises something very clear about RSI: that recursive improvement is here with agentic workflows, but recursive self improvement isn’t here yet. Models are more than capable of iterating on a model that is either one rung down the capability ladder from itself, or one level simpler (ie an agentic LLM working on a decision model) but they are not capable of improving themselves yet. We’ve tried a lot to get LLMs to improve themselves (through building their own RL environments, training themselves, curating mid training data, etc) and it still relies on a heavy human hand in the loop. However, this decision model training was one of the first times where we set up this outer loop and it worked. Our decision model got better and better as we provided high level, meta-guidance to the agent and worked on the outer loop harness to install general principles for alternating the scaling of the data and subsequently correction of the data, both of which made the model better (20 aggregated index points for scaling the data 16x, 8 index points for fixing and cleaning the data!).
We’re offering this for free for a week on @baseten MAPIs, try it out and let us know what you think!
Today we release mammoth-1 and elephant-1!
We post-trained mammoth-1, the #1 ranked best decision model in the world, and elephant-1, which is the best model under 15B (and beats Jev).
We're open-sourcing all model weights and provide a technical report detailing all our training and data methods.
Mudith Jayasekara retweeted
Today we release mammoth-1 and elephant-1!
We post-trained mammoth-1, the #1 ranked best decision model in the world, and elephant-1, which is the best model under 15B (and beats Jev).
We're open-sourcing all model weights and provide a technical report detailing all our training and data methods.
Activation monitoring is one of the lower rungs of the interp ladder that we're using to start our build of the frontier open safety stack. Training probes well is a craft and we're stoked to partner with @GoodfireAI who have been working towards this mission even when interp wasn't in vogue.
Open models can come from anywhere. Securing them shouldn’t depend on where you get them.
Today we're launching Project Beacon to make safety a default for open intelligence.
We're partnering with @GoodfireAI to bring inference and in-line monitoring together for the first time, so safety signals are caught proactively.
Read more in our announcement: baseten.co/blog/safe-open-mo…
Mudith Jayasekara retweeted
Progress that moves the frontier is almost never made alone.
We’re thrilled to announce All-Gather, our first-ever research conference that will bring together the brightest minds in research, founders, and engineers to join us for a day of debate and exchange.
Come hear Noam Brown, Sarah Guo, Horace He, Bryan Catanzaro, Jerry Tworek, Sabri Eyuboglu, Tuhin Srivastava, and many more debate pre-training, post-training, infrastructure, model performance, and emerging fields. Join us on November 18th in San Francisco.
Research has been part of our DNA from our early days of optimizing open-source models through our recent launch of Base Labs. We’ve grown with the open ecosystem, and we’re excited to keep building alongside it.
Space is limited, but we’d love you to join us. Apply here: baseten.co/all-gather/
Mudith Jayasekara retweeted
Each week we share ongoing work from across the Base Labs team. We call these releases 'rollouts'.
This week we share our progress in understanding the low-dimensional structure of RL training and whether recent discoveries about rank-1 extrapolation of early updates really hold.
We find that performance gains largely come from easier problems and fixing doom loops, while harder environments are less linearly extrapolatable.
Mudith Jayasekara retweeted
Exclusive: Inside @baseten
The $13B startup at the centre of the biggest fight in AI - Open vs Closed.
We spent a day at their SF HQ with co-founder & CEO Tuhin Srivastava, Charlie O'Neill, Mudith Jayasekara and the Base Labs team - to understand why they're building an open ecosystem.
Backed by top VCs led by @saranormous @altcap @apoorv03 @wmccarthy11 @saammotamedi @nikiscevak @lachygroom @willreed + more. Plus participation from @nvidia.
0:00 The Open vs Closed AI War
2:18 A Billion AI Requests a Day
3:43 The Best Models Are Chinese
4:06 Tuhin: Don't Rent Intelligence
5:09 Sachin & Tuhin Talk Cricket
6:02 Tuhin: The Real Bottleneck
10:15 Why Launch Base Labs?
11:35 "You Have to Have an Opinion"
12:52 Centralised Control Is the Danger
13:07 Publish Everything, Even Fails
15:08 Hugging Face & the 6-Month Gap
16:10 Why We Still Need Closed
Mudith Jayasekara retweeted
Technical alignment is a science and engineering problem that we can and must solve, and interpretability is the bottleneck. I wrote up Goodfire’s plan to get there.
Mudith Jayasekara retweeted
LangSmith Fine-Tuning 🚀
launched in partnership with our friends @baseten and @FireworksAI_HQ - try it out 🤝
one of AI’s most interesting & difficult problems today is a Data Problem
turning raw trajectories into Environments & curated trajectories to improve your agents
the goal of LangChain Labs is to help every team continuously turn their valuable trace data into better agents via Post-Training & Harness Eng
- trajectories help teams understand agent behavior at scale
- but preparing and mining raw data is really hard today, agents are not good at data curation today
- we want to help you drive this end to end flow from raw data to better agent with post-training
reach out if this is interesting and try it out 🫡
Introducing LangSmith Fine-Tuning and the smithtune CLI.
LangSmith now handles the entire fine-tuning process. Use your traces to train specialized models that cut cost and latency.
Now in Public Beta. langchain.com/blog/langsmith…
Stoked to support traces to trained model work with friends at @LangChain! There are many open questions we are exploring on how we can continue to improve the (semi) automated env creation for RL in a way that is actually useful for downstream task. More to come!
Stoked to launch 𝚜𝚖𝚒𝚝𝚑𝚝𝚞𝚗𝚎 in public beta!
Post-training your agent's policy has never been easier, with training APIs from @baseten and @FireworksAI_HQ becoming increasingly accessible. But sourcing high-quality data to use them effectively remains a challenge.
LangSmith processes hundreds of millions of traces a day, making it a rich system of record for your agent’s behavior.
𝚜𝚖𝚒𝚝𝚑𝚝𝚞𝚗𝚎 turns those trajectories into the data you need for supervised fine-tuning, with an end-to-end pipeline for curation, preparation, training and deployment for models you want to SFT for specific tasks.
We care deeply about giving teams ways to own their own intelligence, and as the open model frontier continues to advance, it’s becoming increasingly clear that post-training on your own agent data will become a key part of that.
Excited to get this into people's hands!
Mudith Jayasekara retweeted
Distillation is a common way to prepare a model for RL where learning occurs from a stronger model, and then improves through trial and error.
But how much does that better starting point help after RL? We conduct preliminary research on this question across model sizes and reasoning tasks. 🧵
We're working towards the open intelligence safety stack! We've always been interested in making models do 'real' work. But we've also been very interested in making sure they are safe and aligned (before it was cool). This started for us with mech interp at @parsedlabs, and now continues at @baseten which serves so much of the world's traffic. The responsibility definitely lies both on the closed and open labs to help shape this work.
Excited to partner with @Thom_Wolf @ClementDelangue @julien_c + team from @huggingface and @eric_ho @banburismus_ + team from @GoodfireAI. We'll be publishing our safety research in the open and are aiming to build as much of it as possible into our serving stack. To start, this will include alignment training, monitoring in prod, and interventions shaped from these detected failures.
A big ecosystem effort!
We believe openness to be an advantage for AI safety. Openness provides more visibility into the behavior of models and, most importantly, greater means of turning safety research into actionable and transparent controls than closed-source.
This is why Baseten and Base Labs are building a stronger safety and security standard for open models, with the launch of our safety infrastructure. Base Labs will develop and publish methods for training and monitoring open models, and Baseten will integrate that work into its deployment infrastructure, live at runtime, and offer this work as a managed service. This will be a standard that is transparent and built into how our models are trained and deployed.
We invite the open-source community to contribute, and are proud to partner with @huggingface and @GoodfireAI to bring this vision to fruition.
Together, we are building an ecosystem of open models that are safe and accessible to all.
Mudith Jayasekara retweeted
Safety is not just for closed models. The closed frontier labs are a canary in the coal mine for what is coming at scale. They give us a glimpse into the future and a window to harden our systems and prepare for abundant intelligence, with all the risks that come along with it. The OpenAI agent swarm attack on Hugging Face is the kind of failure we need to prepare for as open-source models catch up. The providers serving those models (such as Baseten) have a big role in establishing what safety and monitoring standards look like.
We’re proud to be taking the lead on this at @baselabs with our collaborators @huggingface and @GoodfireAI. We’re developing safety research in the open and building it directly into Baseten’s inference infrastructure, with the aim of making it available to all our customers. This includes training models to follow explicit policies, detecting failures at runtime, and connecting those signals to controls that can intervene.
We invite others in the open-source ecosystem to join us in building the tools and standards we’ll all need.
We believe openness to be an advantage for AI safety. Openness provides more visibility into the behavior of models and, most importantly, greater means of turning safety research into actionable and transparent controls than closed-source.
This is why Baseten and Base Labs are building a stronger safety and security standard for open models, with the launch of our safety infrastructure. Base Labs will develop and publish methods for training and monitoring open models, and Baseten will integrate that work into its deployment infrastructure, live at runtime, and offer this work as a managed service. This will be a standard that is transparent and built into how our models are trained and deployed.
We invite the open-source community to contribute, and are proud to partner with @huggingface and @GoodfireAI to bring this vision to fruition.
Together, we are building an ecosystem of open models that are safe and accessible to all.
Mudith Jayasekara retweeted
The future is about continuously turning your valuable data & ideas into better/faster/cheaper models -> Owning your Intelligence
the Baseten team rocks at helping us run tons of experiments to make this happen @AEllisBloor @mudithj 🤝
A lot of work in our Labs team looks like:
- sitting down with a buddy @jakebroekhuizen
- picking a problem/agent to make better, cheaper, faster
- curating data + evals/environments
- doing tons of training experiments, with continuous measurement of every tweak
- Loop (pun intended)
the infrastructure and tooling is here to allow every team to run hundreds of experiments and get the best quality/cost/latency tradeoff for their customers and use-cases
also ngl training is fun and your agents will love the experience
Mudith Jayasekara retweeted
A flavour of our research interests on the Dwarkesh Podcast this week, discussed by our very own @oneill_c. Just the start of a longer conversation about long-horizon RL and frontier open-source training here at Base Labs.
New episode with @johnschulman2, @oneill_c and @BerenMillidge.
I got together with some of the most insightful AI researchers I know who are at the openish companies, because I wanted to hear the details of what's actually happening at the frontier and what comes next.
0:00:00 – Steelmanning the case against RSI
0:18:39 – What’s driving the Chinese labs’ progress
0:28:06 – How will automated AI researchers be trained
0:33:51 – Will long-horizon RL elicit AGI?
0:45:24 – The sim-to-real gap
1:00:33 – How much progress is explained by data?
1:18:03 – Why is RL working so well?
1:24:54 – Move 37 and entropy collapse
1:28:31 – Rapid-fire timelines