hill climbing @PrimeIntellect
at the open source
Joined April 2022
- Tweets1.9K
- Following2.5K
- Followers1.9K
- Likes7.5K
Pinned Tweet
agi was the friends we made along the way
Daniel Auras retweeted
we should not like a future where there are 1 to 2 AIs and the rest are illegal. we should want a cambrian explosion of the types and possibilities of minds, experiencing new joys and thinking new thoughts. ensuring that this is compatible with human flourishing is only step 1
i welcome the rising underswell of questioning the eval-industrial complex
remember the good ol’ times where a nerd’s verdict on r/LocalLLaMA was as important as any benchmark and Chatbot Arena actually meant smth?
folks like @WolframRvnwlf ran a crucible of his own tests and reported on the tiniest variations of model personality and behavior
RL tasks with sanitized git history are a boon and a curse
less prone to reward hacks
but also less suited for synthetic task generation
Daniel Auras retweeted
Prime Inference has the highest cache rate of any third-party provider for DeepSeek V4.1 Flash 1017
High cache hit rate means that more of your prompt is billed at the cache-read price
This offers the lowest effective input cost on the board: $0.00443/M
couldn’t agree more
Training models is the best job in the world. It's important to remember this is supposed to be fun. It's hard, confusing 90% of the time, and almost constantly surprising. Its always crazy to me to see people talk about AI research work sometimes in such nihilistic ways. You get to melt GPUs for a living, enjoy it!
It's important to remember negative results are just as valuable as positive results, finding a strong baseline is important even if it means you don't get to run as many experiments, and always be willing to change your mind when presented with evidence.
It feels like the pressure to have big splashing success all the time is dialed up to 11 with all of the funding and attention in the past couple of years. It might feel like focusing on the fundamentals instead of chasing novelty will hold you back, but strong science always wins out. The facts don't care about novelty.
If you work on a team remember to be kind to everyone and be a source of joy. Everyone else feels the pressure of this moment as well. It means more than you think to crack a joke, or encourage people when they get a bad result.
12
634
Daniel Auras retweeted
got tired of figuring out which RL library to use, which inference engine to pair it with, and how to make it all fit on my GPU-poor laptop🥲(RTX 4060 - 8gb vram).
so decided to put my codex subscription to some use and build something for myself!
(repo in the first reply)
Daniel Auras retweeted
Amazing BenchmarkGate [1] reveal by @xeophon where he found serious (even comical) problems in popular benchmarks:
AutomationBench: "[forces the models to] describe their request bodies in prose rather than in JSON Schema" 😅
He also fixed a number issues with the grader, which when combined with the above result in a near saturation of the benchmark!
OfficeQA Pro v2: the grader uses a regex and Python-based extraction, which results in rejecting many valid answers. Additionally, there is evidence of models using web access to retrieve the answer key from HuggingFace (receipts are in the post).
With these and other fixes, this again results in near saturation of the benchmark!
It's very impt to look at data and not trust metrics from third parties. Better yet, you should rely on your own private evals.
[1] BenchmarkGate: the gap between the [un]trustworthiness of benchmarks and the extent to which people rely on them.
don’t miss our latest piece by the benchlord himself @xeophon
Public evaluations are foundational for assessing model capabilities.
We looked at some popular evaluations and found issues, which we fixed.
Sharing our findings:
2
22
1,019
huge!
now we can start hillclimbing this annoyance out of our models
Daniel Auras retweeted
Evals built by @PrimeIntellect residents, fully open:
- MazeBench by @patience_cave
- PMPP-Hard by @myainotez
- PretrainGym by @HuNtEr_J1324
- OpenTTD-Bench by @thesophiaxu
[and even more :) ]
Super happy to support such amazing work, check it out!
Daniel Auras retweeted
Tübi folks, catch @rasdani_ and @hallerite while they're here tomorrow!
Guaranteed to be fun and insightful
@hallerite and i are visiting tübingen tomorrow 🇩🇪
if you wanna chat multi-agent swarms, frontier RL and data or wanna nerd out about your own research shoot us a DM!
putting a 32b distill into the fineprint under “DeepSeek” is straight up criminal
you ABSOLUTELY need to take a look at the name of the model and how this benchmark table is constructed, this is absolute cinema
ty @mattsntlm
14
973
corollary:
co-design with hardware.
don’t get nerdsniped by analogies to biology.
first IPO then selling billions
🚀 0 → 120 BILLION tokens/day in just ONE month!
Just one month since onboarding Prime Inference to OpenRouter, we've scaled from zero to 120B tokens served daily. Note this is just a small portion of Prime Inference 🔥
openrouter.ai/provider/prime…
