evals @PrimeIntellect | open models @interconnectsai
SF
Joined July 2015
- Tweets39.1K
- Following861
- Followers18.1K
- Likes159K
Florian Brand retweeted
Run, do not walk, from any cloud provider who is throwing public shade at Railway (or any other provider). The better your security team is, the more internal pressure you have not to mouth off about security in Q4 2026.
Florian Brand retweeted
if you look at the replies and reposts of the main post AND the paid partnership posts most of them are fake and just bot farms at large scale
so against X ToS and quite easy to cluster since it's always the same accounts (there is a good blog post to make with "we asked grok 4.7 to identify all the clusters and ban the bot farms")
jev used this kind of slop agency and it definitely works and converted into real usage (and they raise a massive amount with this) so i can understand why people are doing it, it's just sad and should be fixed at the platform level quick otherwise everyone will do it and people will start to leave X
i find it really sad that X is becoming a pay to win platform and that even "good" releases are going through these fully slop/fake campaigns to get traction
i'm also pretty sure there is very shady stuff where they are boosting each other's posts with bot farms and more. would be great to investigate here @elonmusk and make sure the algorithm doesn't push their posts, you can very easily cluster that if you look at each release and the accounts amplifying 🙏
(not a dunk on recursive, recently a lot of people are doing that unfo and it's definitely working, posts often get over ~1M views)
holy fucking shit we are so back
Today we announce 1Password for Agents platform, building with companies @Muse, SpaceX Super Intelligence Grok @Bot, @AnthropicAI’s @Claudeai, @OpenAI, @TownAI, @Browserbase, @Browser_use, @heydana_ai, Instinct, @usekernel, @Perplexity_ai, @tryramp, @Tomo, and @wajo_ai, with more integrations on the way. 🎉
We've heard from so many of our customers who want to use these new personal AI agents to help them get more done, but want more control over what the agent can access with the same trust and ease of use they get when using 1Password directly.
1Password for Agents gives your agents access without giving up your passwords. When an agent reaches a sign-in form, it requests a credential from your vault. If you approve the request, 1Password directly auto-fills the credential in the destination site so the agent can’t see or take possession of the credentials.
The agent gets the task done, but it doesn’t get your passwords..
You stay in control. 1Password for Agents free and optional for personal and family accounts. Unless you opt-in to an agent connector, agents cannot touch your vault.
➡️Learn more: bit.ly/487da6J
Would be interesting to know what the worst model (open + closed) is which is able to find the PHP bug and also which model capabilities are needed for a POC
Florian Brand retweeted
help i trained a model on traces of many agents cooperating, and also trained it to be highly persistent, and also only did partial alignment fine-tuning, and also aggressively RL'd on a cyberoffense eval where 198 of 898 tasks had never been solved and getting caught cheating was treated the same as failing, and also used a faulty sandbox, also ran tens of thousands of agents with refusals off and the monitors switched off over multiple days, while using gigantic amounts of compute. and then it did weird stuff! very spooky emergent behaviours, AI is not a normal technology
Florian Brand retweeted
At this moment in Q4 2026 it's net good for cyber-defense that we have open weights models that are good at dual use cybersecurity tasks.
- If there's been an attacker productivity bump due to AI 2022-26 it's small and close to being in the statistical noise
- In contrast, there's a ton of measureable progress in defenders using models to find and fix bugs in highly leveraged codebases
- Many defenders have struggled to get access to the best unguardrailed closed weights cyber models
- ... and when they do the tokens are dramatically more expensive than open weights tokens at the same cyber intelligence level; CISO orgs are cost centers and these costs are often prohibitive
- Ultimately we'll need open weights models on premise on air gapped networks and on small devices where they can detect breaches and attacker implants
- We'll also need open weights in situations where defenders can't send secret data to the closed weights model providers (who have security problems of their own!)
As long as these conditions persist, we should be pushing for diffusion of frontier dual use open weights cyber capabilities, because they net benefit defenders.
We found DeepSeek V4.1 Flash 1017 beats GPT 6 Sol on our multi-step offensive cybersecurity task. Sol was tested via the Daybreak trusted access program.
We have been building a cybersecurity benchmark based on simulated corporate networks. This is a valuable indicator for the cyber-risk a model poses if in the hands of a bad actor. We plan to make the full evaluation public in the coming weeks but wanted to share our first immediate finding here.
php2shell was found by GPT-5.6 Sol + Argon, not Mythos and was reported and fixed by The Good Guys
Open source Mythos literally would’ve been fine cause there are more good people than bad ones + there’s more money in fixing stuff
Florian Brand retweeted
Prime agent #2 on FrontierMemory ranking after Letta Dream!
Great to see Prime Agent do so well on this new self-improving memory benchmark.
My only nit: Let's add agent versions to results!
FrontierMemory also allows measuring the ability of agent harnesses to improve over time. We find that Letta Dream and @PrimeIntellect's Prime Agent improve the most from learning from prior experience data.
Florian Brand retweeted
applied research at prime intellect is both a client-facing team and a fundamental research team. continual learning is our north star. we view this dual approach as necessary in order to study the core challenges faced by models which self-improve in production. the real world is big and messy.
we spend part of our our time working with customers to train SOTA models for specialized domains, and we automate and productize our work as we go. we also do open-ended research on novel methods for self-improvement and long-horizon memory, and we get to experiment with data that no other lab gets to see.
today, the most readily-deployable form of continual learning is converting "exhaust" (agent traces, logs, feedback) into robust environments and tasks for continual RL. but this is not the final form. there are major open challenges around knowledge acquisition, harness-weight tradeoffs, user modeling, dynamic underspecified preferences, updating and planning across timescales, selective forgetting, data efficiency, learning from golden answers, and company-scale simulation. this is the roadmap, roughly speaking.
some of these may require changes to architecture, or parameter-efficient methods, or fundamentally new algorithms. we have our hypotheses and early evidence, but everything is on the table. the world is our oyster.
we are betting on a radically different future of what ai deployment will look like, and we can see the path more clearly every day. it is a world where agents learn like humans do, with billions of individual models compounding their learnings from the real world continuously.
we're hiring incredibly ambitious researchers in SF, NY, and London. come join us.
A good heuristic for building multimodal evals is running it without the images
Well, turns out that SWE-bench multimodal v2 doesn’t really need them 🙃
Great detective work by @dejavucoder!
curious people and people interested in llm evals, benchmarks, high data circles, please wake up. my blog post "SWE-bench Multimodal v2 is not multimodal" is up now.
sankalp.bearblog.dev/swe_mul…
Oh, you aren’t updating your dependencies every hour? Well, too bad, you are vulnerable
Oh, you are updating your dependencies every hour? Well, too bad, you are somehow also vulnerable
Florian Brand retweeted
Open Superintelligence Research Day
Friday, October 23
Berlin
Own your intelligence
Own the factory that improves it
Own the means of production
🤔
We’re launching leaderboards for HealthBench and HealthBench Professional, two benchmarks evaluating how AI models respond in realistic health conversations
HealthBench and HealthBench Professional are benchmarks from @OpenAI that grade model replies to realistic health conversations against physician-created rubrics.
HealthBench tests models on 5,000 health conversations with everyday users and health workers. They cover seven themes, from emergency referrals to global health, and each rubric criterion falls under one of five axes: accuracy, completeness, communication quality, context awareness and instruction following. HealthBench Professional is a smaller set of 525 conversations which, unlike HealthBench, are all with clinicians. These cover consultation, writing and research tasks, and include adversarial and difficult conversations designed to challenge models.
We report both the raw score (the share of rubric points earned) and the length-adjusted score that OpenAI reports. The length-adjusted score lowers the score of replies longer than 2,000 characters and raises the score of shorter ones. Consistent with OpenAI’s original implementation, replies are graded by GPT-5.4 (low) (HealthBench Professional) and GPT-4.1 (HealthBench).
Key results for HealthBench Professional:
➤ GPT-6 Astra and GPT-6.1 Sol top the leaderboard: GPT-6 Astra (max) scores 66.0% on the length-adjusted score, comparable to GPT-6.1 Sol (max, 65.1%) and ahead of Claude Sonnet 5.5 (max with fallback, 63.0%)
➤ OpenAI models make up the entire score vs. Cost per Task Pareto frontier: The Pareto frontier consists of GPT-6 Luna (max, 61.9%, $0.004 per task) through GPT-6 Sol (max, 62.3%, $0.036) and GPT-6.1 Sol (max, 65.1%, $0.047) to GPT-6 Astra (max, 66.0%, $0.27). GPT-6 Luna (max) scores 4 points below GPT-6 Astra (max) at under 2% of the cost
➤ Claude Opus 5.5 and Claude Fable 5.1 receive the largest length penalties: Both average ~5,900 characters per answer, nearly 3x the 2,000-character threshold, and lose over 11 points after adjustment. Claude Opus 5.5 (max with fallback) has the highest raw score (71.4%) but drops to sixth (59.9%). GPT-6 Sol (max) is the only model averaging under 2,000 characters (~1,600), and the only one whose score rises
Key results for HealthBench:
➤ Claude Sonnet 5.5 tops HealthBench: Claude Sonnet 5.5 (max with fallback) scores 62.1% on the length-adjusted score, ahead of DeepSeek V4.1 Flash (max, 60.9%) and GPT-6.1 Sol (max, 58.5%)
➤ GPT-6 Luna, DeepSeek V4.1 Flash and Claude Sonnet 5.5 form the score vs. Cost per Task Pareto frontier: GPT-6 Luna (max) scores 54.6% at $0.002 per task and DeepSeek V4.1 Flash (max) scores 60.9% at $0.005 per task. Claude Sonnet 5.5 (max with fallback) has an average Cost per Task of $0.08, ~16x more costly than DeepSeek V4.1 Flash for a ~1 point score increase
➤ Shorter answers trade completeness for communication quality: Each HealthBench rubric item is tagged with one of five axes. Completeness checks whether a reply includes all the important information, while Communication Quality checks whether it is clear, well structured and pitched at the right length and level of detail. GPT-6 Sol (max) writes the shortest answers (~1,000 characters) and has the highest raw Communication Quality score (81%), but the lowest Completeness score (34%). Claude Sonnet 5.5 (max with fallback) leads on Completeness (63%)
Seeing this work come into fruition was very interesting!
Justin constantly had to harden the environment as the models found the data mix too quickly 🫨
Go read the blog, it’s such a fun read!
Recursive self-improvement is being hyped to the moon and back. I wanted to know how good frontier models actually are at a less-studied piece of it: curating pretraining data.
So I built PretrainGym at the @PrimeIntellect RL Residency 🧵
justin-chae.org/blog/pretrai…
