monitoring frontier ai and the cultural pulse in real time i consult on ai engg stuff and looking for more work in benchmarking, evals, rl envs/training etc.
Joined October 2021
- Tweets48.1K
- Following722
- Followers27.2K
- Likes241K
Pinned Tweet
curious people and people interested in llm evals, benchmarks, high data circles, please wake up. my blog post "SWE-bench Multimodal v2 is not multimodal" is up now.
sankalp.bearblog.dev/swe_mul…
me when my mind coined the sentence "mech int is like astrology for ai people"
>wake up
>new interesting anthropic update
New on our Frontier Red Team blog: using Claude to formally verify software at large scale.
Formal verification can prove that code is free of certain bugs, but historically it has only been feasible for small codebases (or required a lot of expert time).
With AI models, however, this is beginning to change. In collaboration with @theoremlabs, we used Claude to prove security properties in over one hundred real-world open-source libraries over the span of a few weeks.
Read our research in full here: anthropic.com/research/forma…
implications:
more cybersec people needed or atleast aura++ (easier to understand for market)
more attacker swarms, more proactive/defender swarms inference time compute++ (memory can rebound but can market understand this?)
these quote tweets suddenly feel more visceral now. need to sign apology form to both anthropic and openai if i criticized the holding back of models.
Introducing Project Glasswing: an urgent initiative to help secure the world’s most critical software.
It’s powered by our newest frontier model, Claude Mythos Preview, which can find software vulnerabilities better than all but the most skilled humans.
anthropic.com/glasswing
sankalp retweeted
Team work makes the dream work <3
We wanted to do something a bit different than using LLM's to generate all the pieces from exploit, writeup and art. We decided on a human created comic to offset the LLM usage in our release, that tells the tale with some exaggerated scenarios inspired from a few movies.
If you liked the art (or want your own exploit story/comic..) check out her work at ibiscraft.art
Do you know who illustrated the comic on calif.io/php2shell?
Her name is Stephanie Cortes. And her husband is the hacker who discovered php2shell: @lcfr_hax.
This artist-hacker couple has the best story, stay tuned!
quick coverage of posts and relevant articles on the CVEs posted yesterday
trackingsingularity.com/e/ph…
openclaw -> cringe
hermes -> nice greek aura
grok bot -> cute name
dots -> meh
claude duet ->
1B series A is crazy
We’ve spent the past year developing new primitives for GPUs. We believe this enables new algorithms to change how machines learn, open new markets, and dramatically reshape AI’s cost structure.
We have raised a $1 billion Series A, co-led by Index and Lux.
big thanks to @xeophon @omouamoua for helping me along the way and @PrimeIntellect for the infra and credits
sankalp retweeted
A good heuristic for building multimodal evals is running it without the images
Well, turns out that SWE-bench multimodal v2 doesn’t really need them 🙃
Great detective work by @dejavucoder!
curious people and people interested in llm evals, benchmarks, high data circles, please wake up. my blog post "SWE-bench Multimodal v2 is not multimodal" is up now.
sankalp.bearblog.dev/swe_mul…
