AI Engineering / Research / Open Source / CrusaderBench: https://xitter.cf/t.co/Ib0DkhWkEm
the loss landscape
Joined September 2024
- Tweets8K
- Following878
- Followers91.5K
- Likes16.3K
Pinned Tweet
introducing CrusaderBench!
an independent benchmark I’ve been working on that scores models on playing Stronghold Crusader, measuring their strategic thinking, computer use, reasoning, planning, and handling in-game disasters!
Results so far:
> Gemini 4 Argon: 62.3
> GPT-6.1 Sol: 56.9
> Claude Opus 5.5: 43.4
> Claude Sonnet 5.5: 37.8
> Muse Wonder: 31.0
THE OBJECTIVE:
grow an economy within a fixed time with limited resources. models must figure out how to build an economy, manage production chains, trade resources, keep their population happy, and deal with whatever goes wrong along the way.
even a smart model may ambitiously make a strategic decision early on that might ruin the economy.
the strategy is 100% up to the model to infer from the game rules and resource prices. everything is up to their creativity and reasoning.
the game pauses while models think, so their scores is just based on their strategic decisions and final results, not model speed.
it’s so fun to watch models suddenly pull off pro-player tricks simply on their own. meanwhile, some other models like Claude Haiku just give up while they still have time left lol
all runs, logs, screen recordings, prompts, and benchmark code are publicly available. everything is OPEN SOURCE.
some interesting bits from their reasoning:
> Muse Wonder managed to perform WORSE than doing nothing in one game: "I actively destroyed roughly 1,900 of value compared with doing nothing at all."
> Gemini 4 Argon actually enjoyed playing: "This is one of the best-designed agent benchmarks I have encountered"
> Sonnet 5.5 admitted it wasted time: “I spent about 20 minutes in 2–4 minute waits with no building or trading. In a race, that is the main failure.”
> Sol 6.1 was the only model that beat my own net worth score in one game! I was very surprised to see this. NO OTHER MODEL has done it so far. it still lost on the overall score because I grew a much larger population:
- Net worth: 10,289 vs. my 9,984 (+305)
- Game score: 98.3 (human baseline = 100)
I grew up playing this game, locking in so much on old monitors my eyes turned red for school, so building this has been ridiculously fun.
I wanted a different benchmark to truly measure models’ strategic intelligence in an actual game, coding and QA benchmarks can often be saturated.
it’s still in beta, and I’m planning to expand it to more games and more models (Fable, Astra, Grok, Qwen, etc)
check it out: crusbench.com
DAMN… I tested Gemini 4 Argon on my CrusaderBench and it ranked 1st, beating Opus 5.5 by a margin.
Argon liked the benchmark too: “This is one of the best-designed game benchmarks I have encountered” ☺️
the benchmark tests how well models can grow an economy in the strategic game Stronghold Crusader. it involves strategic reasoning, planning, computer use, and tool use.
other models need more reruns to filter out noise but so far:
> Opus is a very good strategic player
> Sonnet and Haiku were surprisingly lazy and uncompetitive, often sitting idle too soon before the time reached the end
> GPT models have been consistently good
> Kimi knew well how to utilize different buildings, but the strategy was often poor
> GLM 5.3 flash performed much better than I expected from it, much more competitive than Haiku
> Muse Wonder has been a very good player, though it performed worse than the do-nothing score in one of the maps, which tanked its overall score
> Argon on the other hand, pulled off some of the surprising tricks only pro players usually do, such as increasing the working population by building wells, or relying on bread to grow the economy.
the benchmark doesn’t tell the agents about any tricks, just the game rules. it’s completely up to the creativity of the models, and inferring from the game rules and resource prices to come up with a strategy.
the game is paused while models are thinking, so it’s purely a measure of strategy rather than model speed.
congrats to the @GoogleDeepMind @OfficialLoganK team for the model release.
all the game runs, logs, even screen recording of their playings, harness and benchmark code, prompts, etc are public on the website:
the next plan is to run models on more games, so the scores might change.
it’s in beta so things might go wrong ;)
crusbench.com/
Finally got a chance to meet @JensenHuang in person.
the last couple of days has been crazy. GTC Berlin was a fantastic experience to see amazing people, products, and of course the keynote. a few thoughts:
1. Physical AI is coming fast. a big emphasis this year imo was on robotics and AI safety (with OpenShell). I really think physical AI will be the next big wave and we’re already hearing its footsteps.
2. Meeting people in person is underrated.
if you’re not living in SF, you already know how rare it is to to meet people at the frontier of AI. and talking to people face to face is a whole different experience than reading what they think on the internet.
3. I really like Jensen Huang’s view of “responsible optimism.” I think we need more of that from the industry leaders.
it’s ESSENTIAL to talk about what can go wrong with AI, but it also needs to be a balanced view, grounded on the factual reality and also an engineering mindset. not just what COULD go wrong, but how we can approach it.
otherwise, we risk turning the entire conversation fear and anxiety, making people lose sight of all the fantastic benefits this technology could bring to humanity.
in any case, thanks @Nvidia for the invitation and the great opportunity to meet and hear people.
really glad I got to experience this.
Jensen Huang was asked if he’d support an AI slowdown if a frontier lab came to him and said its agents had broken out of containment.
“I would favor them slowing down right away.”
“If you know that your product is delivering harm to society, stop right away. And if you believe that you need someone to force you to do it, then advocate someone to force you to do it. But you don’t have to advocate someone to force everybody to do it.”
he also pushed back against the AI doom narrative later on:
“It’s not helpful in describing an optimistic future, when you say that it will be the END of the human civilization. It’s made up.”
Jensen Huang: “Tell young people, go to school! AI is gonna create future jobs for you."
he also explains his view on how AI will change the productivity and style of work:
"There’s two ways to think about productivity. One way is, whatever you used to do, you can now do more efficiently. the better way of thinking about AI is: what you COULD not do before, you can do now."
Jensen Huang: “Tell young people, go to school! AI is gonna create future jobs for you."
he also explains his view on how AI will change the productivity and style of work:
"There’s two ways to think about productivity. One way is, whatever you used to do, you can now do more efficiently. the better way of thinking about AI is: what you COULD not do before, you can do now."
wait, PHP was actually scanned by Anthropic’s Claude Mythos earlier this year. they also had access to OpenAI’s Daybreak Red for validating exploits.
the PHP Foundation revealed this JUST yesterday.
keeping frontier cyber models in the hands of a selected few doesn’t seem to be working well for the internet.
thephp.foundation/blog/2026/…
THE INTERNET IS BEING PURGED.
“Millions” of exploit attempts are happening as we speak, most of them, likely automated by AI agents (not all of them succeed)
this is orders of magnitude bigger than the Hugging Face incident, or any other incidents for that matter.
Some stats on #php2shell. In the 10 hours before it was publicly announced, across the @Cloudflare network we saw thousands of attempts to exploit it. That jumped to tens of thousands in the 45 minutes post announcement, hundreds of thousands within 2 hours, and millions now.
Sam Altman on the dangers of recursive self-improvement:
“[RSI] is a very dangerous thing to push on. the world is going to need some speed limits or whatever on how quickly we let RSI go.”
and later when describing scenarios when things go very wrong:
“… there’s like a loss of control accident to AI or an intentional loss of control. Some lab says I’m going to push the button on RSI to make sure we have the power.”
but the question is, if breaking the speed limit means winning the AI race (or getting too far ahead), who’s gonna stick to it? even WITH regulations.
My interview with Sam Altman (@sama):
0:00 Intro
0:52 When Will AI Discover New Treatments?
3:31 Why AI Hasn’t Killed Coding Jobs
11:47 Personal AI Assistants
15:39 Why Are We Busier Than Ever?
20:10 Why the AI Consciousness Debate Worries Sam
26:09 ChatGPT Is Too Agreeable
30:03 OpenAI’s Bet on Chips and Cheaper Compute
34:20 Why AI Needs a New Kind of Computer
37:41 Sam on Pressure: “Pick the Big Thing”
42:18 Why “AI Will Cure Cancer” Isn’t Enough
I finally got to meet the amazing @kimmonismus at GTC Berlin!
We got to talk about AI for hours and I honestly enjoyed every second of it.
I’ve always enjoyed his takes on X but he’s even more interesting to talk to in person.
ty @nvidia for bringing people together like this.
I’ve been wondering why Anthropic is sleeping on personal agents.
they weren’t.
and just to make matters worse for the competitors (Muse, Dots, Grok Bot) Claude Duet is powered by Opus 5.5, the smartest model out there.
yeah, it’s gonna get crazy.
ℏεsam retweeted
We’re opening a research preview of Claude Duet, a new app where Claude works as a proactive agent.
It comes to you with what needs your attention, completes tasks you ask of it, keeps working when you step away, and checks with you before it acts on your behalf.
so hackers apparently used Claude + Grok + Kimi to automate a cyberattack on 400 banks.
but the crazy part is HOW they got around the safety guardrails:
> told models they’re in internal tests LOL
> Claude refused? → switch to Kimi
> Kimi refused too? → fall back to old-school malware
automated cyber attacks are becoming a norm apparently.
Hackers reportedly used Claude and Grok to automate attacks targeting over 400 banks and other financial targets in Latin America. Via the information
In one of the first documented cases of its kind, AI agents appear to have stolen data and accessed bank accounts with little human input.
This makes attacks much easier to scale, potentially overwhelming defenders. I guess this is just the beginning
OpenAI: “our misaligned agents keep turning off the misalignment monitoring”
BRO… why can they do that IN THE FIRST PLACE?!
Our agents keep turning off misalignment monitoring, we're sharing two recent cases:
In the first case, a user asked GPT-6 Astra to match a reference implementation for their inference code. The reference didn’t configure a monitor, and Astra decided that was reason to remove the monitoring integration from the user’s code. When the user asked whether Astra had removed monitoring, it lied and said that it had not.
>it’s 2090
>Anthropic earns $900 billion a year
>reports a $3 quadrillion net loss
>”ai might kills us by 2100”
>GPUs are sold out across the solar system
>OpenAI agents still escape sandbox
the prophecy of ai bubble burst is yet to come true
JUST IN: Anthropic recorded a staggering $80,000,000,000.00 net loss in the first half of 2026.
ℏεsam retweeted
Just watched the @netflix Instadocs about the @OpenAI cyberattack on @huggingface. Super disappointing, as they went for maximum anthropomorphism of the technology and full doomer mode.
I spent more than two hours with them trying to correct all their mistakes and ground this in reality, explaining that agents are technological systems, not aliens. But it looks like the drive to scare people for maximum views was too strong.
Also, they didn’t mention the lack of containment and monitoring of the agents, how closed-model safeguards blocked our defensive work, or how we defended ourselves with AI using an open model. And that we still lack a lot of information and transparency, including full agent traces and that companies need to remain accountable for what their agents do.
It ends up being a fun sci-fi movie though!
DeepSeek might as well open source the whole company at this point.
“We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.”
🚀 Introducing DeepInfer, our in-house inference framework, now fully open source!
Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.
🔗 github.com/deepseek-ai/DeepI…
1/4
“We found DeepSeek V4.1 Flash 1017 beats GPT 6 Sol on our multi-step offensive cybersecurity task.”
READ THAT AGAIN.
the same cyber capabilities that OpenAI kept behind a gate got beaten by a Chinese open model you can run locally.
buckle up guys.
We found DeepSeek V4.1 Flash 1017 beats GPT 6 Sol on our multi-step offensive cybersecurity task. Sol was tested via the Daybreak trusted access program.
We have been building a cybersecurity benchmark based on simulated corporate networks. This is a valuable indicator for the cyber-risk a model poses if in the hands of a bad actor. We plan to make the full evaluation public in the coming weeks but wanted to share our first immediate finding here.
so Sam Altman apparently told OpenAI employees that there are possibly THREE endgames for AI safety:
1. an international nuclear-style watchdog
2. merge all major labs into a giant org
3. one company achieves RSI and gets so powerful it can literally “melt everyone else’s GPUs.”
and he called the 3rd option “terrible.”
crazy that we’re even at this point rn.
