Andrew (@AndrewMegalaa) profile picture

Tokenomics @SemiAnalysis_

SF
Joined April 2023
Photo by Andrew (@AndrewMegalaa)Photo by Andrew (@AndrewMegalaa)
I'm fairly confident most harness differences stem from which system prompt is chosen for the model. It's less "native" harness, more "native" system prompt. Some harnesses kind of sandbag their models, like Codex which tells OpenAI models to avoid using subagents and to avoid wasting tokens.
We’re publishing Ramp SWE-Bench v2, featuring 500 tasks built from work shipped to production by over 200 of our engineers, more rigorous verification, and frontier models running in their native harnesses. labs.ramp.com/swebench
2
4
214
This will probably age like milk
Capital One has not experienced any issues impacting customers’ accounts or services and continues to operate normally at this time.
64
Checks out
We looked at the data and the data stared right back at us 🥀 Some insights into AutomationBench, OfficeQA Pro v2, web access in public benches and why monitoring is hard. This has been in the work for months, more to come!
1
3
572
Super excited to launch this with @maxkan alongside our amazing partners!
Today we’re releasing GDPx, a benchmark for measuring agentic capabilities across the most valuable domains. GDPx is a true frontier-level benchmark: even the best model scores just 12% (1/7)🧵
4
2
18
1,252
Imagine having super powerful coding agents that write great @OpenTofuOrg /@PulumiCorp to spin up any infra they need on GCP and still using 15 different more expensive and less secure SaaS providers because they are "easier to use"
2
265
Not surprising though because 1) adversaries don't have Mythos or Astra red access and contrary to popular belief, GLM 5.3 isn't actually there yet 2) frontier hackers aren't targeting your average joe, they are going for institutions and enterprises.
The share of US adults experiencing a cyber incident hasn't grown since Claude Fable 5's release. About 45% reported at least one incident in the past 12 months when we asked in September, compared with 46% in June. Other indicators have increased, such as vulnerability disclosures from notable organizations (around 5× per our previous research). Our poll with @Ipsos measures something different: whether more people experience incidents themselves.
2
279
A $200 AI subscription can be worth $12,000 in tokens. Which plan delivers it is not close. "They're selling to willing buyers at the fair market price. This is capitalism." "You can pay $200 for a Claude plan and get $12,000 of Opus 5.5 tokens at API pricing. So that's obviously a good deal." "You have to say the $200 plan, when running a particular workload on a particular model, gives you some dollar amount. But then you also have 5.5, which is a beast. $6K or $5,000 worth of value." "And when you do that comparison, it's not even a question of who's providing more value. ChatGPT versus Anthropic subscriptions." @maxkan and Andrew Megalaa join @JordanNanos this week
25
5
2
122
38,745
Seems like a lot of people missed this section in our newsletter. Anyone who used both Opus 5.5 and 6.1 Sol knows this isn't even a fair comparison as Opus 5.5 is actually competing for the Astra class in the real world.
The real story is this one
3
174