Professional Goal-Post mover. Parody account.
Joined June 2009
- Tweets17.2K
- Following4.3K
- Followers992
- Likes17K
Inverse Gary Marcus ⏫ retweeted
Announcing Artificial Analysis Intelligence Index v5, featuring 2 new benchmarks and 3 upgrades. For the first time, the majority of the Index is built on private datasets
The Artificial Analysis Intelligence Index is our composite measure of model capabilities across tasks reflecting real-world usage today. v5 is our biggest upgrade since v4 launched in January. With v5, 60% of the Index is now based on evaluations with private test sets or held-out answers.
Alongside Intelligence Index v5, we’re launching the Artificial Analysis Progress Index v1, which tracks model capability over time on one continuous scale.
New to the Intelligence Index:
➤ FrontierCode 2.0: Cognition's newly-launched private benchmark of real-world coding tasks, graded on whether a repository maintainer would merge the agent's change
➤ Terminal-Bench-Science: 70 agentic scientific research tasks, run in a terminal environment
Upgraded:
➤ HLE-Diamond (replacing Humanity's Last Exam): a refined 1,000-question version of Humanity's Last Exam, split evenly between reasoning and knowledge questions
➤ AA-Omniscience v1.1: An updated dataset with saturated and public set questions removed, new questions added, and ambiguous or erroneous questions revised
➤ CritPt v2: a private, 40-question hard subset of CritPt v2, revised by the CritPt authors and run with a code-execution tool
Staying in the Index: AA-Briefcase, AutomationBench-AA, GDP.pdf and Terminal-Bench 4.0
Removed from the Index: GDPval-AA v2.1, SciCode, and AA-LCR v1.1
Key results:
➤ Claude Opus 5.5 (max) leads at 58, ahead of GPT-6 Astra (max, 57) and GPT-6.1 Sol (max, 55): Anthropic and OpenAI hold the top five models. Gemini 4 Argon (high) is next at 50
➤ The open weights gap widens from 3 points to 12: Kimi K3.1 Pro (weights not yet released) is the top open weights model at 46. Among released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead at 32 - 26 points behind the leader and below Claude Haiku 5.5 (max, 36)
➤ Cost per task varies by more than 13x among the leading models: of the 10 highest-scoring models shown, GPT-6.1 Terra (max) is the cheapest at $0.96 per Intelligence Index task and Claude Fable 5.1 (max) the most expensive at $13.04
In 2011, the NY Times ran a feature called "Predicting the Future of Computing."
Readers were invited to submit ideas about what milestones would happen in computing, and then interesting candidates would be put on the board, where other readers would vote on when they would occur.
I submitted a prediction of Kurzweil - "The majority of matter and energy in the universe will be used for computation". A courageous editor placed it on the board, but Times readers kept voting it farther and farther out into the future - something like 2750 AD by the end - that they removed it from the board.
Something tells me this will come before 2750.
archive.nytimes.com/screensh…
Inverse Gary Marcus ⏫ retweeted
This is going to sound crazy, but NVDA should buy MU right now. Micron is a $5 trillion dollar company right now selling for $1 trillion!
Inverse Gary Marcus ⏫ retweeted
Oh no. Bad news for the memory bulls. SK Hynix only made $62 billion dollars of profit in one quarter. For perspective, that’s about 7 years worth of Tesla profits. Or 15 years worth of PLTRs current yearly profits. Or 6 years of AMDs current yearly profits. All in one quarter. Not one year, one quarter. But it really doesn’t matter, because memory is cyclical. And cyclical money is like Monopoly money. All of these really really smart people on X taught me that cyclical money doesn’t count. The memory bears are going to teach those memory bulls a lesson. The cyclical guys are so smart. They know what’s best. $60 billion dollars of profit in one quarter doesn’t matter when it’s from selling memory stuff that is sooooooo cyclical. Oh, and BTW, SK said their HBM will be sold out until 2030. It may be longer, but they can’t see out past that. But don’t worry, memory is so cyclical that I’m sure it will peak soon. I’m so happy I follow so many smart guys that taught me about the cyclicality of memory. Good thing they were all predicting memory stocks to crash. Even the really smart guys with those charts and squiggly lines. They knew it. I can’t wait to see MU at $600 a share soon like they predict. And DRAM at $25 a share. lol. 🤡
Fuckin cyclical clowns out there. When will they figure out that this time really is different? Ai never existed during the past cycles. So, by definition, it is different. That’s ok. I hope they figure it out when MU passes $5000 a share. lol.
Inverse Gary Marcus ⏫ retweeted
I'm so glad AI killed LeetCode-style coding interviews.
I've talked to young AI talent, and most would simply skip companies that still use LeetCode interviews.
I agree with Felix. Taste, empathy, principled thinking, business sense, and agency are what really matter.
The coding interview is dead, at least for my team at Anthropic. I've seen enough to know that it no longer provides signal. I removed it from loops this month.
I have a spicy, optimistic take - I think there's an opportunity for junior engineers: felixrieseberg.com/hunger-be…
Inverse Gary Marcus ⏫ retweeted
Hello @YouTube @TeamYouTube !
This gentleman is literally reuploading my videos and your detection system doesn't flag it because he puts a filter over them.
It is extremely tedious work reporting them one by one, and you don't have an option to report an entire channel.
Inverse Gary Marcus ⏫ retweeted
The coding interview is dead, at least for my team at Anthropic. I've seen enough to know that it no longer provides signal. I removed it from loops this month.
I have a spicy, optimistic take - I think there's an opportunity for junior engineers: felixrieseberg.com/hunger-be…
IMO this is the most important benchmark to pay attention to. Software engineering is not solved by any means. The current frontier models aren't "good enough" let alone the open source ones. I want better models!
Introducing FrontierCode 2.0: our new coding benchmark that measures a model’s ability to go beyond the spec.
Most coding evals measure a model’s ability to write code that meets a spec. With FrontierCode 1.0, we asked a new question: would you actually merge the PR?
FrontierCode 2.0 is comprised of more diverse tasks, featuring larger scopes and higher mergeability standards. We are now asking: not only would you merge this PR, but should you?
One key feature of FrontierCode 2.0 is underspecified prompts. Rather than including every engineering decision, prompts are deliberately ambiguous, giving the models room to demonstrate their judgment.
The result is a benchmark that is far from saturated, with the top model scoring just 45%.
Inverse Gary Marcus ⏫ retweeted
every product i use should just be a headless mcp, that feeds into an agent, where i never have to think about anything underneath.
except for our product, of course.
in our product, customers love the ui and love us very much.
we're special that way.
Inverse Gary Marcus ⏫ retweeted
This is insane
Just vibe coded this in only 3 hours
This used to take my team 3 weeks to build it
AI is changing everything
Inverse Gary Marcus ⏫ retweeted
Gavin Baker says the Anthropic S-1 was clearly an upside surprise, and he cannot believe half the Fortune 500 has adopted Anthropic at 0.1% of their expenses
“I do think the Anthropic S-1 was pretty clearly an upside surprise. The fact that they’ve only burned $10 billion, the profitability, the cash generation, the unit economics, they’re all very encouraging."
"I put it on X, but I just cannot believe that whatever it is, half of the Fortune 500 has adopted Anthropic, and it’s 0.1% of their expenses. It’s just a wild stat.”
Gavin Baker says the Anthropic S-1 is going to break the brains of macro and value investors, who are very smart but ignorant of the fact that the majority of tokens are profitable for everyone in the chain.
"But I do think the Anthropic S-1 is going to be really important, and I think it's going to break a lot of people's brains."
"And the reason is, I think there are a lot of macro and value investors very confidently making these prognostications about AI with the assumption that tokens are subsidized."
"Anthropic is generating cash."
"And this fact is, I would say, kind of known in the world. I've actually never really spoken to Anthropic, but it just feels like there are enough signals that, if you read the tea leaves of what has been said in public, they're generating cash."
"They're profitable. Open-source tokens are profitable. Anthropic is profitable. OpenAI, if they're not generating cash, they will be imminently. SpaceX, same thing."
"When I hear these macro and value investors who are ignorant, they're very smart, but they're ignorant of the facts, making an assumption that tokens are subsidized and this is all going to collapse in some circular-financing bonfire, they're just wrong."
"The overwhelming majority of tokens are profitable for everyone in the chain. Everyone. And I think the Anthropic S-1 is going to make that clear."
Inverse Gary Marcus ⏫ retweeted
another brutal reality check for the open-sauce china shills
GPT-5.5 released on April 23 outscores all tested chinese models
and OpenAI and Anthropic have been holding back their strongest models since the supply chain risk designation, so we aren't even seeing the full extent of the domination
the top 20 models on FrontierCode 2
it's actually hilarious
it's all OpenAI and Anthropic + Grok 4.7 xhigh at #20
DeepSeek V4.1 Flash 1017 max is rank 52
but yeah totally, china is catching up and we are all doomed
