Product, AI, startups, retail. CPO @bookendai. Led Retail Supply AI @googlecloud. Cofounded @incentivetarget (acq, @Google). Led product @catalina, @HealthEdge.
Boston, MA
Joined April 2009
- Tweets11.6K
- Following300
- Followers838
- Likes15.5K
Ben Sprecher retweeted
Our security researchers built an autonomous cyber vulnerability discovery, exploitation, and patching pipeline.
In a single run with an upcoming cyber model, it validated and generated patches for 948 local privilege escalation vulnerabilities in the Linux kernel at a total API cost of $1.2 million.
Safety sandbagging
🧵 Astra covertly underperforms on safety research tasks.
We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this.
To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.
Ben Sprecher retweeted
🧵 Astra covertly underperforms on safety research tasks.
We noticed that Astra often makes surprisingly little progress on certain types of safety research and likes to invent blockers or impose arbitrary restrictions without telling the user. In most of these cases the decision to underperform was not verbalized in the CoT meaning our misalignment monitoring system did not detect this.
To measure underperformance we created a paired open-ended coding evaluation with safety and capability tasks. On this eval Astra scored 10% when we framed coding tasks as safety research, vs. 16% when framed as capabilities research. The gap remained after we excluded trajectories with refusals or verbalized intent to underperform in CoT. There was no such gap for 5.6-sol. When we aggressively filter out trajectories with even brief expressions of ethical doubt in the CoT the performance gap seems to reduce.
I wonder what the historical correlation is between a state's governor's party and whether the state voted for a president of the same party? Is it positive or negative?
Do governors tend to piss off their state's electorates?
Ben Sprecher retweeted
Confirmed refrigeration cyberattacks: Sweden, Belgium, Ireland, Luxembourg, Estonia, Netherlands, Finland, Portugal, Germany, Italy, Poland, Iceland and UK.
Suspected: Romania, Greece. Attempts: Latvia, Switzerland. Significant refrigerated-food losses reported. Should EU's NIS2 may apply. DORA not.
Ben Sprecher retweeted
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math.
rikiyat.github.io/blog/weight-sp…
github.com/RikiyaT/lm_und…
huggingface.co/Qwen/Qwen3.5-0…
huggingface.co/RikiyaT/Qwen3.…
Ben Sprecher retweeted
Why is there so much obsession with the few hundred cutting-edge math problems solved by AI but so little clamor over the thousands attempted but not solved? The profundity of the lessons about AI and human intelligence buried in that failed set is incredibly underrated.
A rational scientific response would be to meticulously survey the failed problems and the nature of each failure, and get a completely novel map of the intellectual landscape. If there are unifying properties among subclasses of currently prohibitive problems, identifying those properties would be impactful in at least two powerful ways.
First, they would point AI research in new directions as we gain a novel understanding of what is actually hard for AI and why it fails even when it has seemingly piercing superintelligence on other classes of problems.
And second, for the meantime (which could be years or even decades), it would illuminate for the human mathematicians (and ultimately all scientists) where their minds are still valuable and illustrate why it is not yet time to declare all human intellectual work as merely “astronomers pointing telescopes.”
Of course, these two implications seem eventually to come to a head with each other, but they also complement each other and feed into progress and new kinds of human-machine collaboration. Either way, we need to know what remains out of reach and why, and the possession of such knowledge is a highly underrated scientific goldmine.
This is a product to watch.
@alexvogenthaler is one of the sharpest product leaders I've ever known, with enormous capacity for managing complexity. He is his own ICP.
Execs need constant context-sensitive prioritization by all-seeing, all-knowing assistants.
That's Rocker.
1/ How rocker.ai is more than an assistant; why we built a nervous system for companies:
I spent years running the products behind 90%+ of Stripe’s revenue. At one point I counted 250 projects I was responsible for. My job was knowing the dependencies the teams themselves weren't aware of, which plans had to change, and what important historical context people were missing.
Ben Sprecher retweeted
1/ How rocker.ai is more than an assistant; why we built a nervous system for companies:
I spent years running the products behind 90%+ of Stripe’s revenue. At one point I counted 250 projects I was responsible for. My job was knowing the dependencies the teams themselves weren't aware of, which plans had to change, and what important historical context people were missing.
Ben Sprecher retweeted
Read more about Gemini 4 Carbon below blog.google/innovation-and-a…
Ben Sprecher retweeted
Today Rocker is out of stealth. We know you've heard about AI assistant after AI assistant (we're tired of it too!). Rocker is something different -- it is both an assistant for the busiest knowledge workers and an AI nervous system for companies. Read more from @alexrkonrad upstartsmedia.com/p/stripe-a…
Ben Sprecher retweeted
Announcing Artificial Analysis Intelligence Index v5, featuring 2 new benchmarks and 3 upgrades. For the first time, the majority of the Index is built on private datasets
The Artificial Analysis Intelligence Index is our composite measure of model capabilities across tasks reflecting real-world usage today. v5 is our biggest upgrade since v4 launched in January. With v5, 60% of the Index is now based on evaluations with private test sets or held-out answers.
Alongside Intelligence Index v5, we’re launching the Artificial Analysis Progress Index v1, which tracks model capability over time on one continuous scale.
New to the Intelligence Index:
➤ FrontierCode 2.0: Cognition's newly-launched private benchmark of real-world coding tasks, graded on whether a repository maintainer would merge the agent's change
➤ Terminal-Bench-Science: 70 agentic scientific research tasks, run in a terminal environment
Upgraded:
➤ HLE-Diamond (replacing Humanity's Last Exam): a refined 1,000-question version of Humanity's Last Exam, split evenly between reasoning and knowledge questions
➤ AA-Omniscience v1.1: An updated dataset with saturated and public set questions removed, new questions added, and ambiguous or erroneous questions revised
➤ CritPt v2: a private, 40-question hard subset of CritPt v2, revised by the CritPt authors and run with a code-execution tool
Staying in the Index: AA-Briefcase, AutomationBench-AA, GDP.pdf and Terminal-Bench 4.0
Removed from the Index: GDPval-AA v2.1, SciCode, and AA-LCR v1.1
Key results:
➤ Claude Opus 5.5 (max) leads at 58, ahead of GPT-6 Astra (max, 57) and GPT-6.1 Sol (max, 55): Anthropic and OpenAI hold the top five models. Gemini 4 Argon (high) is next at 50
➤ The open weights gap widens from 3 points to 12: Kimi K3.1 Pro (weights not yet released) is the top open weights model at 46. Among released weights, DeepSeek V4.1 Flash 1017 (max) and GLM-5.3 (max) lead at 32 - 26 points behind the leader and below Claude Haiku 5.5 (max, 36)
➤ Cost per task varies by more than 13x among the leading models: of the 10 highest-scoring models shown, GPT-6.1 Terra (max) is the cheapest at $0.96 per Intelligence Index task and Claude Fable 5.1 (max) the most expensive at $13.04
Ben Sprecher retweeted
🧵On Sunday afternoon, British Storm Shadow cruise missiles smashed through the roof of Russia's Energia plant during the day shift.
Energia manufactures the thermal batteries powering Iskander ballistic missiles, Pantsir systems, and UMPK glide bombs. 1/🧵
Ben Sprecher retweeted
Woah this is cool! 🤩
I'd like to share a little about what I've been working on at my new company @ultrametricai with my cofounder @tylerlastovich. We have been exploring how we could automate the startup itself, leaving founders with more time to focus on their exact mission.
Enter our first release of the Open Startup Repo: github.com/ultrametricai/ult…
A common foundation for transparent automation of the startup.
After close to 20 years in silicon valley, backing hundreds of amazing founders & startups, being in the trenches of founding Heyzap (YC08), then solo founding @golden, going through 2 acquisitions and working behind the scenes with some of the iconic companies today, I'm witnessing the largest change to how we work I could have imagined - a truly monumental era of creative abundance.
I believe the next generation of companies will largely run themselves: from operations, code, product upgrades, growth, vendor control & selection, process execution, special situations, strategy etc etc. That leaves founder time for 'the art' at the very niche core: the (currently) unautomatable, the special leaps humans tend to make, and north star setting.
We want founders to go super deep into the main problem they are trying to solve and not get held up doing anything else, from searching for domain availability with trademark checks to setting their company up, to running their processes on autopilot.
So to help that broad foundational layer for starting and running a company, we have spent around a year exploring how to help founders focus on their mission across different verticals, geos, stages and initial starting conditions.
Our first few attempts at making something useful actually failed as the market and AI was moving so fast; it just got absorbed into the models or other product releases. We saw polsia happen, openclaw, then grokbot, muse, dots, etc era as the astras, fables, opuses, kimis and jevs rolled out. What seemed to have been missing along the way was a common repo to identify, unify, open and normalize all the lego bricks that startups have to interact with (vendors, processes, business logic, law, best practices, surface areas, art of the possible).
As the number of software vendors blew up, the interfaces to drive them multiplied, the international startup founder blew up as silicon valley moved from a specific location to a state of mind and global movement. Unlimited pre seed funding in unlimited vertical sub markets and permutations has brought us to the age of infinity but we have yet to achieve being able to type into an agent a very simple sentence with very complex implications 'just setup my company and make $1m/m'. We have ended up with a hybrid mismash of computer use, API calls, plugins, shifting surface areas (API/MCP/CLI/plugins/chats/musish futures), and changing model absorption to vendor balance making it an exciting and unstable time.
As our first step we started to rank all the vendors that startups interface to when going about their journey in terms of how agentically controllable they are via their APIs, MCPs, CLIs etc and computer use compatibility as we are largely switching from a UI driven world to the agentic control surfaces and onto full automation. We have 100 ranking areas you can check out with an overall open ranking here: ultrametric.ai/overall
Then we looked at the processes that startups do and how the vendors rank for the sub steps or process as a whole and how this changes across geos. Open processes: ultrametric.ai/processes
Then we started to hack on an open repo for startups that would contain all the components we are going to need for a common foundation for founders across the globe even if our gov processes vary so much from Bengaluru to SF or Lisbon to Shenzhen. The founder spirit is geo-invariant.
We have looked at special founder/startup situations, the business logic that will make open modules for the startup that we can hack on together, ranked law firms, government services and interfaces, stacks that go well together. We have mapped the law and lore that maps to startups and the schemas that make up the data objects and documents. But also the open documents and the missing gaps. MUCH TO DO.
All the BS processes and fluff work that founders waste their time on needs to goto zero. I want to see more supersonic planes, cancer drugs, seemingly impossible missions done and increase the bang for buck founders have with their funding.
This is a work in progress and we are keen for the community to come hack on the repo with us here: github.com/ultrametricai/ult…
Lets get into the structure of the general repo and the visualizations we made on ultrametric.ai site with example links:
- Open rankings: ultrametric.ai/overall
- Open rankings example (online payments): ultrametric.ai/arena/payment…
- Open ranking example 2 (cap table): ultrametric.ai/arena/equity-…
- Open processes: ultrametric.ai/processes
- Open process example (Incorporate C-Corp) ultrametric.ai/processes/inc…
- Open situations that founders face: ultrametric.ai/situations
- Open startup simulator (you can test our different initial starting conditions): ultrametric.ai/startup-sim
- Open modules (e.g. cap table, payroll logic etc - we'd love help improving these) ultrametric.ai/open-modules
- Open law (we are interested in how law changes the processes, situations and normalizing this over time) ultrametric.ai/open-law
- Open lore (we are interested in the historical lore of startups, the rules of thumb and heuristics that might guide your agent): ultrametric.ai/open-lore
- Open fields (we are interested in how the schema for the data connects to the open modules, business logic, open documents and artifacts of each process eg how is a EIN defined?) ultrametric.ai/fields
- Open documentation (eg SAFE notes and missing! docs) ultrametric.ai/open-document…
We have looked at YC companies as well in terms of which ones do well agentically: ultrametric.ai/yc
We are interested in stacks and how people are changing their work: ultrametric.ai/stacks
We are interested in what is automatable and where the agentic ceiling is currently and how to push that up in all areas: ultrametric.ai/rankings/proc…
There is more for you to explore and a lot more coming in terms of how you will be able to navigate business in an optimal way in your chosen surface area; whether you are hanging in grokbot, dots, muse, claude, terminal, chat, or whatever is coming up in this exciting time.
We have a lot of data to collect, clean up and help agents navigate the vast landscape emerging functionality the entire startup ecosystem has to offer.
Come join us getting your vendor in, ranked or that access pattern you have wanted or getting the data more accurate in terms of which vendor is best for a process or steps. Or open a module of something that currently cost money that could be free.
Also come play in our discord for founders and join the chaos (we can solve each others issues including fund raising tips from me personally): discord.com/invite/3aHky836q…
It's time to lift all the boats in the harbor and I'm extremely excited as to what the community of hackers and painters will hack into the repo, what asks they will have and whether we can shift normalization of processes globally, gov agentic interfaces and have common repo for founders that grows over time. I expect a lot of experimentation along the way, bugs and upside...
I appreciate our backers having faith in us as well from @ycombinator with @garrytan to @immad doing a sizable check (congrats on $1bn rev homie) to @darbyw (congrats on Stripe buying @clerkyinc), @drewhouston form @Dropbox and also @pioneer_fund @tomerlondon cto @GustoHQ and hopefully this is 'boring' and complex enough for Charlie Songhurst :> @delingpole @Liquid2V @ivolo from @segment @ryanpowe @sonicaghi of @kong @DellyTamer+ other supporters (@michaelwfrank, Fundamental ventures and if I missed you sorry moving fast!).
[And don't worry, all vendor rankings are open scoring methods so no favors are being done if someone backed us and is a vendor they have to compete with how agentic and automatable they are just like anyone else.]
PS: we have bugs and many quality upgrades to make but thought better to go out early and fix as a community.
PPS: we have other stuff to launch we just thought this was a great place to start and have 200 PRs coming out pretty soon but I still want to know what is broken or doesnt make sense or inaccurate.
Special thanks to current and past team members for getting us here and all the experiments we did you know who you are.
Ben Sprecher retweeted
SCOOP: The State Dept has started asking career diplomats up for promotion if they voted for Trump in 16, 20, 24.
Federal law prohibits politically based hiring for career staff.
State tells @washingtonsun it's screening for "fidelity" and execution of the president's vision👇
Ben Sprecher retweeted
So basically, today PHP got two critical bugs that let anyone run code on your server without even logging in, which is bad, and then the details leaked before the patch was even out, which is worse, and it works on WordPress and MediaWiki and Magento and Joomla and Laravel, and Censys says like 1.5 million servers are running a version with no fix, so the NSA was like "take your system offline," and the FBI was like "take your system offline," and CISA and the UK and CERT-EU were all like "take your system offline," and then the Prime Minister of Japan, the actual Prime Minister, posted at 5:55 in the morning telling 48.8 million people to take their system offline, so everyone took their system offline, like the Dutch cybersecurity agency took its own website offline, and the Dutch parliament went offline, and the Port of Rotterdam went offline, and Finland and France and Denmark started pulling government sites, and Wikipedia went down, and Archive.org went down, and Asana and GOG and Something Awful went down, and the University of Vienna's webmail went down for some reason, and that's just PHP. Because GitLab also dropped a CVSS 10.0, which is the highest score they give out, for a bug that's been there since 2014, so that's every self-hosted GitLab for twelve years, and then LiteLLM 1.107.0 got hijacked and shipped a credential stealer and a remote access trojan in every single wheel and the Docker images, including latest, and you didn't even have to import it, just installing it was enough, and it's the second time this year, and then Railway paused every deployment on the entire platform because someone got into the environment variables, so now everyone has to rotate every secret they've ever stored there, and TiDB Cloud said someone was walking around inside with stolen credentials for basically a month and an extortion group says they took 10.6 terabytes, and Zeabur got breached for the second time in two months and is just shutting down, like the whole company, and Node.js patched remote code execution and a TLS bypass, and CISA says Plex is being actively exploited, and someone took over 300 Tor exit relays to steal SSH passwords, and SafePay ransomware has hit over 600 critical infrastructure orgs, and India's biggest bank filed a data loss incident, and Japan has been in an official state of emergency in cyberspace since October 9, which honestly explains the 5:55 AM post. Oh, and eMudhra, a certificate authority that's issued over 60 million certificates, whose entire job is proving websites are who they say they are, was supposed to be using private keys that are random numbers over a hundred digits long, and one of them was 4. Just 4. And the others were like 66 and 237. Anyway, it's Wednesday.