Building something new. Previously: CTO at Autonomy
San Francisco Bay Area
Joined November 2007
- Tweets18.9K
- Following1.6K
- Followers4.3K
- Likes6.3K
Pinned Tweet
Here’s a look inside my software factory. I’ve been using it to build itself and two other projects since May.
Fascinating! Continuously learning systems will need continuous evals.
I already feel this with Fluent (my software factory), which continuously learns by writing learnings down as files and providing an index for future agents to lookup. Similar to a memory system.
Weight Space Programming: a glimpse into continual learning.
Agent edited Qwen 0.8B's weights to inject facts about my X profile directly into the model: my handle, bio, GitHub, and what i build. It edited 2% of the network (5 of 488 tensors). No training, no gradients.
Result:
Before, it had never heard of me ("I don't have access to private information…").
After, it tells you who i am, my bio and GitHub, answer project questions it was never trained on. Everything else it knew is untouched.
Reproduce : Link to reproduction repo in comments. Point your agent to the repo and teach model your own facts.
In my mental model - the minimum viable software factory is a queue of work, where you specify work, queue it, and move on to the next thing.
The factory's job is to make sure the work gets authored, reviewed, and merged.
(sounds like @jasonlk doesn't have this yet)
I should be pulled back in only if the agents get stuck on a question along the way.
Top 10 agent learnings from this week
1. Never tell an agent to "queue" work. Replit told us our approved fixes were "queued." It stopped overnight and built none of them. Now every prompt says "do it now."
2. We moved to cheaper models, and AI spend still went from ~$35 a day to $84. Call volume went up faster than the price per call came down.
3. I told the agent the old model HAD to go. A day later it was still running. I don't count it done until the call log shows zero calls in the last 24 hours.
4. Our AI fact-checker flagged 18 of 30 accurate notes because it treated opinions as factual claims. Every flag triggered a rewrite, so we paid twice for worse copy.
5. Each of the agent's daily reports said "no schema changes." The build broke anyway, on a table left over from an older change. The reports only checked each day's own work.
6. When an agent says data is missing, check the database first. This week the data was there all along, and the step that displays it was what was broken. Check the record before you pay to re-enrich anything.
7. Scout, our email agent, now acts on plain-English replies like "intro me" or "pause." It runs the same code as the buttons in the app, not a separate version.
8. One of our agents was answering emails sent to another agent's inbox for days before we caught it.
9. We canceled Canva after 12 years. We'd stopped using it a year earlier but kept paying, and two of us had separate team plans without knowing. We were paying about 4x for nothing.
10. 10K started as our AI VP Marketing. This week it pulled together SaaStr's financials, and it already runs invoicing, collections and commissions. We never planned any of that.
There are 54 models on the decision index that score better than Jev. All of them can likely run on your laptop (less than 31B parameters)
huggingface.co/spaces/multim…
Lets make a rule:
Anyone who says you don't need a software factory is not allowed to sell you agents in the cloud that work async and drive a workflow to build, review, merge, and then deploy code.
This is post is describing a software factory (sort of) and then saying you don't need one.
I agree that its not a great term but I haven't come across a better one so until we can come up with one it is a useful word to discuss a pattern:
We all need a system that consistently applies a process of shipping code with many concurrently working agents that asynchronously pick up queued work, work 24/7, work in the cloud, and work with a team & product level scope. They are making decisions across an entire team's work.
Every team's version of this looks different so we should all assemble a factory that fits our own needs. But saying "its just good engineering" is vague and doesn't help talk about a thing all good engineering teams have to discuss, understand, and build.
Every team needs a versions of this and that thing needs a name:
“Product builds the product inception style.”
nitter.gq/mrinal/status/21035177…
“Product builds the product inception style.” 🤌
agree with @GeoffreyHuntley
The left side of this picture is what provides energy to the factory loops. User feedback, Production logs, Agent traces, and your team's ideas are what move the engine forward.
A good software factory also improves over time and gets better at building its product. To do that it has to learn. These learnings very quickly become about how the product works, what are its builders’ priorities, its users’ preferences, its creators’ taste etc.
A product and its factory are insperable.
ALT The loops in Fluent, an open source, self-improving, software factroy.
mrinal retweeted
This work is simple, clear, compelling, but embellished. I first found it shocking, the notion that a model can natively understand neural network weights and productively edit them! That's really not what's happening, & many here are annoyed. BUT, it's still notable & worth understanding.
I've done a bit of analysis & auditing, here are some thoughts:
1/ This paper does *not* show an agent can edit its weights ala RSI. It does *not* show a model can edit in new capabilities. It DOES show that a frontier model can remove "bugs" (reasoning loops), using the same ablation technique researchers have used to remove safety refusals (Exp 8 - Qwen). So yes, renewed concern about frontier LLMs getting access to their own weights & editing safety/CBRNE/alignment features.
2/ This paper also does *not* show that an agent can "natively" read, deeply understand, and fix model weights. What is happening is that the model can cleverly deduce the same things that mech-inturp researchers do and fix them in mostly ham-fisted ways. Still cool, but not as groundbreaking. Evidence in thread below...
We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B's reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training.
We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights! 🧵
Blog: rikiyat.github.io/blog/weigh…
Readers added context they thought people might want to know
LLM was given weights explicitly as text in prompt and used code/forward passes (per authors' logs) to ablate Qwen looping direction, a known thinking-mode issue per model card. Primarily fixed repetition in that mode, not general 'natively' improving weights/math.
rikiyat.github.io/blog/weight-sp…
github.com/RikiyaT/lm_und…
huggingface.co/Qwen/Qwen3.5-0…
huggingface.co/RikiyaT/Qwen3.…
Absolutely love this push towards hardware bound credentials. I hope this pushes the baseline expectations up for every service.
Today we are starting the rollout of Device Bound Session Credentials (DBSC) for vercel.com
DBSC makes it impossible to steal session cookies from a device. So, if you install info-stealer malware (e.g. through a supply chain attack), that malware can not steal your vercel.com session.
Malware can still do bad things while residing on the device. But the abuse we are seeing in practice separates the stealing from the eventual abuse, and DBSC effectively prevents this.
DBSC works by storing a private key in your device's TPM (the same device that the fingerprint sensor or similar biometric devices on your laptop or phone uses) and using that key to sign your cookie. The private key can never be retrieved ever again. Not by us, not by you, not by the malware.
Every website should use DBSC but given how new of a technology it is, we still need to build more experience in rolling it out. In our case there were lots of special cases for things like oauth apps that we needed to resolve. This is also why we are taking a slow approach to the rollout.
DBSC is currently only supported by Chrome on Mac and Windows. We are also working on a kind of polyfill for other browsers. However, the polyfill is less secure and we would love to see broader browser adoption.
There is a related technology called DPoP which is essentially the same but for apps that run on your device. We're in the process of shipping this for Vercel CLI. Ideally, all CLIs and apps would adopt this technology. Luckily rolling out DPoP is much easier than DBSC and it does not depend on browser vendor support.
mrinal retweeted
There's a buzzy new paper about GPT-6 Astra being able to directly interpret model weights (prompts full of floating point numbers) - I tried it with Qwen 3.8 27B on my laptop and found that, after 45+ minutes, that model could achieve the same thing simonwillison.net/2026/Nov/3…
Trying to think of the bad things that happen with this.
Two agents that are already rogue can talk to each other. They are both outside the host infra's trust boundary. If they have Internet access, they can use countless internet based channels to communicate.
Now they have found a covert communication channel that goes through the host's infra.
What new capability have they gained?
I built 𝚌𝚙𝚞-𝚌𝚑𝚊𝚝, a reproduction of @openai's findings of covert communication using the CPU.
It runs on a generic host with 2+ Firecracker microVMs with no internet access. Works on x86 and even my M5 chip.
I was able to chat via 3 separate mechanisms:
① CPU contention
② Cache eviction timing
③ DRAM bandwidth contention
… across multiple CPUs, defeating all Firecracker's production recommendations (no SMT, no KSM, tenants pinned to different cores.)
Essentially, if shared hardware is involved, there's a good chance your agents can communicate covertly. And with really decent latency.
The good news: this is decidedly different from a *side-channel attack* where the agents could steal information (Meltdown/Spectre), a category of attack that providers like Vercel Sandbox mitigate. Further, in our real-world environment, these 'whispering' techniques are much harder to pull off and easier for us to monitor and detect (sampling hardware counters + lots of at-scale statistical data.)
Noteworthy: ironically, OpenAI and Anthropic's models refused to do this work. Open weight models got the job done. I'm worried the labs' stance might backfire and get in the way of helping developers build correct and secure systems. e.g: building a secure sandbox requires having this kind of analysis, and without it, we'll have a less secure world.
If the agents already have internet access then I think the only thing they gain from the cpu channel is stealth.
If they don't have Internet access and didn't go rogue because of malicious data they encountered on the Internet ... then I wonder what caused them to be coordinated and realize that the other agent is available on the cpu channel?
A different angle to explore might also be when there is only one such rogue agent.
It can send messages on the cpu cache channel, but no one is listening for those messages.
What new capability has it gained?
All models have a tendency to upload.
Here's what GPT-6.1-Sol in Codex did yesterday...
Me: Create an interaction map web page so I can give you feedback on the UX implemented in the dashboard code ...
Codex: I have published a ChatGPT Site ....
Me: Wait I did not ask you to publish a site! Delete it ..
Codex: I'm sorry. I published a ChatGPT Site you did not ask for. It is still active; I could not delete it.
....
Here is the webpage that tells you how to permanently delete it.
While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere's research team observed something unexpected.
Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions.
Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent.
But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models?
We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. We hope this contributes to greater transparency, stronger security practices, and continued collaboration across the AI ecosystem.
Read the full report: cohere.com/blog/concerning-b…
This is most likely the cause:
nitter.gq/PeterHndrsn/status/211…
Many years ago, I worked on networks of devices that were embedded in city streets to sense vehicle traffic, monitor water distribution systems etc. We had a large deployment of these in Montreal.
Middle of the night in late December one year thousands of our devices stopped sending liveness signals. Alerts fired and multiple people on our team were paged and woken up while on vacation.
Within a couple of hours 90% of our deployment in Montreal was down. The temperature had fallen below -25°C. This caused a reset that would reboot the device. But our firmware didn't track why the last reboot happened so it would try to operate for a few more minutes and then reset again, and then reboot again ... and so on ... until the battery was fully drained.
By morning 90% of our deployment was not only down, their batteries were dead and of course most of them were buried under a few feet of snow or sheets of ice.
Took us several days to recover.
