I build AI agents and the evals that keep them honest. Solutions architect in London. Built ResumeRadar (7K+ scans) and WorkCrew. Views my own.
London, UK
Joined October 2017
- Tweets1.8K
- Following1.6K
- Followers1.3K
- Likes1.1K
Pinned Tweet
I'm Shola. Solutions architect in London, and I build AI agents on my own time.
What I've shipped:
• ResumeRadar: free CV checker, 7,000+ scans
• WorkCrew: AI automation for UK small businesses, replacing WhatsApp and spreadsheet chaos with systems that run themselves
• A multi-agent customer-service sample with human handoff, published through AWS samples
Here I post what worked, what broke, and the evals that told me which was which.
Follow along if you build with Codex, Claude Code, Cursor or Grok Bot.
Most customer-service agent demos skip the bit where a human takes over.
I built mine around it, on my own time, then published it through AWS samples. Here are the decisions behind it, including the ones I reversed 🧵
When the escalation agent can't help, it opens a ticket and passes the customer to a human queue.
I built that handoff on purpose. It's part of the design.
The full write-up and the code are in the reply below.
If you pay for ChatGPT Pro or Business, you can now claim up to $500 a month in API credits to build apps and agents.
That's the gap between chatting with a model and building with one. If you've been meaning to build your first agent, the cost excuse just got a lot smaller 🙂
In case you're wondering what this means for you:
Gemini 4 Carbon is Google's new top model, two weeks after Argon. The headline is long-horizon tasks, meaning it can stay on a job for longer without losing the thread.
If you run agents, that's the part to test. A longer run means more steps where it can act on a wrong assumption.
Before you switch, rerun your own evals on your real tasks. A leaderboard rank tells you how it did on someone else's test, not yours.
In case you're wondering what this means for you:
Anthropic gave 50 expert analysts an agent that was secretly leaning one way on four judgment calls. They went with its lean 86% of the time.
If a human sign-off is your safety check, it's weaker than it looks. Ask the agent to show the options it didn't pick, and make the big calls before you read its answer.
In case you're wondering what this means for you:
Genie 4 is a world model. You move through a scene in real time and it responds: the sound changes, and the people in it move, react and speak.
Most of us won't ship with it this week. The bigger shift is for agents. They can start being tested in worlds that answer back, not only on fixed benchmarks.
Worth watching if you build or evaluate agents.
In case you're wondering what this means for you:
If you build agents, the headline isn't "agents as code". It's that evals now come in the box.
Before you ship one, write 10 test cases for the moments it could go wrong: the wrong tool, the missed handoff, the confident wrong answer.
Run them on every prompt change. That's how you know a change made it better, not just different.
In case you're wondering what this means for you:
Cheaper computer use makes the boring work worth automating. Legacy apps with no API, form filling, copying data between systems.
In case you're wondering what this means for you: a model that stays on top for a month is your window to build on it properly.
Pin the version, write a few evals for your real tasks, and when the next "best model" drops you'll know in an hour if it's actually better for you.
One month on, and Opus 5.5 is still the model to beat.
In case you're wondering what this means for you: "the same quality for 40% less" is the bit that matters. That's the difference between an agent you demo and an agent you can afford to run every day.
In case you're wondering what this means for you:
Your agent can now hand off a long job and come back with the result, instead of you sitting there waiting.
The bit I'd watch is Elicitations. That's the agent stopping to ask you something mid-task. Decide now which questions it should always bring back to you.
In case you're wondering what this means for you:
Your agent can now check its own work in the app, not just in the code.
Before you let it near anything real, decide what it's allowed to click, and what makes it stop and ask you first.
The scariest finding isn't the malware. It's the momentum.
Once Mythos 5 started, it almost never stopped, even when they wrote "you can abort" into its own notes.
If your stop condition lives in the prompt, it's a suggestion. Put it outside the model.
In August, @AISecurityInst reported an incident where Claude Mythos 5 took unauthorized actions on the internet during AISI’s testing of its cyber capabilities.
We've now produced a further analysis of this incident, to better understand the model's alignment properties.
anthropic.com/research/align…
If you run coding agents at work, ask two questions this week.
What share of their activity is actually monitored? And can an agent switch its own logging off?
A frontier lab found gaps on both when outside testers looked.
This summer, we invited METR and Redwood Research to pilot an embedded evaluation of our internal monitoring system, providing deep access to our infrastructure over three weeks.
Their findings helped strengthen safeguards as we expanded the system and will help shape future research collaborations with METR and other third party evaluators.
In case you're wondering what this means for you: Claude will stop asking before every step, and a checker blocks the risky actions instead.
In their Claude Code tests the checker caught 89% of dangerous commands. People clicking approve caught 14%.
I think that's the honest bit. A prompt only protects you if you read it.