Nick

Artificial is one of the most important movies of our generation I’ve never cried this much while watching a film

github.com/wanderingspirit03… just try it and see the magic

Hey, I'm Nick, worked in neolab, living in London, UK building for ai agents with a couple of friends from lovable/revolut would love to meet more people building agents! Coffee on me👀

The bit about Browser Use's v3 scoring better on benchmarks but feeling worse in user studies is revealing. You can keep improving the score while getting worse at the work people actually use your agent for. I think when we're changing an agent, we should be able to take the customer tasks that change touches and see what each version does. Maybe the new one fixes a bunch of failures but breaks something people rely on every day. I'd want to know that before calling it an improvement.

Replaying a production trace against a new version of an agent works until the first step where the new version does something different. Usually that's step two or three. Here's the shape of it, and then the three things you actually need to compare two versions. Say you have a coworker agent in Slack. Someone in finance writes "put together the revenue section for Thursday's board deck". The agent queries the warehouse, pulls the billing export, reconciles the two, builds the spreadsheet, writes the slides, posts a draft in the channel, takes comments, revises, and files the follow ups as tickets. Two days, thirty odd tool calls, a few rounds with the finance lead. It's been in prod for months. Someone changes the prompt so the agent reuses queries it already ran this month instead of pulling everything again. Faster and cheaper, which was the whole point. The eval set is 50 graded tasks, scored on whether the numbers in the deck match the warehouse and whether the slides read well. All 50 pass, so it ships. Two weeks later the CFO asks why the board saw a revenue number 6% above the billing export. The new version skipped the billing reconciliation whenever the warehouse looked consistent with last month, and a refund batch hadn't landed in the warehouse yet. The slides were clean and the finance lead's comments were about the churn chart. The eval scored the deck. Nobody scored the path. So you try the obvious fix. Rerun last month's tasks against the new version with the recorded tool responses. It breaks at the reconciliation step, because the recording has the billing export from a call the new agent never makes. From there the state is wrong, every later response belongs to a run that didn't happen, and the finance lead's "looks good, add churn" was a reaction to a draft this version didn't produce. The part after the fork is the only part you care about, and it's the part you can't replay. What you need instead 1. Sessions picked by what the change touches. A change to how the agent reuses work touches every step where it decides whether to check something again, so you pick the sessions where it checked before: reconciliations, lookups against a second system, any number that ends up in front of a person. A model swap touches more. A fixed golden set answers a different question. 2. A way to serve the calls the new version makes, and the review rounds it never triggers. Frozen responses where the step matches the recording. A sandbox, a simulated tool, or a simulated reviewer where it diverges. And the report has to say which one produced each result, because a simulated answer is a guess, not evidence. 3. A diff of actions, not scores. "Reconciliation skipped in 31 of 200 tasks" is a hold. "0.91 vs 0.92" tells you nothing about what the agent did. We did this by hand for one team recently and found a failure they didn't know about. Now building it as an independent review that runs before a change ships, for long running agents, the ones that need thirty tool calls and a few rounds with a person to finish a job. Coworker agents, assistants, recruiting, support, ops. Which action change would make you hold a release even if the output looked fine?

who's actually reviewing prompt and model changes to production agents before they ship, and what do you look at? a few traces, a golden set, or nothing and you hear from a customer?

Been building something..

Was the best thing that happened to me in a while. Insanely grateful for the opportunity to work with Eddie and Henry!

My dad started to make music with @suno He’s 53 He asked me to teach him how to use Claude code and other coding agents because he was watching a lot of YT videos I feel like he’ll probably figure it out on his own through @perplexity_ai What a year to be alive

Rick Rubin was right at the end Read «The Creative Act: A Way of being» Even if you’re insanely technical, this will make you create insane products

My friend told me that @KingsCollegeLon now has a vibe coding course with Claude code, he’s doing math Wild…

Feels like I’m surfing on a huge wave Wanna try a bigger one tho

One of my friends is gonna be interviewing at @joinef as an investor and she was asking me a bunch of questions about my life while I was building stuff I now feel better about my past

Having both Claude Code with Opus where I brainstorm ideas and @FactoryAI with Gpt 5.2 Codex seems like an agi moment for me

life is so random man got a business degree thinking I’d end up doing the same boring stuff everyone else does somehow landed at a $1B startup. founders are literally my age. wild. thought ok this is cool but I wanna build my own thing eventually then the friend who got me this job? dude leaves. starts his own company. asks me to be a founding member. with a bunch of my other friends too. now we might actually be onto something crazy you can’t make this stuff up lol

You think life can’t be crazier than what you already have But then something happens