Well then. I guess I have to read this for real, haven’t I?

Speaking of metastable failures, one of the longest open tabs in my browser is this. Yet to read the post, but wonder how much of this methodology overlaps with what you did for Aurora. Only way to find out is to get around to reading the literature. arxiv.org/pdf/2606.00942

I found this unpublished blog post from 6y ago (look at the hand drawn diagram lol). I’d totally forgotten about this, but the key ideas here (embracing non-determinism as a first class citizen via “error kernel” centric design etc) work super well as guidelines for my agents.

The change in question came from a refactor. I have a separate “parity test” suite for any and all refactoring, and those have proven very good at preventing bad regressions, both for refactors involving code reorganization, code modifications, and code deletions.

There’s some overlap, but I’ve also been using new ideas for testing code not covered in the testing series, like getting agents to deliberate plant bugs at every layer and see how tests behave, mutation testing etc; which are new (to me) techniques I haven’t covered before.

Integration tests are great, but they don’t replace unit tests; they come later in the process, whereas bugs that can be caught by a good unit test should be. It allows for speedier iteration. As for the guidelines, I’ll try to make my Claude config public, including testing.

We’ll still need end to end tests, and frankly, we’re going to need all the forms of testing I’ve been talking about for a decade. But “unit tests are now slop” doesn’t have to be the foregone conclusion. Unit tests can be very useful even if a human will never again read them.

5 months and many experiments later, this was totally a case of “skill issue” on my part 😅 Like with code, agents don’t magically write great tests If your unit tests are slop (as mine were), that’s because you don’t have proper testing guidelines for your agents (as I didn’t)

Truly, she’s one of the vilest. Just astonishingly and unfailingly awful. She admits she gets a torrent of racist abuse, but engages in it herself. Despicable piece of shit. She and Indian_Bronson, both cut from the same cloth.

I don’t know if I’m the best person to answer this. Build new stuff. There’s still a world of software that’s yet to be built. Start with any tool you use, and see how you can make it better. Ideas are cheap to try out. The more you do, the better you get at picking better ideas.

lol, I refuse to pay for this app, so typos shall remain. It’s also how you know I’m rawdogging every tweet myself, rather than get AI to write it (which I learnt is a thing, esp with ultra long tweets some influencers and execs post)

Unfinished coding side projects are now way easier to complete than unfinished blog posts. I’m perfectly OK with AI writing my code, even the docs. But I flinch at the idea of AI writing my blog posts, which feels like an egregious violation of a fundamental ethnical contract.

Ideally the folks building software factories or harnesses should be evaluating how best to get agents to debug, because using Claude code out of the box isn’t going to cut it.

All super valid criticisms. I’m not a researcher or on a team at work that’s tasked with building better debugging primitives using evals, testing different mechanisms etc. I’m just an infrastructure engineer who has access to claude, and am chronicling how I use it to debug.

That should be covered by the data integrity axis. How data integrity is handled for stateful services which rely on replicated, partitioned data under different scenarios should answer most questions.

There are probably more patterns, but I need to go read more papers lol. Doing it will take time, hence the early 2027 (more like Feb or March) goal.

Lot of it holds up pretty well, actually. The main changes in the last ~7 years have been in: - move towards more flexible consistency - use of replicated logs for all manner of things - move away from strict consensus and synchronous replication - control plane design choices

I started this series on testing nearly a decade ago, but never published the final part copyconstruct.medium.com/tes… There’ve been so many great advances in the storage space in the ~7y I feel this is worth completing. Have a draft from years ago I hope I can wrap up in early 2027.

Ok, so now I guess I have to do this lol. Will try to get this done on Sunday, but if not, will have to wait until next weekend. Might even blog, explaining where some of these ideas came from (specific senior devs I worked with, conf talks of the past, oss projects, etc)