Building @cleric_io. An AI SRE for on-call engineers that diagnoses production issues before you have to.
San Francisco
Joined March 2009
- Tweets1.5K
- Following186
- Followers205
- Likes105
Jev is cheaper and faster but I think it’ll drive MORE LLM usage.
Sounds weird, but hear me out.
Your agent today is broadly doing two kinds of tasks:
1. Open ended tasks that require reasoning - e.g. reading logs, scanning code, interpreting a metric
2. Bounded decisions from a known set of actions e.g. deciding what to do next like check Kubernetes or Grafana
Asking an LLM to do #2 over and over again is horribly inefficient. You can do it, and before Jev, you likely HAD to, but it’s shooting an ant with a shotgun.
Mike ran an experiment to move the high-level next action choice in Cleric's investigation agent out of the LLM and into a Jev-based harness. We ran it offline on our evals and we got the same average recall, while running 25% faster and costing 98%(!!) less.
To be clear - this prototype is not close to GA but the results are telling.
The reason this is such a big deal is that for use cases like ours, we’re on a clock and a budget. The agent gets a limited amount of time to find everything it can given a problem in production.
Our instinct is to take those savings and deploy more LLM analysts so Cleric can gather even better evidence.
I think Model labs will 😍 Jev.
Post: cleric.ai/blog/jev-the-decid…
Short version: Opus 5.5 is good.
How good? We've been stuck on Opus 4.5 for our Claude-based agents for ~10 months. That's an ANCIENT model!
We tried, believe me. Every new Opus release, we ran our evals and the performance was jagged: better in some areas, worse in others. Upgrading meant accepting a regression somewhere.
The best we managed was a limited upgrade: Opus 4.6 for some analysts, 4.8 for others.
Opus 5.5 is the first release that beats 4.5 with no trade-offs.
On the average eval run: 5.5 has slightly better accuracy, is 20% cheaper and ~12% faster.
👏👏👏
I have a love/hate relationship with session limits. Frustrating... but they're also my cue to go take a walk and resist the temptation to continue with another one.
Shahram retweeted
"Everyone doesn't need to be using Opus 5 or Fable 5. It's like bringing a tank to a knife fight."
-Affaf (Head of Eng, LaunchGood) on matching the model to the job as AI costs climb
Full ep: youtu.be/aQbVZA-WHSA
Achievement unlocked... scrapers are coming for our blog posts.
p.s the scraper has good taste, it's a technical post on how we grade our agent without requiring human in the loop.
p.p.s. for technical posts like these, it'll be fun to get your coding agent to read it and review your existing agent code against it to propose improvements you can make.
cleric.ai/blog/measuring-an-…
HUGE. Verification closes the self learning loop.
Coding agents get fast feedback: tests pass/fail. Merged code gets none. The signal exists in production, but it arrives days later and nobody closes the loop.
We built a verifier that watches production after a change lands and answers: did it work?
cleric.ai/blog/verifying-fix…
Wife went to her first hackathon in SF yesterday.
My reaction 😊
We discuss if she should solo it or find a team member.
My helpful suggestion - find an engineer to team up with.
“I have Codex and Claude - why do I need an engineer???”
My reaction: 😊 😊 😊
She built a full game on her own and sent me this message before demo time.
SF gets a lot of heat for ONLY having tech billboards. Why? I like it that way.
Regular ads are awful - "You're too fat, buy GLP-1". "Your hair looks like shit, get this new hair gel" - on and on and on.
Bring on the ads insulting my build pipeline speed and the AI agent for X.
Context has always been THE problem, but every new agent you add is making it worse. I call it context dispersion.
Say you need an agent for X. The agent is easy to build but context is the hard part. You have two options:
- Build your own and own your context.
- Buy a vendor's black box and rent your context back from them.
My guess is that you're doing both... and your context is siloed everywhere.
That context is critical for agent accuracy. It's how an agent knows whether it's right or wrong about your systems, instead of guessing from its own generic model weights.
For instance: a sports streaming company we work with has a specific definition of "live," a game happening NOW. For another streaming company, "live" could mean a user is now playing a show. Same word, completely different system implication.
Pre-AI, context (tribal knowledge) had a similar split. You outsourced it to consultants or in-housed it by staffing a function, but it was always on borrowed time. Consultants came in, built deep context working with you, then walked out with your knowledge when the project ended. In-housing wasn't much better: employees leave, go on vacation, move teams. Most of us gave up on documentation and just turned to Slack search.
Agents are the first meaningful opportunity to codify our tribal knowledge in a way that's accessible to all, agents and humans.
Self-learning, verification, and engineer-provided memories all feed one goal: the best possible context on your production environment. We believe the right approach to agent interoperability is shared context. Cleric collects and organizes your context but it belongs to you, so you can expose it via MCP to your own agents or whatever vendor agents you have.
My prediction: operational memory will start vendor-specific, every vendor building their own. But it'll converge on a universal standard, so agents can read and write across all of them. Your SRE agent learns the auth service has a quirk on weekend deploys, and your coding agent and support agent both know it too. Context dispersion ends.
The memory layer is not just the asset, it's YOUR asset.
Making this layer accessible to all is step 1.
Wrote more about it here cleric.ai/blog/youre-renting…
yep, constructive criticism is rare. Converting non constructive ones to improvement feedback is a super power.
always fun hanging out with Demetrios @mlopscommunity.
Big topics we got into, lessons learned on:
- How to apply systems thinking to get the most of agents for your team
- What we got right and wrong in what would be important in building @cleric_io's SRE agent
New episode is up with Shahram Anver, co-founder & CEO of @cleric_io (built an AI SRE). A good chunk of it is him walking through how their thinking changed as the models got better, & which parts of building an agent turned out to matter versus which ones they over-invested in.
Shahram retweeted
New episode is up with Shahram Anver, co-founder & CEO of @cleric_io (built an AI SRE). A good chunk of it is him walking through how their thinking changed as the models got better, & which parts of building an agent turned out to matter versus which ones they over-invested in.
❤️
Mind boggling to me that I can make a thing faster and there's always people that ask "but why?" What kind of mentality is that? The pursuit of excellence does not need justification. Also, I find in so many cases, we can't know the impact of an improvement until we do it.
For example, one I've talked about before: Ghostty's high IO throughput has enabled terminal program (emulator and TUI) fuzzing at a speed thats incomparably fast to prior solutions. This has resulted in upstream patches to resolve issues in popular projects like btop, tmux, and more.
Speed enabled that anecdotally example that lifted the tides of adjacent communities that don't rely on Ghostty technology at all. I didn't predict this.
Make things better because they can be better and let the results naturally play out.
We have Jester Lavorre, a tiefling Cleric walking around @aiDotEngineer - she has a quest for you - meet her at the expo hall!
Aaaaand That’s a wrap for day 1 at @aiDotEngineer world fair in SF.
We’re launching an expanded @cleric_io at the conference that works within your SDLC instead of just waiting for an alert.
Come by the booth! We’re right next to AWS.
"Couldn't find anything" as a lazy out.
We require a specific reason to mark a hypotheses as unverifiable. e.g. "No logs" is unacceptable but "Splunk not configured" is ok. We use an auditor to flag vague reasons before a human sees the report.