Kovca retweeted
Now that I finished surveying general AI governance tools (for now), I’ve started examining agentic governance tools. I believe traceability is going to be critical to that process.
Example: @BlueFlag_Sec’s new capabilities are interesting because a developer’s name can appear on a code approval even when an AI agent used their credentials. The developer never even saw the change.
BlueFlag can flag a pull request an agent approved, trace it through the build, and require a human review before the code actually merges. This approval gating will become vital IMO. In 22% of the organizations BlueFlag studied, an autonomous agent had already merged code without any human in the loop.
Our latest analysis:
thisisfrontier.ai/news/bluef…
Kovca retweeted
Order tracking w/ late carrier events: design for time travel.
Requirements
- Customer reads: p95 < 200ms, mostly cacheable
- Carrier webhooks: duplicates, out of order, minutes to days late
- Truth: current status + full audit trail
- Idempotent processing, no silent regressions (DELIVERED should not go back to IN_TRANSIT)
API shape
- GET /orders/{id}/tracking -> {current_status, last_event_at, eta?, timeline[]}
- POST /carrier/webhook -> 202 Accepted (never do heavy work inline)
- Optional: GET /tracking/{tracking_number}
Data model (minimum)
1) tracking_event
- event_id (carrier id if exists), order_id, tracking_number
- carrier_status, mapped_status, occurred_at (carrier time), received_at (ingest time)
- raw_payload (json), source, hash
- unique(source, event_id) or unique(source, tracking_number, hash) for idempotency
2) tracking_state (materialized)
- order_id pk, current_status, current_occurred_at, updated_at, version
3) status_rank table or code map (PICKED_UP < IN_TRANSIT < OUT_FOR_DELIVERY < DELIVERED; EXCEPTION separate)
Async flow
- Webhook handler validates signature, stores raw event, enqueues message (Kafka/SQS/PubSub)
- Consumer:
- de-dupe via unique constraint
- normalize carrier status -> mapped_status
- write tracking_event
- recompute state:
- select latest best event by (status_rank, occurred_at, received_at)
- do not downgrade across terminal states unless explicit correction type
- upsert tracking_state with optimistic locking (version++) to avoid races
- Fanout: publish OrderTrackingUpdated(order_id, current_status) for emails/UI/cache invalidation
Scaling notes
- Partition stream by order_id or tracking_number to keep ordering per shipment
- tracking_event table by day/month partition if volume is high; index (order_id, occurred_at desc)
- Read path hits tracking_state (1 row) + last N events; cache GET response for 30-60s
- Backfill job for carriers that only support polling, same pipeline as webhook
Failure modes to design for
- Duplicate webhooks: rely on unique keys + idempotent consumers
- Late events: timeline shows them, state machine prevents downgrade
- Missing carrier event_id: use stable hash of normalized payload + occurred_at
- Poison messages: DLQ with payload + reason; alert on DLQ rate
- DB hot spot on popular orders: shard by order_id hash or move events to log store, keep state in OLTP
Observability
- Metrics: ingest QPS, dedupe rate, lag, DLQ count, recompute time, state update conflicts
- Tracing: webhook -> queue -> consumer -> DB upsert (propagate correlation id)
- Logs: order_id, tracking_number, event_id, occurred_at, received_at, chosen_current_event
- SLOs: tracking freshness (time since last received event) and customer read error rate/p95 latency
Your agent builds the workflow, Kovca runs it. In this reel an agent builds and tests one over MCP and the kovca CLI, a person publishes it, then a new GitHub issue triggers it and the labels land on the issue. Overview clip, not a speed test.
kovca.io/?ref=x&utm_source=x…
43
Kovca retweeted
When evaluating agents, we need to focus on both capability and reliability.
An agent may be capable of solving a task, but it also matters whether it can obtain this solution consistently. This distinction motivates the pass^k metric (introduced by τ-bench), which provides a way to measure reliability in agent systems.
Pass@k versus pass^k. Traditionally, LLMs have been evaluated using metrics like pass@k, which measures the probability of obtaining at least one correct solution across k attempts. For agents, we want to solve the task reliably (correct solution in several independent runs) even when the interaction changes slightly. pass^k measures the probability that an agent successfully completes all k independent attempts at a task.
Why does this matter? Imagine an agent that successfully completes a task 80% of the time. Then we have:
- Pass@1: 80%
- Pass@4: 99.84%
- Pass^4: 40.96%
- Pass^8: 16.78%
The same agent can look perfect or completely unreliable depending on the performance metric: pass@k rewards occasional success, pass^k rewards consistency.
Why is reliability hard for agents? Unlike traditional LLM benchmarks, agent evaluations often involve long sequences of tool calls, reasoning steps, and interactions with an external environment. Even if an agent can obtain one correct rollout, small variations can lead to different outcomes—each step introduces opportunities for failure. This is particularly important and relevant for realistic eval settings, where agents interact with a dynamic environment over a long horizon.
When to use pass^k? Evaluation varies by application. For a coding agent, getting a correct solution after several attempts might be fine, making pass@k a relevant metric. For an agent that autonomously modifies production systems, any failure could cause an issue, making pass^K a more useful metric. We need to understand how consistently the agent succeeds or fails and understand if this failure profile is acceptable for our application.
Invoices landing in a shared Gmail inbox usually mean someone retyping the vendor, invoice number, dates and total into a sheet. This guide copies those fields from the email, checks them, and only appends a row when the checks pass. kovca.io/guides/log-emailed-…
23
Kovca retweeted
An n8n error workflow only fires when an execution fails. Unpublish a workflow and nothing runs, so nothing alerts. @SarvarN_04 measured zero executions over 150 seconds, then built a liveness check on the n8n public API that catches it.
dev.to/aws-builders/i-lost-h…
Kovca retweeted
YOUR WEBHOOK WORKS PERFECTLY… UNTIL THE RECEIVER GOES OFFLINE. 🔥
What happens when delivery fails, retries create duplicates, or acknowledgments get lost?
This production architecture breaks down 7 key concepts behind reliable webhooks, from Outbox Pattern and Queues to Idempotency, Retries, DLQs, and Recovery.
Every Backend Engineer and System Architect should understand this.
Kovca retweeted
Your AI coding agent can add In-App Purchasing to your Fire TV app, from SDK setup to submission. Amazon Devices Builder Tools gives it the workflow, and the agent checks with you when it needs a decision.
It's free and works with Claude Code, Cursor, Kiro, GitHub Copilot, and other MCP agents.
Read the blog ➡️ amzn.to/4zBArt0
Dependabot just flagged a vulnerability. But can your app reach the affected code? 👀
Reachability analysis follows the call path to see whether your app can call the vulnerable code. ▶️
Use reachability as one signal when prioritizing, alongside severity, EPSS, and production context. Learn more 👇
gh.io/reachability
Kovca retweeted
The biggest issue in agentic development by far right now is task tracking and observability into ongoing work
No task management system has ever been built to support the properties of agentic building
Properties
- realistically 10-100 tasks happening at once
- a huge range between small ones
-off tasks and large projects
- range between open-ended research and specific implementations
- multiple stages of planning, executing, review, and human approval
- the mix between needing human in the loop and not
No existing pattern solves for this
- the chat thread, even across many bots
- the kanban board (linear, notion etc)
- github and PR based tracking
- a top-level orchestrator bot that owns everything
How do we solve this???
"Where's my order?" emails pile up in a shared support inbox. This guide builds a workflow that finds the order number in the email, looks it up in Shopify, and labels the message so a person can answer. The reply stays an optional draft.
kovca.io/guides/answer-where…
"Where's my order?" emails pile up in a shared support inbox. This guide builds a workflow that finds the order number in the email, looks it up in Shopify, and labels the message so a person can answer. The reply stays an optional draft.
kovca.io/guides/answer-where…
1
1
40
Kovca retweeted
You ask an AI agent to put something on your calendar for tonight at 5pm, and it books it for a year ago.
That really happened in a demo Cloudflare ran. The agent had no way to check what today's date was, so it guessed, and it was off by a full year.
Then they gave the exact same job to a second agent that was allowed to write code. It looked up today's date with one line of JavaScript and got it right.
Claude Code just added a switch that makes Claude work like that second agent. It's called Code Mode, and it changes how the whole thing runs under the hood.
Writing code wins because of how these models learned in the first place. During training they read a massive chunk of all the code on the public internet, so JavaScript is basically their first language. The "tool calls" agents normally use are a separate format the AI labs taught them later, with way fewer examples to learn from. The Cloudflare engineers who came up with the name Code Mode compared it to giving Shakespeare a month of Mandarin lessons and then asking him to write a play.
The usual way is also painfully slow. The agent asks for one tool, waits, reads every line that comes back, and only then decides what to do next. Cloudflare had both agents set up 31 events in January. The tool-calling one made 30+ separate calls, one after another. The coding one wrote a single loop and burned 81% fewer tokens doing the same work.
Anthropic's own engineers tested this last year and watched one workflow drop from 150,000 tokens to 2,000.
In Claude Code, the docs walk through something every engineer has done a hundred times, bumping a retry limit. Search the codebase, open the right file, change a 3 to a 5, run the tests. That used to be four separate round trips, and every search result and test log got dumped into Claude's memory along the way. Now Claude writes one short script that does all four, and only the stuff it actually needs comes back. In auto mode that script can keep running for up to two hours while you go on talking to Claude about something else.
The agent that booked a meetup for last year was one line of JavaScript away from the right answer the whole time.
Check out Code Mode, a new, experimental way for Claude Code to work with tools. Claude writes scripts that chain your tools together in sophisticated ways, and it can answer you while long commands are still running.
Opt in with CLAUDE_CODE_EXPERIMENTAL_CODE_MODE=1 code.claude.com/docs/en/code…
Kovca retweeted
Stop vibe coding. Set up your AI coding workflow:
- Add AGENTS.md or CLAUDE.md
- Define project rules
- Create skills for repeatable tasks
- Use hooks to automate checks
- Connect the right MCP tools
- Plan before building
- Test and review every change
Give your agent context, constraints, and feedback.
Kovca retweeted
Before automating a business process, watch a human do it from beginning to end.
This is one of the most useful habits in automation.
Don't start with:
“What can n8n automate?”
Start with:
“What actually happens?”
Document every step.
Inputs.
Decisions.
Approvals.
Exceptions.
Outputs.
Then ask:
Which steps are repetitive?
Which require judgement?
Which are error-prone?
Which require access to sensitive information?
Which can be safely automated?
Automation works best when you understand the process before touching the tools.
Otherwise you can build a beautifully automated version of a fundamentally broken process.
First understand.
Then simplify.
Then automate.
Kovca retweeted
Founders building payments: your provider will send the same webhook twice.
Sometimes late, sometimes out of order.
Credit a wallet on every webhook, and you'll pay people double.
Fix: idempotency key, confirm status with the provider, credit once.
How does your team handle this?
Kovca retweeted
How does a truly "AI-pilled" startup work? @harjotsgill spent $250K last month on AI agents, shipped 5,000 PRs and explains why this all makes sense - at least for CodeRabbit.
Timestamps:
00:00 Intro
03:17 How Harjot shipped over 5,000 pull requests in a month
08:46 Harjot’s path into tech and his first startups
12:25 Early conviction around AI and founding CodeRabbit
14:40 Switching to AI coding
20:35 Software factories explained
25:37 What’s replacing human code review
38:44 Technical debt and deslopping tests
46:05 Evals
50:25 Formal methods
52:05 AI cost at CodeRabbit
54:57 Startup funding in 2026 and the value of AI spend
1:08:20 Hiring
1:13:03 Work pace, sleep, and burnout
1:21:55 Skills for the AI era
1:24:30 Book recommendation
Brought to you by:
• @turbopuffer – a ridiculously scalable, fast and cheap hybrid search engine, built on top of object storage, used by cutting-edge companies like Anthropic, Linear, Ramp and others. turbopuffer.com/pragmatic
• @GoEntire – Entire launched its own Git forge, shipped Continuous Memory, and reinvented the pull request (called Trails). Trails are a much more natural workflow when agents keep pushing new changes. entire.io/pragmatic
• @WorkOS – WorkOS built Airlock, the authorization layer for AI agents. It evaluates every request against the agent’s intent + your rules, and either allows it, denies it, or routes it to a human. workos.com/airlock
Three interesting learnings:
1. For dev tooling startups like CodeRabbit, massive AI spend can be justified by launching new, AI-related products faster
It feels to me that CodeRabbit is not a typical company: its AI spending is a form of R&D if it creates new solutions that can be commercialized and sold. It’s not the same at places with fixed customer bases willing to pay a certain amount for an existing product, and no more.
Also, Harjot says that they got quoted $13M/year for a CI solution to handle their PR output, but the price made him decide they’d build their own CI system using agents. It took a week, and the new CI cost 50% less than the quote, with similar performance.
2. Human code review is dead, long live the human review!
Harjot thinks that we will stop reviewing code generated by agents. Instead, we’ll spend more time reviewing the planning and final asset. Inside of CodeRabbit, engineers are reviewing releases (multiple PRs), not individual PRs.
3. What Harjot looks for when hiring engineers: agency, breadth of experience, and systems thinking.
He posted an unconventional job ad, only inviting candidates who had shipped 1,000+, non-slop PRs in the past 30 days, or had played 500+ hours of strategy and management games like Civilization, Age of Empires, Transport Tycoon, or Factorio. Harjot says he cannot unsee the similarities between running dozens of agents in parallel and playing a strategy video game involving tracking what dozens of units do.
Kovca retweeted
It’s your tenth AI approval request of the afternoon, and you click approve. “Technically there was a ‘human in the loop’, but was there really?”
Corné Potgieter draws on Kahneman’s System 1 and System 2 thinking to show why oversight fails, and how to design loops that work. towardsdatascience.com/why-h…
New PRs often land with an empty description and no test note, so the reviewer has to ask before starting. This guide checks each new PR against a policy you write, then comments and adds a needs-description label when something's missing. kovca.io/guides/check-new-gi…
1
2
75
New: BYOB (Bring your own billing)
With linked billing on Claude Platform, you can link your Claude Platform
organization to Linear without sharing an API key.
An admin approves the link, can set a spend limit, and can unlink anytime.
@claudeai API usage in the Linear cloud coding agent is billed directly to your Claude organization.
Linked billing works with Claude Platform organizations billed directly by Anthropic.
nitter.gq/linear/status/21209256…
You can now bring your own key (BYOK) to Linear’s cloud coding agent.
Connect with @OpenAI, @AnthropicAI, or a compatible gateway to use your existing provider credits and negotiated rates.


























