ai engineer. i build with AI and care about taste and agency. building https://xitter.cf/t.co/dcwVoJZhRa | 7x hackathons
pune
Joined February 2017
- Tweets253
- Following493
- Followers148
- Likes749
part 5 of the evals series. evaluating AGENTS.
in 2024, @SierraPlatform made a test called τ-bench. a simulated customer talks to an agent. the agent uses tools to change orders and flights.
why am i talking about a 2024 paper? isn't it stale by now? idk about stale. but it introduced smth i still find useful today:
τ-bench added a new metric: pass^k
pass^k asks: does the agent succeed on ALL k runs of the same task?
this is different from pass@k, which asks does the agent succeed AT LEAST ONCE in k runs.
this is important becuase agents are non deterministic.
today, the τ-bench leaderboards show pass^1. that is ONE run per task.
the newest models get high scores on one run.
but one run tells you that the agent CAN do the task. it DOES NOT tell you that the agent does the task EVERY time.
for example: if your refund agent succeeds on 4 of 5 runs. that looks good.
now 3 customers ask for the same refund. the chance that all 3 refunds succeed is approximately 1 in 2.
did you have a fucking stroke reading that? yeah me too. let me try again:
your refund agent passes 4 of 5 runs. that is 80%.
now run the SAME refund task 3 times.
the chance that all 3 runs pass is 0.8 × 0.8 × 0.8 = about 50%.
so an agent that passes 80% of runs gets the same task right 3 times in a row only about half the time.
that is what pass^k measures.
that gap between “can do it” and “does it every time” is what pass^k measures.
i dont care about your average. i care about what each run is probanly gonna do.
TLDR: the two metrics
pass@k: the agent succeeds at least one time in k runs. use it when a human can retry or pick the best output. example: code generation.
pass^k: the agent succeeds on all k runs. use it when the user expects success every time. example: refunds, bookings.
how did τ-bench decide pass or fail?
it did NOT read what the agent said. it compared the database at the end with the correct database.
the agent says “refund done”? check the refund in the database.
also... do NOT grade the exact order of tool calls.
an agent can find a correct path that you did not expect. a fixed-order test fails that agent for correct work.
articles are also a way to hone my craft for me.
counter intuitively, writing clearly has become probably THE most important skill in a world of slop.
part 4 of the evals series. WRITING the evaluators.
part 3 was question + delete. whatever survived gets an evaluator.
this is step 3: simplify and optimize. the question now is WHICH kind of evaluator.
1. code first. always.
if code can check it, then let it: valid json, required fields present, right tool called, refund row actually exists in the db.
cheap, fast, and it will never let you down.
2. grade the OUTCOME, not the claim.
when your agent says “refund processed”. great. did it though?
don’t ask an llm to read the transcript. check the database.
@AnthropicAI and @langfuse both make this point: the transcript is what the agent SAYS, the environment is what it DID.
your eval harness should be able to independtly verify OUTCOMES.
3. llm judge ONLY for what actually needs reading.
things like: is the answer grounded in the retrieved docs? did it answer what was asked?
one judge per failure. also go for simple PASS/FAIL.
i've found that a rating like 0-5 gives the llm a middleground (eg. 3). FORCE it to pick sides.
@eugeneyan calls that the “God Evaluator”. don’t build god.
4. write the judge prompt yourself. do NOT OVER ENGINEER.
i like to include:
the context they need
ONE criterion, and what to ignore
examples from the traces you graded in part 3
reasoning first, verdict last
let it say “unknown” instead of guessing
5. think about this question:
your judge agrees with your labels 92% of the time. should you ship it?
if only 10% of your traces are failures, a judge that ALWAYS says pass agrees with you 90% of the time. and catches NOTHING 😭
thats why you should measure two numbers separately:
TPR: of the real failures, how many did it catch
TNR: of the good outputs, how many did it pass
and test on labels you did NOT use to write the prompt. otherwise you’re grading your own homework.
@HamelHusain and @sh_reya split labels into train/dev/test for exactly this.
6. a judge needs 100+ labels and someone maintaining it every week.
so before you build one ask: can code check this? can a prompt fix kill it?
most of the time, yes.
how are AI engg interviews different from normal sde interviews?
in a normal system design round you design for scale. how much traffic it takes, and what breaks under load.
AI rounds have all of that, plus the harness.
imo you should build it around how humans already work and find where you can speed up their loop.
for me a generic harness that does a bit of everything is my WORST ENEMY and introduces all sorts of failure modes in the process.
been listening to @hubermanlab for yearsss now its so cool to see him distill down the work of his life to 663 pages.
i can feel the value radiating out of this thing, just gotta absorb it properly now.
goated gift by my goated gf <3
part 3 of the evals series. choosing WHAT to evaluate
last time i listed everything you can measure on live traffic and during development.
you probably DONT need to run evals on MOST of it.
this is steps 1 and 2 from the first thread: question the requirement, then delete.
then group your notes into failure types and count them
now you have a list of what breaks, sorted by how often. thats your metrics shortlist, and every item on it actually happened
most of that list DOES NOT need an eval.
if a prompt change fixes it for good, fix it and move on.
if the score wouldnt change a decision like blocking a deploy, LET IT GO!
whatever survives gets an evaluator.
my favourite example is from openai's cookbook. a receipt auditing system got merchant names wrong 85% of the time
they didnt fix it, because the shop name never changed the decision. it didnt matter.
a wrong output ONLY matters if it changes what happens next
developers.openai.com/cookbo…
part 2 of the evals series.
ACCELERATE CYCLE TIME.
think of it like loop engineering but for evals.
1. offline loop: the golden dataset (GDS)
during development, you build a set of inputs for the things your system HAS to get right.
core features, past bugs, queries your clients need to never break.
EVERY stakeholder (technical and nontechnical) has to sign off on this dataset.
it should be cheap enough to put in CI and run against every prompt change or model swap.
@AnthropicAI's agent evals post splits the GDS in two
capability evals are supposed to be hard for the agent, and they give you a hill climb
regression evals sit near 100% and catch backsliding
once a capability eval saturates it graduates into the regression set. cool right?
article: anthropic.com/research/demys…
2. online loop, after you ship
two things you can do here:
a) evaluate prod traces using llm-as-a-judge
b) compare prod traces with GDS and propose changes according to new trends in production traces. a gds refinement pipeline
what you can actually measure on live traffic:
code checks: valid format, PII leaks, right tool called
llm judges: is the answer grounded in the retrieved docs, did it answer what was asked
user signals: thumbs down, rephrasing the same question, asking for a human
ops: cost, latency, time to first token
a bit more about the GDS refinement loop
you can run it after every big change or on a schedule:
read the traces your online checks flagged
real failures become new GDS items
capability evals near 100% move to the regression set
items your graders keep getting wrong get fixed or deleted
this might seem overkill but overtime if you skip that and the GDS slowly stops looking like prod
i have seen this exact question in an interview:
your evals show a 95% pass rate but users keep complaining. what's wrong?
very often the GDS is testing stale patterns.
i used to think you cant unit test llm outputs because theyre probabilistic
thats only half true. code checks are unit tests for the deterministic parts (valid json, right tool called). the GDS is a test suite with fuzzy assertions
you dont need the whole loop on day one
early dev: trace everything and read outputs
clients who need certain queries to never regress: build the GDS first
live users: sample traces and run online judges
the loop only works if a human keeps reading the traces
@eugeneyan said it well: "automated evaluators amplify our existing annotation and feedback processes"
tomorrow: choosing what to evaluate