Suman Dey

Here's how to actually master LLM evals, step by step, with real projects to prove what you've learned: 1. Build something to break first Don't learn evals in the abstract. Build a small LLM app with a real failure surface: RAG over a docs site, a text-to-SQL bot, or a tool-using agent. Ugly is fine. You need traces, not polish. 2. Read 100 traces by hand Dump input, output, retrieved chunks, and tool calls into a spreadsheet. Write one line per failure in plain English. Cluster them. Count them. Most people skip this and build metrics for problems they don't have. Project: publish your failure taxonomy with counts per category. 3. Build your first golden dataset 50 cases from your top 3 failure categories, 20 easy happy-path cases, 10 adversarial ones. Columns: input, expected output or rubric, category, difficulty, segment. Commit it to git and tag every prompt change with the dataset version it was scored on. 4. Generate synthetic data properly Write your dimensions (persona × intent × complexity × edge case), generate the tuples with a script, then have an LLM turn each tuple into a natural query. Review and throw out the bad ones. Dedupe near-duplicates with embeddings. Project: a 300-case synthetic set with a coverage matrix showing which combinations you tested. 5. Write code checks before judges pytest + pydantic for schema validation, regex, required fields, length limits. Run generated SQL against a real SQLite test DB. Execute generated code in a sandbox. Try to make at least half your failure categories checkable without an LLM. 6. Build and align an LLM judge Hand-label 100+ examples pass/fail. Split into train/dev/test. Binary verdict, written rubric, critique before verdict. Iterate the prompt on dev until TPR and TNR are both high, then report on test once. Pin the judge model version. Project: a judge-alignment notebook with a confusion matrix and every prompt iteration you tried. 7. Split your RAG evals Label the relevant chunks for 50 queries. Compute recall@5, MRR, and nDCG. Then score faithfulness by breaking each answer into claims and checking each one against the retrieved context. Change chunk size or embedding model and watch which number moves. 8. Score agent trajectories Log tool name, args, result, and tokens for every step. Write reference paths for 30 tasks. Score tool precision/recall, argument correctness, step count, and error recovery. Make a tool fail 20% of the time on purpose and see if your agent recovers or spirals. 9. Simulate users for multi-turn Give a simulator LLM a persona, a hidden goal, and constraints ("won't share the order number unless asked twice"). Run 50 conversations with a 15-turn cap. Grade goal completion, turns-to-resolution, and policy violations. 10. Measure reliability, not luck Run every case 5x at your production temperature. Report pass@1 and pass^5 side by side. The cases that flake are your real production bugs. 11. Learn enough stats to not fool yourself Bootstrap your confidence intervals. Compare models paired on the same examples (McNemar's test works for pass/fail). If your test can't separate the two runs, you didn't improve anything, you got lucky. 12. Debias pairwise judging Run every A/B comparison in both orders and only count verdicts that agree. Plot win rate against answer length. If longer always wins, your judge is grading word count. 13. Wire it into CI A GitHub Action on every PR that touches prompts, models, tools, or retrieval config. Fast deterministic checks on every PR, the full judge suite nightly or on a label. Per-category thresholds. Post the score diff as a PR comment. Project: a PR where CI blocks a real regression. Screenshot it. That's your proof. 14. Go online Sample 5% of live traffic, run judges async, build a per-category dashboard, and alert on drift. Shadow-deploy a new model before switching. Build the loop: thumbs-down → triage queue → new labeled test case. 15. Make cost a metric Log tokens, cost per successful task, and p50/p95 latency. Swap in a cheaper model; if quality holds in CI, chart the savings. "Same quality, 40% cheaper" is the easiest win to sell to any team. Tools: pytest, pydantic, promptfoo, DeepEval, Ragas, Inspect, Langfuse, LangSmith, Braintrust, Arize Phoenix. Pick one stack, learn the concepts. Tools change. Read: Hamel Husain and Shreya Shankar on evals, Eugene Yan's posts on LLM evaluation, and the τ-bench paper (where pass^k comes from).