yesterday i got a book about cool cafes in thailand. i thought, cool, let's put them on google maps. easy task for haiku 5.5, right? nope. it failed. so i made opus 5.5 the boss agent, with sonnet on backup for the hard parts. big brain energy, small brain result.
9/ lessons:
- every public form that sends email gets the same protection as login. no exceptions
- a feature nobody uses is pure attack surface. delete it
- read the actual log before theorizing. sending IP first
- set a sending cap/alert at your mail provider
8/ fix: i could have bolted turnstile onto the invite form too. but basically nobody ever used it. so i deleted the whole feature, 540 lines. the "copy link" button next to it does the same job and can't email strangers.
7/ the stupid part: my magic-link login form already had bot protection. cloudflare turnstile + a honeypot + a "submitted too fast" check. the invite form had none of it. i locked the front door and left the side door open.
developers.cloudflare.com/tu…
6/ it sat there since january. bots found it in august. sender name on one of them: "MarvinEstiz". classic.
5/ the cause: a public "invite a family member by email" form on my landing page. no login, no captcha, no rate limit. visitor types the recipient, a name and a free-text message, my app emails it. that's a free spam relay with my domain's reputation on it.
4/ then i finally read one full log event instead of guessing. two fields killed the theory:
- originating-ip = my own server
- subject = my own email template
nobody stole anything. my app was sending the spam.
3/ so i went hunting for the leak with fable: git history, repo visibility, docker image, /.env on prod, shell history. all clean. only other copies of the key: old AI agent transcripts, because the agent had read my .env. not the cause, but yikes.
2/ first thought: api key leaked. mailgun logs put an `api-key-id` on every sent message. i matched it: my app's production key. panic.
too lazy for that:
paste the article link to your assistant and tell it to find where this fits in your stuff.
claude.dev/blog/automating-e…
how to start:
1. pick one thing. support is fine.
2. grab real cases + good answers.
3. type /claude-api build-eval in claude code
4. then type /claude-api hillclimb
back to the hill:
height = test score. like how many emails the ai sorted right.
step = one small change. better prompt. cheaper model (hello smaller api bill).
after every step claude reruns the test and keeps the change only if the score went up.
i'm plugging it into sales too.
i have my old leads and i know who paid. so i can check if the agent picks the right people to reply to, and with what offer.
turns out claude code can just run the tests for you.
you give it real tickets + the good answers. it changes the prompt or swaps the model, then checks: number go up or number go down?
up = keep.
down = trash.
why care: it tells you if your agent is actually good. like the bot answering support for your little saas.
how we all test it now: vibes.
read 3 replies.
"looks fine lol."
ship.
hillclimb = you climb a hill with a bag on your head. can't see the top.
you take one tiny step and feel with your feet: am i higher?
yes = stay.
no = go back, try another way.
repeat like an idiot until you're on top. that's the whole algorithm
ok so claude has a thing that everybody slept on 😳
it's called hillclimb.
it's basically one lego brick of a company that fixes itself. 🧵
