AI Evals & Benchmarks @ Zapier
San Francisco, CA
Joined January 2007
- Tweets3.4K
- Following1.7K
- Followers1.6K
- Likes22K
Had fun! Even though we chewed through my prepared questions in the first twenty minutes 😅
One thing I realized today is that innovation and model improvement isn't something that just comes out of the labs, it's a collaboration across labs, people building benchmarks, users...
@zapier, one of our Bench Summit partners, led a panel discussion on the realities of managing highly influential evaluation signals.
The panel, moderated by Lukas Bergstrom (@lukasb), included insights from Robin Salimans (@SalimansRobin) of Zapier, Greg Kamradt (@GregKamradt) of ARC Prize Foundation, and Shaiyon Hariri of Ramp.
Attendees heard about what defines a successful benchmark, the challenges of maintaining adoption, and critical lessons learned from V1 to V2+.
My home office has a bottlebrush tree outside the window that gets a lot of hummingbirds. Turns out hummingbirds have been around for at least thirty million years. Oldest fossil is Eurotrochilus inexpectatus, called that because no one expected a European hummingbird.
Lukas Bergstrom retweeted
kinda wish we had a better metaphor for how I think of LLMs at the mechanistic level, which is "bags of contextually activated circuits, heuristics, and algorithms"
New in tidepools.ai
I love that tidepools formulas let you insert live dashboards anywhere. But 90% of the time I just want a quick dashboard right where I'm working - today's journal page.
So meet the new Todos section:
Heads up that the AutomationBench numbers they're reporting here are based on our public dataset. Scores on the public dataset are higher, empirically - likely because the public dataset was used in training. research.meta.ai/blog/introd…
Will definitely happen. Will definitely make traffic worse.
I was driving around in our @Tesla Model Y today and just couldn't find a parking spot anywhere. Small town, busy, touristy weekend. Drove around a few times, no dice. I just needed to run in somewhere for 15 minutes.
Made me wish I could drop myself off, tell my car when I want to be picked up, and the Tesla could just drive around until it's time and then come pick me up again. Don't need to park.
It could do loops out of congested areas so it's not making traffic worse.
Sort of like Actually Smart Summon, but rather than me parking and then the car coming curbside later, it would just drive itself around, not park, and come back when I tell it I'm ready.
Maybe once they're cleared for fully driverless driving.
Watching the Timber Fire burn through one of my favorite places on earth.
At the same time, stoked about how mature Watch Duty has become. This is a high-quality update. app.watchduty.org/i/113784
If you have the choice, highly recommend working at a company that runs on its own products. Dogfooding is the minimum, making it critical path is best.
Frontier might be getting even more jagged. Not surprising, probably 10000x as much training data on writing code.
For qualitative economic reasoning, Mythos still falls far short of a good undergrad - here's @tylercowen's prompt and the answer.
Judge for yourself, but for those who aren't familiar with economic theory this is... not good.
I don't know how personalized this was to Tyler
1/
Lukas Bergstrom retweeted
Gemini 3.5 Flash ranks #1 on Automation Bench (from Zapier), beating every other frontier model at a much lower cost
Lukas Bergstrom retweeted
Imagine every pixel on your screen, streamed live directly from a model. No HTML, no layout engine, no code. Just exactly what you want to see.
@eddiejiao_obj, @drewocarr and I built a prototype to see how this could actually work, and set out to make it real. We're calling it Flipbook. (1/5)