@amankhan

AI PM @google on evals, benchmarks | ex @arizeai views my own

amank.ai
Joined December 2011
  • Tweets792
  • Following1.1K
  • Followers3.2K
64Media
1/ Been trying out more AI coding Agents - here's a quick breakdown of my experience with the AI coding agent space and what I use each agent for
29
137
10
963
129,428
superhuman intelligence is literally VC subsidized what an insane time to be alive
1
2
335
Aman Khan retweeted
Abundant
@abundant_labsOct 23
Abundant is partnering with @baseten to empower customers to own their capabilities. Excited to leverage our experience working with every frontier lab to build environments tailored to the customers' unique needs.
3
13
5
63
6,809
btw this is why you need an internal eval benchmark flywheel - not just a one and done set of tasks evals will become a core skillet (still early) for every company that wants to use AI and the more “up to date” they are, the higher impact new models will have on that business.
The Information
@theinformationOct 18
OpenAI, Anthropic and other leading AI labs face a growing challenge: Better models aren't necessarily attracting more usage. A 10-point intelligence advantage now earns just 20% more developer traffic, down from nearly 69% in late 2025. Full story: thein.fo/4hGJ9jY

Leading-Edge AI Models Are Losing Their Allure With Developers

As a lifelong, overly enthusiastic San Francisco Giants fan, this was a rough summer. Essentially out of contention since week six, the Giants ended the season with a dubious distinction: one of the...

theinformation.com
2
323
We are hosting our first "Bench Summit" to bring together people excited about pushing the frontier of benchmarking, model evals, and more. Great benchmarks are a gating function for model progress, lots to do here! Join us in SF: rsvp.withgoogle.com/events/b…

Bench Summit by Kaggle

Bench Summit unites creators and practitioners advancing the next era of AI evaluation. Join experts from Google DeepMind, Kaggle, and Zapier for keynotes, real-world benchmark case studies, hands-on...

rsvp.withgoogle.com
131
38
6
833
107,987
Should everyone on your team have the same personality?
Anthony Morris ツ
@amorriscodeOct 12
I don’t want to manage a agents with different personas. If intelligence is truly super there should be no need for roles. Smart generalists rule the world. If you’re making agents with personas, are you actually finding value from it?
1
1
3
1,008
pacing the frontier is a marathon not a sprint
226
what if every query had its own harness
3
5
760
agree
Rayan Krishnan
@RayanKrishnanOct 7
Auditing RL environments and reward models is one of the highest-leverage tasks evaluators can do in service of building aligned models (embedded or otherwise).
3
319
token based pricing is honestly very strange - maps roughly to compute time pricing, except because LLMs are non deterministic this is basically unpredictable - no direct hop to value or capacity - incentives don’t exist to lower token spend w/o competition (see: billable hours)
1
13
817
extreme outcome distributions are seemingly becoming the norm even more - in using AI products (and how to get value from them) - in fraud, abuse and unexpected outcomes (below) - in perception of AI systems (they will cure everything vs kill us all)
This quoted post is unavailable.
Try viewing anyway · View on X
1
118
could you imagine a world where inference just turned off for the weekend
2
6
381
somehow, for the last few years, this is always the time of year when things get really interesting
1
3
441
the most stressful part of the job now handled by agents curious what friends at @datadoghq @cleric_io @sentry @pagerduty @arizeai @resolveai think!
andre --dangerously-skip-permissions
@andrezfuSep 30
Agents write application code, but you still get paged at 2am when it breaks. We wanted to know whether AI could handle that part of the job too. Introducing Incident Arena: a benchmark that puts coding agents on call! Check out our paper & full dataset release below!
2
10
1,593
the energy in the personal agents space is kind of electric. can’t wait for the LinkedIn takes once someone actually builds a good one for enterprise. 👀
10
437
today’s a good day to work on evals
1
9
958
does anyone else spend their Sunday after a big model release week just updating their workflows and code?
3
329
so many processes, layers and systems just don’t make sense anymore in a world of capable agents
1
1
6
373
Aman Khan retweeted
Aman Khan
@amankhanSep 21
Sometimes classification is all you need. What's old is new again.

Jev and the bittersweet lesson

Article · @amankhan
4
11
2
81
19,236
Aman Khan retweeted
Viv
@Vtrivedy10Sep 21
very clear that harnesses across the industry will become specialized boxes optimized for a given task -> some bundling tooling like Jev! Model-Harness-Task fit some of it “just-in-time”, most prob not the best vertical agents today have already been doing this for a while -> unbundle all prompts/tools/skills/models and rebundle them for your Task domain everyone will calibrate via evals, secret sauce is still in the data layer
Aman Khan
@amankhanSep 21
Sometimes classification is all you need. What's old is new again.

Jev and the bittersweet lesson

Article · @amankhan
6
1
46
6,004
Sometimes classification is all you need. What's old is new again.

Jev and the bittersweet lesson

Article · @amankhan
4
11
2
81
19,236
Load more