Pratik Karki (@ai_evals) header
Pratik Karki (@ai_evals) profile picture

🇳🇵🇺🇸 co-founder https://nitter.gq/t.co/Vz65cBoc7N | ex @google AI | training data and evals for frontier labs | https://nitter.gq/t.co/2EFb7GLRu4

San Francisco
Joined May 2025
Photo by Pratik Karki (@ai_evals)Photo by Pratik Karki (@ai_evals)Photo by Pratik Karki (@ai_evals)Photo by Pratik Karki (@ai_evals)Photo by Pratik Karki (@ai_evals)Photo by Pratik Karki (@ai_evals)Photo by Pratik Karki (@ai_evals)Photo by Pratik Karki (@ai_evals)Photo by Pratik Karki (@ai_evals)Photo by Pratik Karki (@ai_evals)
somehow a downgrade from the original
casting Monica Barbaro as temu Mira Murati is crazy work
47
GTA 6 so realistic I'm already getting hit by data brokers offering money for my crime organization's data.
1
1
130
Between RPatt bond announcement and 007 First Light being nominated for game of the year, great time to be a bond fan.
43
When an agent gets something wrong, was the information it needed even available? Your eval needs to distinguish a hard task from an impossible one. This is a legit question! And it's super important when we're doing evaluations with our customers. Let's say you have an agent deciding which construction blueprint to go with. But there are two construction drawings for the same date with conflicting issues. It has to make a guess. And that's where a lot of wrong things happen. For that construction issue, an updated drawing with the correct address and blueprints would make it easier. But without that evidence, or even somebody to ask, it cannot know which is right. Missing available evidence is inherently different from being expected to know something that was never given. It's a very important point that a lot of folks get wrong. So evals need to account for this. Clarification or flagging missing context should be accounted for in your evals - so the agent knows when it's right to move ahead instead of finishing confidently and being CONFIDENTLY WRONG. You see this across security and medical domains for instance A lot of times the information isn't available - it's outside the context of rules, unaccounted for by both the humans coding it and the people that created the system of records. Definitely seen this issue pop up quite a bit with our customers! So when you're building a test, make sure the agent is tasked with "hard" problems, not "impossible" problems. If it's a hard problem, no issue. It's a feature, not a bug. But if it's impossible, the agent is DONE FOR.
1
123
Pratik Karki retweeted
Mechazilla catches Starship! Congratulations to the entire SpaceX team on this incredible achievement!
6,797
27,354
9,103
141,208
17,411,998
Over-preparing >>> over-thinking. I have legit spent 20 hours preparing for a 20-minute meeting with some well-known names in the AI industry. It may seem obvious, but here are the differences: My standard prep, end to end, involves compiling as many resources as possible, going over them manually, thinking through things, and finding gaps. I do a lot of talking with my AI agents about this and talk aloud a lot. Also making a list of possible questions and practicing the answers out loud. Even having practice conversations where I look at myself in the camera to make sure I'm putting forward the first impression I want to. Finally, it's a way to talk about these things until they become natural, vs looking at a script in real time, which is never natural tbh. Preparation should not be skipped, even by the smartest folks. In the heat of the moment, it's difficult to adjust when you're unprepared. Famously, Winston Churchill used to do this - he had a lisp and put in about an hour of preparation for every minute of speech delivery. No one could tell about the lisp and the prep made him a great leader in my opinion. Whereas with overthinking, you have ten million thoughts without putting them on paper and working through them. You just end up overthinking and messing up situations. So if you have a big meeting/presentation/interview coming up, prepare as much as possible! If never not worth it.
48
Pratik Karki retweeted
BREAKING: Literally everything
151
5,210
125
35,658
387,007
NVIDIA's ARR is actually (Affairs Run Rate)
Jensen Huang Had A Third Affair—With Another Nvidia Employee, Report Says go.forbes.com/rna5YR
50
It's only Full Self Driving if it's from the Austin TX Tesla Giga factory. Otherwise it's just Sparkling lane assist.
48
When you first join a workplace, it takes you a while to learn the processes. It's no different for an AI agent. What really takes your time when you join a new place? It's not really learning the tools, but learning how the particular place works. Like you could be: - an engineer finding an old service still runs in production even though it's retired or out of scope - a recruiter learning that the conversation with the hiring manager works in certain ways - a retail worker learning what exceptions need approval from a supervisor. The context lives in various places: in somebody's head, old messages, docs that don't make sense, "final V2" files, or just office gossip. Now with agents, just having really good software doesn't automatically give them the context. When everything is tidy and every instruction is explicit, that does NOT reflect the real world at all. Training environments need to reflect that mess too! Really well-built agents from our customer companies can also go through this. The production environment is just so much different from something built in test. So being able to find the relevant files, and weigh which ones actually matter, is where a lot of the work is. So the main question here is: what should people building these environments be paying attention to? I know at Anthromind what we're paying attention to, and that's led to quite a few great agent adoptions.
5
1
2
194
This whole thing sucks, and I wish none of it had happened. It’s a really sad day! But I want to make one thing clear: we’ll keep fighting for open-source AI, harder than ever. This is too important to slow down now. Let’s go!
448
940
161
10,571
613,712
Pratik Karki retweeted
Jensen likely already out there raising for his next thing. $2B seed at $100B post.
11
7
465
38,790
Sounds like a great opportunity. Hit me up if you take this class and if you wanna use your skills in production. We're always looking to expand the team with great talent.
Big little announcement: @karinanguyen and I are excited to announce a brand new course on Full-Stack Post-Training, RL & RSI at @UCBerkeley. If you are a Berkeley student and want to learn about the frontier, enroll for the next semester.
1
125
Why is a federal holiday in the middle of the week again? Why not long weekend like other holidays?
71
Ramayana Part 1: Growing up Hindu and reading the comic books (ACK) and watching the TV serial as a kid, I've been waiting for this for a very long time. In short, it's epic in all the right ways. Let's start with a few cons: CGI, I think, is 95% good, 5% a little wonky but not too bad in the scheme of things. For such a fantastical story it is inevitable. I think the music was a miss too. Aside from a few awesome fight scenes the music was kinda generic. They could have removed all Western music honestly. The good: everything from the characters, the actual backstory of Ravan, being super faithful to the story, lore tidbits, giving a lot of heart to characters like Lakshman and Jayatu, as well as a better role for Sita - I really liked it. I thought the pacing was great. 3 hours and 15 minutes went by in a blink of an eye. It looked great in IMAX too! The themes of duty, dharma, and family were refreshing as well. We're too used to the cynical postmodern philosophies where nothing really matters. But having a character who is stoic and graceful and follows duty and honor to a tee - that is commendable and refreshing. I think Ranbir Kapoor as Ram is actually very underrated. Very restrained performance. Yash as Ravan, obviously a scene stealer. Amazing screen presence, especially the fight at the end. Shoutout to Anupam Kher, made me tear up a bit🥲 Finally, Nepali is a language that is very Sanskrit-based. We do not have any Urdu words so the dialogue was very refreshing because it followed how we speak very closely and it matches how we're taught it in school. Words like Dhanyabaad, for instance. I was a big fan of that commitment to the spirit of the text. Overall a great movie and clearly what I imagined a Lord of the Rings-style Ramayana to be when I was a kid. Part 2 is probably gonna be so damn epic, can't wait!
7
1
35
891
It's been 2 months since I made Instinct my chief of staff. If you're considering doing the same, here's my no-BS thoughts: The small stuff first: daily priorities, reminders, vendor subscriptions, personal admin etc. Even handles weekly movie night theater tickets coordination with the wife's instinct! That cleared a lot of daily friction, super valuable imo. What works is I set up golden docs on how I like things done, and handing stuff off by WhatsApp voice note is easy, even from my watch. Another good use of Instinct: practice reps out loud. We go back and forth on customer conversations, it raises objections at me. I prep event speaking the same way. Pair with granola notes and you're golden. You catch a LOT listening to yourself say stuff out loud. What I don't hand off: my emails and posts, prep for customer calls, thinking about customer work, day-to-day coding decisions. Those stay with me and I rely on voice notes for stuff like emails so it's authentic. What still sucks: - Long-term context - memory is still hard to solve. - One-off rules get followed relentlessly, long after I wanted them to!! - Sometimes it fails to run on schedule. - Can't make phone calls on my behalf yet (would have been useful to not have to deal with a recent eSIM setup for instance) Not magic, but I get to spend more of my time on the work I actually care about now. Personal AI is still in its infancy, but excited to be an early adopter! @noahrshinn
2
4
477
You either die a "neo" tech or live long enough to see yourself become a "legacy" tech
52
Pratik Karki retweeted
Narrative violation: Met a fisherman from Santa Cruz last night who’s been fishing for over 50 years. He said the ocean is the healthiest he’s ever seen it. More fish than ever, whale populations are booming, species that had nearly disappeared are coming back. Not all doom and gloom out there.
162
262
46
6,288
486,179
Modern problems require modern solutions
🤖 Made with AI
64
I think 100% die eventually dude. Maybe @bryan_johnson will be the exception?
This quoted post is unavailable.
76