levanto β˜€οΈ πŸ‡°πŸ‡ͺ πŸ‡΅πŸ‡Έ (@levanto_0) header
levanto β˜€οΈ πŸ‡°πŸ‡ͺ πŸ‡΅πŸ‡Έ (@levanto_0) profile picture

Cypherpunk. #EndPeriodPoverty

Joined May 2019
Photo by levanto β˜€οΈ πŸ‡°πŸ‡ͺ πŸ‡΅πŸ‡Έ (@levanto_0)Photo by levanto β˜€οΈ πŸ‡°πŸ‡ͺ πŸ‡΅πŸ‡Έ (@levanto_0)Photo by levanto β˜€οΈ πŸ‡°πŸ‡ͺ πŸ‡΅πŸ‡Έ (@levanto_0)Photo by levanto β˜€οΈ πŸ‡°πŸ‡ͺ πŸ‡΅πŸ‡Έ (@levanto_0)Photo by levanto β˜€οΈ πŸ‡°πŸ‡ͺ πŸ‡΅πŸ‡Έ (@levanto_0)
levanto β˜€οΈ πŸ‡°πŸ‡ͺ πŸ‡΅πŸ‡Έ retweeted
Can we reliably measure whether frontier models know they are being evaluated? πŸ”¬ New paper from me, @xinningli6, @levanto_0,@AlexandraSouly, @_robertkirk: EvalDetectBench, a benchmark for measuring evaluation awareness in frontier LLMs 🧡
4
7
3
42
9,373
levanto β˜€οΈ πŸ‡°πŸ‡ͺ πŸ‡΅πŸ‡Έ retweeted
Have you debugged your training data? You might not like what you find. Introducing predictive data debugging: reveal and shape what your model will learn before training. In DPO datasets, we found broken guardrails, hallucinations, and fish fart fan fiction (seriously). (1/9)
25
107
56
908
187,451
levanto β˜€οΈ πŸ‡°πŸ‡ͺ πŸ‡΅πŸ‡Έ retweeted
Where does evaluation awareness and evaluation gaming come from? πŸ”¬πŸ“‰New post from @arbdwj and me tracing evaluation awareness through OLMo 3 training! 🧡
1
5
1
23
2,271
levanto β˜€οΈ πŸ‡°πŸ‡ͺ πŸ‡΅πŸ‡Έ retweeted
πŸ₯ΉπŸ’œπŸ‡°πŸ‡ͺπŸ™
345
3,001
129
27,837
399,738
levanto β˜€οΈ πŸ‡°πŸ‡ͺ πŸ‡΅πŸ‡Έ retweeted
From top of the world to the heart of Africa, we are with you. Stay strong, Kenya!πŸ‡°πŸ‡ͺ
153
5,297
107
15,722
341,110
levanto β˜€οΈ πŸ‡°πŸ‡ͺ πŸ‡΅πŸ‡Έ retweeted
"kwani watanichapa" this mindset will take you places
42
1,018
20
4,675
127,026
levanto β˜€οΈ πŸ‡°πŸ‡ͺ πŸ‡΅πŸ‡Έ retweeted
Recently, there has been a lot of talk of LLM agents automating ML research itself. If Llama 5 can create Llama 6, then surely the singularity is just around the corner. How can we get a pulse check on whether current LLMs are capable of driving this kind of total self-improvement? Well, we know humans are pretty good at improving LLMs. In the NanoGPT speedrun challenge, created by @kellerjordan0, human researchers iteratively improved @karpathy's GPT-2 replication, slashing the training time (to the same target validation loss) from 45 minutes to under 3 minutes in just under a year (!). Surely, a necessary (but not sufficient) ability for an LLM that can automatically improve frontier techniques is the ability to *reproduce* known innovations on GPT-2, a tiny language model from over 5 years ago. πŸ€” So we took several of the top models and combined them with various search scaffolds to create *LLM speedrunner agents*. We then asked these agents to reproduce each of the NanoGPT speedrun records, starting from the previous record, while providing them access to different forms of hints that revealed the exact changes needed to reach the next record. The results were surprisingβ€”not because we thought these agents would ace the benchmark, but because even the best agent failed to recover even half of the speed-up of human innovators on average in the easiest hint mode, where we show the agent the full pseudocode of the changes to the next record. We believe The Automated LLM Speedrunning Benchmark provides a simple eval for measuring the lower bound of LLM agents’ ability to reproduce scientific findings close to the frontier of ML. Beyond scientific reproducibility, this benchmark can also be run without hints, transforming into an automated *scientific innovation* benchmark. When run in "innovation mode," this benchmark effectively extends the NanoGPT speedrun to AI participants! While initial results here indicate that current agents seriously struggle to match human innovators beyond just a couple of records, benchmarks have a tendency to fall. This one is particularly exciting to watch, as new state-of-the-art here by definition implies a form of *superhuman innovation*.
40
193
36
1,189
572,512
levanto β˜€οΈ πŸ‡°πŸ‡ͺ πŸ‡΅πŸ‡Έ retweeted
KRA Portal looking at me struggling to log in because I waited till the last minute to file my returns
29
1,067
39
6,850
178,188
levanto β˜€οΈ πŸ‡°πŸ‡ͺ πŸ‡΅πŸ‡Έ retweeted
Tonight we dream. πŸ’œ #Breaking4
476
1,382
212
6,443
289,824
levanto β˜€οΈ πŸ‡°πŸ‡ͺ πŸ‡΅πŸ‡Έ retweeted
Chasing history, creating legacy πŸ’œ Still the fastest mile run by a woman in history, @Kipyegon_Faith gives everything in her quest to break the 4️⃣ minute mile and stops the clock at 4:06.42. Thank you, Faith, for making us dream. Maybe not today, but soonβ€¦πŸ˜€
94
1,255
130
3,532
187,112
levanto β˜€οΈ πŸ‡°πŸ‡ͺ πŸ‡΅πŸ‡Έ retweeted
Faith Kipyegon had the audacity to dream of doing the impossible. The fastest woman to ever run the mile just ran it faster at #Breaking4, pushing the world closer to the 4-minute barrier. It’s not a matter of if a woman will break 4, it’s when.
117
1,427
131
6,784
350,115
levanto β˜€οΈ πŸ‡°πŸ‡ͺ πŸ‡΅πŸ‡Έ retweeted
Kenyan police cornered peaceful protesters in a blocked alley, beat them up, then teargassed them. "Larry ukienda tutauliwa," one of them said. "If you leave, we'll get killed." Police brutality during protests against police brutality
1,196
22,344
962
41,993
1,883,112
levanto β˜€οΈ πŸ‡°πŸ‡ͺ πŸ‡΅πŸ‡Έ retweeted
Today is definitely bigger than last year’s, nothing can bring them back but we owe them our memories and justice to be served. This is Kisii town. #SiriNiNumbers
29
2,732
21
6,218
115,286
levanto β˜€οΈ πŸ‡°πŸ‡ͺ πŸ‡΅πŸ‡Έ retweeted
So proud of each and everyone who came out to honour our heroes and fallen comrades. Umoja✊🏽 Undugu❀️
6
2,842
28
5,900
61,334
levanto β˜€οΈ πŸ‡°πŸ‡ͺ πŸ‡΅πŸ‡Έ retweeted
🧡THREAD: How to Help an Unconscious Protester β€” The Right Way to Save a Life.
1
23
2
33
7,000
levanto β˜€οΈ πŸ‡°πŸ‡ͺ πŸ‡΅πŸ‡Έ retweeted
BTW, if you're in your 20s, normalise growing around a community. Invite people over, eat out, do house warming, etc. Don't do life alone. A plus if all of you have shared values. Have a space where you can talk about your dreams, and someone would follow up weeks later with "BTW, how is your kitchen garden doing?", "Were you finally able to run 5k in under 30 minutes?" Don't do life alone. Life is beautiful with the family you create. Don't be so much into hustle culture and forget what really matters.
32
1,305
60
4,289
153,441
levanto β˜€οΈ πŸ‡°πŸ‡ͺ πŸ‡΅πŸ‡Έ retweeted
It’s a heavy day. Still we rise. #RutoMustGo
29
7,044
82
15,843
416,072
levanto β˜€οΈ πŸ‡°πŸ‡ͺ πŸ‡΅πŸ‡Έ retweeted
No one else will do it for us #OccupyEverywhere
8
2,970
34
6,385
156,865
levanto β˜€οΈ πŸ‡°πŸ‡ͺ πŸ‡΅πŸ‡Έ retweeted
Ndo kudonjo. @KenyaAirways wamecheza kama wao
1,413
7,086
436
33,638
952,538
levanto β˜€οΈ πŸ‡°πŸ‡ͺ πŸ‡΅πŸ‡Έ retweeted
Today marks the first anniversary of our fallen comrades. We remember. We remember ✊🏿✊🏿✊🏿
48
3,929
46
5,438
141,466