Berkeley EECS PhD
UC Berkeley
Joined June 2025
- Tweets9
- Following11
- Followers47
- Likes24
Shulu Li retweeted
Scaling self-verification with DeepSeek V4 Flash beats Claude Fable 5 on Terminal-Bench 2.1, while being 11x cheaper 💰
As open-source models become more capable, they can now generate large numbers of high-quality candidate solutions and verify their own outputs at very low cost.
For example, we find that sampling just 5 solutions with DeepSeek V4 Flash and ranking them using the same model with LLM-as-a-Verifier can lead to a significant boost in accuracy (79% → 88%), outperforming closed frontier models on Terminal-Bench.
Try it out today: github.com/llm-as-a-verifier…
More on verification scaling in my previous post.
How can we extract richer signals from AI Feedback?
Introducing LLM-as-a-Verifier✨— a simple verification scaling framework that achieves SOTA on agentic benchmarks 🚀
The key idea:
- Use fine-grained scoring granularity (e.g., 1-20 instead of the standard 1-5 scale)
- Take the expectation over the full logprob distribution of score tokens
- Scale repeated evaluation and criteria decomposition
You can use these fine-grained signals for more effective test-time scaling, RL, and agent monitoring! It achieves SOTA across Terminal-Bench V2, SWE-Bench Verified, RoboRewardBench, and MedAgentBench 👑
Advised by @Azaliamirh @istoica05 @drmapavone @chelseabfinn
🧵👇
Shulu Li retweeted
How can we extract richer signals from AI Feedback?
Introducing LLM-as-a-Verifier✨— a simple verification scaling framework that achieves SOTA on agentic benchmarks 🚀
The key idea:
- Use fine-grained scoring granularity (e.g., 1-20 instead of the standard 1-5 scale)
- Take the expectation over the full logprob distribution of score tokens
- Scale repeated evaluation and criteria decomposition
You can use these fine-grained signals for more effective test-time scaling, RL, and agent monitoring! It achieves SOTA across Terminal-Bench V2, SWE-Bench Verified, RoboRewardBench, and MedAgentBench 👑
Advised by @Azaliamirh @istoica05 @drmapavone @chelseabfinn
🧵👇
SOTA on Terminal-Bench 2 and SWE-Bench verified with verifiers!
We release LLM-as-a-Verifier 🧠: A general-purpose verification framework that achieves SOTA 👑 on Terminal-Bench 2 (86.4%) and SWE-Bench Verified (77.8%) by scaling:
- scoring granularity
- repeated verification
- criteria decomposition
📄 Blog & Code: llm-as-a-verifier.notion.sit…
4
509
Shulu Li retweeted
We release LLM-as-a-Verifier 🧠: A general-purpose verification framework that achieves SOTA 👑 on Terminal-Bench 2 (86.4%) and SWE-Bench Verified (77.8%) by scaling:
- scoring granularity
- repeated verification
- criteria decomposition
📄 Blog & Code: llm-as-a-verifier.notion.sit…
I built a tool that analyzes how Claude Code actually works under the hood.
Every prompt. Every tool call. Every task decision. All mapped out visually.
One command to try it yourself:
github.com/depetrol/claude-a…
1
2
90
Shulu Li retweeted
🎯 AI optimizes congestion control for datacenter networking
[ADRS Blog #13] We apply ADRS frameworks to improve congestion control algorithms. Using OpenEvolve, we discover an algorithm that reduces queue length by 49% on the NSDI ’22 PowerTCP 10:1 incast benchmark! 📉⚡️
✍️ Read the Blog: adrs-ucb.notion.site/cca
📖 ADRS Blog Series: ucbskyadrs.github.io/
📄 ADRS Paper: arxiv.org/abs/2510.06189
👩💻 Code: github.com/UCB-ADRS/ADRS
