Public LLM Evaluation // https://nitter.gq/t.co/rNS39rhVQ9 @8vc @BloombergBeta @pearvc @a16z @HRTVentures @nextladder
San Francisco, CA
Joined March 2024
- Tweets2.4K
- Following280
- Followers29K
- Likes844
Pinned Tweet
For one weekend, we are benchmaxxing on purpose!
Join us for the first Vals AI Hackathon on November 14–15 in San Francisco. We're giving 150 hackers 24 hours and a lot of compute to improve small models, challenge the frontier, and see what they can build.
Track 1: Post Train a small model and see how high it ranks on three Vals benchmarks. Live leaderboard, compute from our sponsors.
Track 2: Build a Your Own Eval on Valkyrie for tasks that still stump frontier models. We cover model and compute costs.
Come meet the Vals team. Expect great prizes, notable guest speakers and fun side quests, including an optional cybersecurity CTF.
Large 4 OSS is a more efficient model without a major change in overall accuracy. It makes meaningful gains on app-building and legal research while using fewer output tokens and steps per task.
The model has a 1M-token context window and 256k max output tokens. It was run with default settings: temperature 1, top_p 0.95 and high reasoning effort.
Congrats to @MistralAI on the release!
To round out the Vals College Tour, we are wrapping up at the @Princeton Career Fair on Oct 29-30th🐯!
Princeton alumni, @conjfrnk will be there to answer your questions about Vals, post grad decisions, and hand out exclusive merch. Come say hi, we can't wait to meet you.
Carbon generates 64% more output tokens than Argon, but the increase is concentrated in Finance Agent, Tax and Legal Research. On EMB, Terminal-Bench 4 and Harvey, it uses roughly the same or fewer.
Even with that additional output, Carbon generates less than half as many output tokens as Sonnet 5.5, the only model ahead of it, and Opus 5.5.
This model has a 1M context window, 1M max output tokens, and is multimodal. We ran it with Google's default provider settings: temperature=1, top-p and top-k= default, and high reasoning effort.
Congrats to @GoogleDeepMind. Full results at vals.ai/models/google_gemini…
About half of U.S. adults now use AI chatbots, and questions about visas, work permits and immigration status are among the most consequential they ask. Can the models be trusted?
With the Talent Mobility Fund, a program of @RenPhilanthropy, we are releasing Immigration Bench to test whether frontier models are able to answer these high-stakes questions.
For immigration attorneys, AI can help with research and drafting, but cannot replace advice tailored to a client’s situation. For AI labs, improving reliability means working with immigration experts on current, practice-grounded data.
This benchmark is a first step. AI use in immigration shows potential, and we want it used with a clear view of where it falls short and what is at stake.
Immigration Bench is a joint effort with the Talent Mobility Fund. We thank the immigration experts who wrote the questions and cross-reviewed every rubric.
Results and methodology: vals.ai/benchmarks/immigrati…
Full report: vals.ai/blogs/can-ai-get-imm…
