Being non-technical in San Francisco must feel like divine retribution At least Fitness SF knows their customer base

🚀 Introducing DeepInfer, our in-house inference framework, now fully open source! Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable. 🔗 github.com/deepseek-ai/DeepI… 1/4

We’re launching leaderboards for HealthBench and HealthBench Professional, two benchmarks evaluating how AI models respond in realistic health conversations HealthBench and HealthBench Professional are benchmarks from @OpenAI that grade model replies to realistic health conversations against physician-created rubrics. HealthBench tests models on 5,000 health conversations with everyday users and health workers. They cover seven themes, from emergency referrals to global health, and each rubric criterion falls under one of five axes: accuracy, completeness, communication quality, context awareness and instruction following. HealthBench Professional is a smaller set of 525 conversations which, unlike HealthBench, are all with clinicians. These cover consultation, writing and research tasks, and include adversarial and difficult conversations designed to challenge models. We report both the raw score (the share of rubric points earned) and the length-adjusted score that OpenAI reports. The length-adjusted score lowers the score of replies longer than 2,000 characters and raises the score of shorter ones. Consistent with OpenAI’s original implementation, replies are graded by GPT-5.4 (low) (HealthBench Professional) and GPT-4.1 (HealthBench). Key results for HealthBench Professional: ➤ GPT-6 Astra and GPT-6.1 Sol top the leaderboard: GPT-6 Astra (max) scores 66.0% on the length-adjusted score, comparable to GPT-6.1 Sol (max, 65.1%) and ahead of Claude Sonnet 5.5 (max with fallback, 63.0%) ➤ OpenAI models make up the entire score vs. Cost per Task Pareto frontier: The Pareto frontier consists of GPT-6 Luna (max, 61.9%, $0.004 per task) through GPT-6 Sol (max, 62.3%, $0.036) and GPT-6.1 Sol (max, 65.1%, $0.047) to GPT-6 Astra (max, 66.0%, $0.27). GPT-6 Luna (max) scores 4 points below GPT-6 Astra (max) at under 2% of the cost ➤ Claude Opus 5.5 and Claude Fable 5.1 receive the largest length penalties: Both average ~5,900 characters per answer, nearly 3x the 2,000-character threshold, and lose over 11 points after adjustment. Claude Opus 5.5 (max with fallback) has the highest raw score (71.4%) but drops to sixth (59.9%). GPT-6 Sol (max) is the only model averaging under 2,000 characters (~1,600), and the only one whose score rises Key results for HealthBench: ➤ Claude Sonnet 5.5 tops HealthBench: Claude Sonnet 5.5 (max with fallback) scores 62.1% on the length-adjusted score, ahead of DeepSeek V4.1 Flash (max, 60.9%) and GPT-6.1 Sol (max, 58.5%) ➤ GPT-6 Luna, DeepSeek V4.1 Flash and Claude Sonnet 5.5 form the score vs. Cost per Task Pareto frontier: GPT-6 Luna (max) scores 54.6% at $0.002 per task and DeepSeek V4.1 Flash (max) scores 60.9% at $0.005 per task. Claude Sonnet 5.5 (max with fallback) has an average Cost per Task of $0.08, ~16x more costly than DeepSeek V4.1 Flash for a ~1 point score increase ➤ Shorter answers trade completeness for communication quality: Each HealthBench rubric item is tagged with one of five axes. Completeness checks whether a reply includes all the important information, while Communication Quality checks whether it is clear, well structured and pitched at the right length and level of detail. GPT-6 Sol (max) writes the shortest answers (~1,000 characters) and has the highest raw Communication Quality score (81%), but the lowest Completeness score (34%). Claude Sonnet 5.5 (max with fallback) leads on Completeness (63%)