Song Han retweeted
🚀 Excited to share our latest work: Delta-Matching — Closing the Final Gap in Native 8-bit LLM Training!
Delta-Matching lets us train softmax attention with all seven core matmuls in FP8. In our experiments, it matches BF16/FP32 training loss and preserves overall downstream performance where evaluated.
Delta-Matching computes the correction from the same operands as the gradient. We derive the error introduced by stale delta and track selected channel gains growing before loss rises. In our GDN/GQA runs, larger models show faster gain growth; longer runs reveal degradation that short tests miss.
We tested
• Attention variants with and without QK normalization, RoPE and NoPE, and head dimensions 128 and 256.
• GDN/GQA, GDN/MLA, KDA/GQA, Mamba-2/GQA and pure-GQA model architectures.
The experiments span 569M to 5.29B parameters, 8K pretraining and lower-learning-rate context extension from 8K to 64K. We also tested AdamW and Muon-based training, different global batch sizes, and FP8 FFNs and attention projections.
🌐 Blog: jet-ai-projects.github.io/de…
📄 Paper: arxiv.org/abs/2609.37852
💻 Kernel Code: github.com/jet-ai-projects/d…
#EfficientAI #LLM #FP8 #DeepLearning #OpenSource
Song Han retweeted
🚀 Open-H3-Regenerate-2K: 3-Step Video Regeneration, Native Output up to 4K
📝 Blog: nvlabs.github.io/Sana/Open-H…
💻 Code: github.com/NVlabs/Sana/tree/…
🤗 HF Weights: huggingface.co/Efficient-Lar…
Song Han retweeted
After ICLR, we finally got some time to top first again on lean-eval leaderboard with updated Humanize workflows 🤗
> LeanEval is the last unsaturated benchmark in mathematical formalization.
> New Humanize flow learnt a lot of insights from OpenAI and Anthropic's blogs, and Axiom's tech report.
> More progress on "big name theorem" is coming :)
nitter.gq/IlinVasily29521/status…
LeanEval is the last unsaturated benchmark in mathematical formalization. Saturating it will require formalizing Fermat's Last Theorem. @axiommathai is currently in the lead.
Leaderboard: lean-lang.org/eval/legacy/
Song Han retweeted
Happy to share that our LongLive-Plug has been integrated into ComfyUI. Check it out at registry.comfy.org/publisher…
📄 Paper: arxiv.org/abs/2609.38154
💻 Code: github.com/NVlabs/LongLive/t…
🤗 Models: huggingface.co/collections/E…
Song Han retweeted
🚀 Introducing Robota-Embed — a unified Vision-Language-Action (VLA) embedding model that enables robots to retrieve and learn from past experiences.
Key Highlights:
🎯 Better Retrieval: 47.33% R@1, outperforming Gemini Embedding 2 by +22.09 points.
🤖 Better Robot Learning: Success rate improves from 47.0% → 56.3% on DROID-Sim and 86.4% → 92.8% on LIBERO.
⚡ Self-Evolving Agents: 24% → 68% task success with 43.7% fewer policy tokens on ManiSkill, compared to the direct GPT-6 Astra baseline.
The right experience. Better decisions. More efficient physical intelligence.
📄 Paper: arxiv.org/abs/2610.19495
🌐 Project: robota-ai.github.io/robota-e…
#Robotics #PhysicalAI #EmbodiedAI #VLA
Song Han retweeted
🚀 Introducing Lightning Weave — a new approach to composing capabilities from post-trained accuracy and efficiency specialists, creating reasoning models that are both more accurate and more efficient.
⚡ Results on Qwen3.5-4B:
HMMT: 59.2 → 64.0 (+4.8 points) with 10.7% fewer tokens
LiveCodeBench: 41.7 → 54.2 (+12.5 points) with 9.6% fewer tokens
🧠 Better reasoning. Fewer tokens.
📄 Website: jet-ai-projects.github.io/Li…
Song Han retweeted
Excited to release Long-WAM, our latest exploration in embodied AI 🤖 @AaronWeiHuang
Memory helps it track object motion, predict trajectories, and improve on long-horizon tasks as context grows. Our infra brings real-time control to edge devices.
Explore Long-WAM 👇
🌐 Project: nvlabs.github.io/LongLive/Lo…
📄 Paper: arxiv.org/abs/2610.10528
💻 Code: github.com/NVlabs/LongLive/t…
Five findings 🧵👇
1/ Longer context, better WAMs.
RoboCasa GR-1: 66.3% → 78.7% (2.4s → 19.2s). LIBERO-Long: 94.5% → 99.5%. Memory helps predict object motion.
2/ Pretraining matters.
LongLive2.0-Robot, pretrained on ~10,000 hours of robot/egocentric video, scales with context; the bidirectional baseline does not.
3/ Smoother async control.
RoboTwin 2.0: 94.2% async vs. 94.4% sync. Action jerk is ~3× lower than Fast-WAM and ~15× lower than LingBot-VA, without blending or prefix guidance.
4/ Real-time, edge-ready infra.
Optimized for RTX 5090, DGX Spark & AGX Thor. Just 107.4 ms per action chunk on RTX 5090, including IDM-style video prediction.
5/ Strong planners still need fast policies.
GPT-6 Astra plans; Long-WAM executes. RoboCasa365: 31.4% → 54.4%. Unseen composite tasks: 6.1% → 35.0%.
Song Han retweeted
🚀 Excited to introduce StreamRender-H3 — bringing real-time interactive worlds to life with diffusion models!
Built on Sol-Engine, we post-trained @MiniMax_AI H3 Ref2VA into a real-time streaming neural renderer.
🎮 Code drives the world. A game engine handles physics, object states, and player interactions, while H3 generates the visuals.
⚡ Real-time streaming: 2 denoising steps, 24 FPS, and ~273 ms per chunk on 8×GB200 GPUs.
🎨 One world, endless appearances: Change the reference image and prompt to render the same interactive world in entirely different styles.
🔥 Powered by streaming post-training, distillation, a fast causal H3 VAE, and full-stack inference acceleration.
Check out our demo and code! 👇
demo: nvlabs.github.io/Sana/Sol-En…
code: github.com/NVlabs/Sana/tree/…
Song Han retweeted
Excited to release Long-WAM, our new exploration in embodied AI 🤖！
With memory, Long-WAM precisely perceives how objects move, predicts their trajectories, and gains emergent scaling on long-horizon tasks; our infra runs it fast on edge devices.
nvlabs.github.io/LongLive/Lo…
Song Han retweeted
We’ve built a really strong team at @NunchuxAI , and we’re still hiring.
We’re looking for great people across product, R&D and GTM to join us for the next stage!
We're growing our SF team. Join us and build the frontier of multimodal inference at Nunchux AI.
Engineering
✦ Product Engineer
✦ ML Engineer, Visual Generative Models
✦ ML Engineer, Model Integrations
GTM
✦ Founding Marketer
✦ Founding GTM
nunchux.ai/careers
Song Han retweeted
Splash 1.3.0 is out, and it's about the wait before your local agent starts answering ⚡
If you switch projects and come back, SSD offloading brings the first token down from 19 seconds to 1.
New tasks start 1.5× faster, thanks to the Neural Engine. All on a 24 GB M6 Mac.
Song Han retweeted
Excited to share our latest NeurIPS 2026 work, Jet-Long — extending LLM context with no fine-tuning and ≤4% generation overhead, while delivering strong long-context accuracy! 🚀
🚀 Introducing Jet-Long — efficient long-context extension for LLMs with Dynamic Bifocal RoPE.
Extend Qwen3 from 32K → 128K, Qwen3-30B-A3B-Instruct from 256k → 1M context with:
• 🚫 No fine-tuning
• 📈 +4.79/+2.18/+2.03 pp on RULER (1.7B/4B/8B) over the strongest baseline
• 🧠 Best overall HELMET-RAG & lowest PG-19 perplexity
• ⚡ ≤4% generation overhead
Jet-Long dynamically adapts RoPE to the current sequence length while exactly preserving the base model within its pretrained context window.
Long context, essentially free at inference.
📄 Paper (NeurIPS 2026): arxiv.org/abs/2607.07740
💻 Github: github.com/jet-ai-projects/j…
Song Han retweeted
31× more video per GPU, at 1/15 the cost ⚡
That's @linewiseAI's video agent on Inco vs. their own pipeline.
The agent is thorough. Before it calls a moment idle, it checks what came before and after, reads the sensors, and zooms in. Lots of model calls, and we make them fast.
One week of our video data: 16,190 hours. A full-time worker would need 8 years to watch it.
Our agent-native infra analyzes it in a day, with inference on @inco_ai: 31× the throughput per GPU at 1/15 the cost per hour of video.
Song Han retweeted
🚀 Introducing LongLive-Plug: train one plug-and-play acceleration LoRA, then reuse it across 54 downstream video tasks—from world models and robotics to video editing—without re-distilling every model.
Open source, with support for any models based on MiniMax-H3 or Wan 5B/14B.
🔗 nvlabs.github.io/LongLive/Lo…
Song Han retweeted
A video world model can make a convincing clip and still get the physics wrong.
Our researchers just released Physis-Lang, an open self-evolving framework that adds physics reasoning to video captions. The captions explain why and how a scene unfolds. We use them to fine-tune world models and add that reasoning to prompts when generating video.
Adding physics reasoning to the prompt alone improved NVIDIA Cosmos 3’s PhyGenBench score by 5.62 points without retraining.
Paper and project: physis-intelligence.github.i…
Song Han retweeted
Can language teach video models physics?
Physis-Lang evolves a shared guideline to improve physical descriptions and video generation.
#1 & #2 by mean score on Physics-IQ Verified. +5.5 points over the previous leader.
elemmire1.github.io/physis-l…
Song Han retweeted
🚀 Kernel Design Agents to Optimize Kimi Delta Attention (upto 2.96x)
When the KDA (Kernel Design Agents) project was initiated, its name happened to coincide with another KDA (Kimi Delta Attention). More than once, people asked us , "Can we use KDA to write a version of KDA?" Today, right at a moment with a full moon good for shooting, we are thrilled to share the latest results of KDA(gent)-v0.6: using KDA(gent) to optimize KDA(ttn)!
KDA-v0.6 achieves this through
* Humanize2 Flow & LLM Collaboration Optimization: Introducing multi-round iterative evolution with Flame Chase, GPT-5.6-Sol, and Fable-5, paired with an automated Workspace cleanup mechanism to significantly broaden the operator search space.
* Multilingual Domain-Specific Skill Stack Support: Fully supporting CuTe-DSL, CUDA C++, as well as recent Agent-Native CAKE IR and TIRx. Combined with customized diagnostic tools (such as CuTe's dedicated IKET pipeline analyzer and TIRx's CPU-side numerical simulation and static checks), achieving deep performance bottleneck diagnostics.
* Self-Evolving Kernel Wiki: Building a dynamically updated knowledge base that automatically cleans up invalid/erroneous implementations, corrects classification tags, and streamlines search indexing, significantly improving the Agent's knowledge retrieval quality and code generation accuracy.
KDAgent ultimately optimized the KDAttn kernel to 2.96x of FlashKDA, and we face a lot of unexpected hacks during the optimization. More details shared in our blogs nvlabs.github.io/kda/blog/20…
Release results of KDA for KDA: github.com/NVlabs/kda/tree/2…
Community & Production Deployment:
All kernels are open-sourced for community verification! For deploying NVIDIA Agentic CUDA acceleration in production environments, we recommend using the production-level version rigorously verified by FlashInfer.
By the Team: Dongyun Zou, Yixin Dong, Hongyi Jin, Junxian Guo, Yahui Cui, Avery Huang, Zihao Ye, Junru Shao, Changye Li, Zijian Zhang, Sihao Liu, Song Bian, Ligeng Zhu.
Welcome to submit to MLSys 2027. I am thrilled to serve as a PC Co-Chair this year alongside Rashmi Vinayak. We are actively looking for cutting-edge research and industry track papers at the intersection of Machine Learning and Systems. mlsys.org/Conferences/2027/C…
Song Han retweeted
Qwen3.8-27B at 144 tok/s on an M5 Max MacBook Pro ⚡
Meet Inco Splash: our open-source inference engine, built around the model and around Apple silicon.
Up to 3× the decode speed of Ollama, 2× oMLX, and almost 4× when an agent fans out into sub-agents.
Song Han retweeted
🚀 We’ve released the full SoL-Pi report!
It includes the complete methodology, extensive experiments, scaling studies, and detailed analysis behind SoL-Pi.
Would love your feedback — and an upvote if you find it useful! 🙌
huggingface.co/papers/2609.2…
