Anish retweeted
Come hang out! We'll be sharing some of our latest work on serving agents and RL at scale!
Join @baseten, @nvidia Dynamo, and SGLang on 9/10 for an evening dedicated to the infrastructure behind reinforcement learning post-training!
We will present the latest on SGLang & Miles. Miles is an RL training framework using SGLang as the rollout inference engine to make RL post-training fast and reliable. Come meet us in person!
🕐 Thursday, September 10, 6:00–9:00 PM
👉 Register now
luma.com/BaseDynSGL
Anish retweeted
When an LLM engine crashes, a cold restart can mean minutes of lost capacity.
Shadow engine recovery, a new preview feature in NVIDIA Dynamo, keeps a standby engine warmed up and ready to take over.
In our GLM-5.2 test, it restored capacity in 7.3 seconds, nearly 39x faster than a cold restart.
Read all about it: nvda.ws/45OhcA0
Anish retweeted
vLLM's model loader and KV connector both have an @Azure Blob path now: weights in, KV out. @Microsoft and @NVIDIAAI shipped a recipe for each.
Nothing serves until weights land in HBM. Dynamo ModelExpress plugs into the loader, up to 7.3x faster than the default on H100/A100.
Anish retweeted
We're co-hosting a @vllm_project x @nvidia Dynamo Meetup during vLLM Conference week in San Francisco ⚡
Tech talks on serving LLMs efficiently at scale + food, drinks, and the inference community all in one room
📅 Aug 24, 6–9pm PT
Space is limited — RSVP: luma.com/r8o604o0
Anish retweeted
Cold starts used to be why you’d avoid scale-to-zero for GPU inference. Not anymore, we got them down to seconds.
Built with the @nvidia Dynamo team using GPU memory checkpointing. Our engineer @david_lc98 wrote up what it really takes to run in production, gotchas included: vist.ly/5deq8
Hyped to see Dynamo being used in this RL run!! Congrats Cognition team
Anish retweeted
Baseten are cooking!
Baseten have been working with us on Dynamo since 0.1, and have been nothing but incredible partners. Really excited to see the impact that Dynamo brought to this SOTA endpoint (2x TPS! For free!)
Anish retweeted
Introducing Dynamo Snapshot, our approach for fast startup for inference workloads on Kubernetes, which reduces startup time from minutes to under 5 seconds.
In production inference deployments demand fluctuates over time. Cold-starting inference workloads can take minutes, leaving idle GPUs that generate no tokens and serve no requests.
Snapshot leverages GMS to enable concurrent weight restoration over a high-speed interconnect, while using Linux native AIO and parallel memfd restoration to accelerate CRIU restore performance.
Anish retweeted
Elastic parallelism for wideEP deployments is critical for operating large-scale inference systems efficiently. What started as an idea at a Dynamo after-party nearly a year ago is now finally available to everyone.
Congratulations @nvidia dynamo nixl and @vllm_project team!
A vLLM MoE deployment's DP/EP topology used to be locked in at launch — scaling or swapping config meant a full restart, in-flight traffic dropped. Elastic Expert Parallelism changes that. One API call resizes a live deployment:
curl -X POST localhost:8000/scale_elastic_ep \
-d '{"new_data_parallel_size": 16}'
Under the hood: standby comm groups span the target topology, EPLB redistributes experts across the new EP group, and weights are transferred directly between GPUs over NVIDIA NVLink/RDMA. The same runtime reconfiguration path is what fault-tolerant serving needs: evict failed ranks, redistribute their experts, bring replacements back, no restart.
Thanks to @NVIDIAAI, Sky Computing, @anyscalecompute, @RedHat_AI, and the community.
📖 vllm.ai/blog/2026-05-14-elas…
dstack team stays cooking! Being able to deploy Dynamo in any environment is super valuable and dstack gets rid of any friction🫡🤝
dstack 0.20.20 is out 🚀
New in services: Prefill-Decode disaggregated inference with @NVIDIA Dynamo, with workers running @lmsysorg SGLang, @vllm_project, or TensorRT-LLM.
Also:
⚡️ Easier provisioning and management of @awscloud clusters that use EFA
⚡️ Sharing gateways across projects
github.com/dstackai/dstack/r…
Anish retweeted
Jensen Huang: It's easier to fall in love with what you do than to find what you love
“A lot of people say, ‘Find something you love.’ I don’t know about that. I guess I’ve fallen in love with many things that I do. I loved it when I was a dishwasher. I loved it when I was a busboy. I loved it when I was delivering papers. I loved it when I was waiting tables.”
Jensen continues:
“I’ve loved every single job that I’ve ever had, and I’ve loved every single day at Nvidia that I’ve ever had. I just learned to love what I’m doing. It’s hard to find something that you love, but it’s easier to fall in love with what you’re doing. And once you fall in love with what you’re doing because you desperately want to do a good job at it, it’s easier to do it well and work hard.”
Source: @NorgesBank (Nov 2023)
Powered by Dynamo!
We are excited to have @baseten as a day 0 launch partner for Kimi K2.6!
Their inference stack brings KV-aware routing, NVFP4 on Blackwell, multi-modal hierarchical caching, and prefill-decode disaggregation, so K2.6 runs the way it's meant to in production.
Try it out at: baseten.co/library/kimi-k26
