Engineering Leader for Planetary Scale Inference with NVIDIA Dynamo
San Francisco
Joined March 2021
- Tweets404
- Following128
- Followers1.7K
- Likes792
We're looking to hire DevRels for Dynamo!
Awesome parts about the job:
- Getting to be a community face for one of the most important domains in the world (Inference)
- High visibility
- Extremely motivated and fast-moving team
- Working in open source
- Lots of compute
/more
Super cool to see the community interest in DeepInfer support for Dynamo!
github.com/ai-dynamo/dynamo/…
github.com/ai-dynamo/dynamo/…
We have a PR with disagg support we hope to drop tomorrow, so if you'd like to collaborate, join us on slack!
ai-dynamo.slack.com/
NVM pulled it up to tonight:
github.com/ai-dynamo/dynamo/…
For those that want to dig deeper than the repo, read the Zhihu post:
zhuanlan.zhihu.com/p/2095560…
🚀 Introducing DeepInfer, our in-house inference framework, now fully open source!
Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.
🔗 github.com/deepseek-ai/DeepI…
1/4
me: Can you complete the port to WebGL?
Claude: I’ll start the weeklong GPU prototype, are you ok with that?
me: Are you overestimating with a week?
Claude: You’re absolutely right, I’m speaking about agent days. I’ll start now!
Claude finished in 300 agent minutes 😭
Kyle Kranen retweeted
Qwen3.8 Flash Next on Vera Rubin with NVIDIA Dynamo + SGLang:
Up to 6.6× throughput per chip and up to 76% lower modeled cost per total token vs. GB300, at matched per-user speed.
Thanks @NVIDIAAI for the partnership and early access, and @SemiAnalysis_ for the results!
ALERT🚨: NVIDIA SGLANG INFERENCE RUBIN IS 4.23x BETTER 🚀 PERF PER DOLLAR THAN EVEN GB300 NVL72. This is on the widely used production LLM engine called @sgl_project. (1/2)🧵
As I’m seeing them used a lot more, a reminder that stats like OpenRouter share, cache hit rate, latency, etc can easily be gamed for marketing.
This isn’t OpenRouter’s fault.
More below.
Unless you know the exact details of the deployment that backs an endpoint (HW, fleet size, IB connectivity, exact config, concurrency, etc), it’s impossible to determine the economics of an inference provider.
OpenRouter only sees observed traffic, not the deployment itself.
Kyle Kranen retweeted
Excited to chat with @NVIDIAAI this Thursday about how we are using Dynamo to scale VLM serving for computer-use agents at H !
Looking forward to the discussion and live Q&A! 🚀
Live with @NVIDIAAI this Thursday: how we serve vision-language models for computer-use agents.
Thu Oct 8 · 1PM CEST / 7AM ET
With @AvshalomM (inference team lead) and @EmrickSini (research engineer). Live Q&A.
It’s been an absolute blast working with Prime Intellect on, among other things, an incredibly fast and efficient inference service!
Check out the Prime blog on how they built it, including how they use Dynamo to maximize their performance!
Super excited to share something we’ve been working on:
Coreweave Forge’s RL rollouts run on Dynamo!
We’ve been doing incredible work to make Dynamo a SOTA rollout backend! Check out the perf numbers in the blog post!
Congrats @CoreWeave on RL Rollouts!
RL post-training involves a lot of back and forth: train the model, generate responses, then train again. Inference workers need to load the updated model weights each time. As models get bigger, that can leave GPUs waiting.
CoreWeave’s new service uses ModelExpress and Router in NVIDIA Dynamo to speed up those reloads with minimal downtime.
Working with us and @youdotcom, CoreWeave achieved 15× faster model reloads compared with its baseline while post-training Nemotron 3.5 Lightning.
Check out their blog below for details
Kyle Kranen retweeted
(9/13) UniServe also runs as an experimental backend for NVIDIA Dynamo: uniserve-dynamo-worker embeds UniServe's Rust scheduler and engine in a Dynamo worker, behind Dynamo's OpenAI-style /v1/videos frontend.
Selecting a configuration for your LLM is a critical step in the LLM serving process. Really enjoyed the methodology proposed in this blog!
Awesome work by the VLLM team, and thanks for the great resource!
vllm.ai/blog/2026-09-21-qwen…
For all UC Berkeley bears interested in distributed inference, my team (specifically @flowpow123) are at Cal in the Soda breezeway recruiting until 4 PM!
Other teams are there recruiting too!
Come say hi!