@NVIDIA Sr. Research Scientist | UIUC PhD All opinions and tweets are personal. Tweets about AI Inference, CUDA and GPU systems.
Joined November 2008
- Tweets881
- Following665
- Followers4.5K
- Likes475
Agentic AI is changing how we think about LLM inference infrastructure.
With @nvidia Dynamo, we’re moving beyond request-level optimization toward session-aware inference, preserving context, maximizing KV reuse, and improving efficiency across multi-turn agentic workloads.
Agentic workloads generate long, expanding context across repeated model calls, tool execution, and parallel subagents. Much of this context remains stored in the KV cache while an agent waits on external tools or pending operations.
Efficient serving relies on preserving and reusing this PyTorch tensor state while maintaining sub-second latency targets across concurrent sessions.
Read our latest blog from @nvidia to learn how NVIDIA Dynamo integrates its router, inference engine, and KV-cache manager to deliver seamless end-to-end serving for agentic workloads: bit.ly/4emlt21
Great collaboration with the @PyTorch team!
Learn more below: pytorch.org/blog/session-awa…
Congratulations to @PrimeIntellect team!
We have been helping @PrimeIntellect to run on @nvidia Dynamo for RL and several other usecases!
Learn more.
Congratulations @CoreWeave on releasing Forge.
@CoreWeave uses @nvidia Dynamo to perform large scale RL rollouts at scale! This is an extremely complex problem and glad we could help!
Congrats @CoreWeave on RL Rollouts!
RL post-training involves a lot of back and forth: train the model, generate responses, then train again. Inference workers need to load the updated model weights each time. As models get bigger, that can leave GPUs waiting.
CoreWeave’s new service uses ModelExpress and Router in NVIDIA Dynamo to speed up those reloads with minimal downtime.
Working with us and @youdotcom, CoreWeave achieved 15× faster model reloads compared with its baseline while post-training Nemotron 3.5 Lightning.
Check out their blog below for details
Excited to share our latest work with Cedana: serving a portfolio of frontier coding models on a B200 node.
TLDR: @nvidia Dynamo handles inference, NeMo Switchyard intelligently routes & escalates across models, while Cedana enables stateful model switching in ~1 min! (1/2)
More enterprises are running a portfolio of models. Easy tasks go to smaller, cheaper models, while frontier models take the hard tail.
Hosting that portfolio is a GPU utilization problem. The models don't fit on a single node together, and 9-34 min cold starts make switching between them impractical.
Excited to show our work with NVIDIA's Dynamo team. We show how to serve a portfolio of frontier coding models on a single 8x B200 node, switching models in ~1 minute with in-flight sessions intact.
cedana.com/blog/coding-model…
Vikram retweeted
What does “fungible” mean for AI infrastructure? The flexibility to run different workloads: every type of AI, every phase of AI and even workloads that aren’t AI at all.
One platform, built for maximum “productivity”, “durable” years after deployment, and “fungible” for every model and workload.
Read the blog to learn more: nvda.ws/4hphGmQ
Dear PhD students!
Application for NVIDIA Graduate Fellows is now open!
Apply!
We’re looking for our next cohort of graduate fellows.
Our 2027–2028 Graduate Fellowship Program is open to Ph.D. students worldwide with awards up to $60,000, plus mentorship and technical support.
Apply by October 30: nvda.ws/4rGKDhv
@Pinterest is using @nvidia Dynamo with vLLM and NVIDIA Blackwell GPUs to serve its vision-language models enabling them to process 25× more visual context per request while avoiding repeated image processing.
(1/2)
We’re expanding our collaboration with @Pinterest to scale AI that understands images and words together.
Pinterest serves 640M users monthly and every search, every recommendation, every visual moment has to be instant.
With NVIDIA Blackwell and Dynamo, Pinterest's AI responds 85x faster, with 7.3x less wait time and 25x more visual context per request.
Learn more: nvda.ws/3Vd8I38
The easiest way to become expert in inference is contributing to Open source frameworks like @vllm_project @sgl_project @NVIDIAAIInfra Dynamo.
We’re always on the lookout for excellent developers who make impactful contributions to Dynamo through high-quality PRs.
Multimodal models are becoming increasingly important for computer use and everyday AI tasks.
A key to serving these models efficiently is encoder and PD disaggregation.
Read more about how we are enabling this in a performant way using @nvidia Dynamo!
Multimodal models put different demands on vision encoding, prefill and decoding.
Separating vision encoding from the other stages can reduce resource contention and help models respond faster, but only for the right workloads.
See how EPD disaggregation works, when it helps and what to consider before using it: nvda.ws/4xUFUv2
ALT A performance comparison chart of Colocated and Disaggregated Encoders with percentage changes.
Vikram retweeted
GPT-6 Astra, trained on ~100K+ NVIDIA Grace Blackwell NVLink72. From ChatGPT to o1 to Astra in 4 years.
AGI has arrived. Congratulations @OpenAI team.
400K GPUs coming online next.
Excellent to see more adoption of NIXL! We created a simplist way to transfer data between any source and destination for exactly this reason.
Fast but simple to use!
Our RL stack now supports NIXL weight transfer, reducing trainer-to-inference transfer time 9x compared with NCCL: from 86 seconds down to single-digit seconds for an 800B-parameter model, and even <4 seconds in our experiments.
For prime-rl users, this means over 25% more throughput end-to-end compared with our previous speed. It also clears the way for fault-tolerant, elastic inference scaling that NCCL's rigid process groups made difficult.
Vikram retweeted
Exciting day for NVIDIA and @huggingface.
Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty. They allow every developer, startup, university, industry and country to build with, customize and benefit from AI.
Thank you @ClementDelangue for coming to me.
NVIDIA is going to be a great home for Hugging Face, its community and the future of open models. 🤗
blogs.nvidia.com/blog/nvidia…
Vikram retweeted
(1/7) Last week FastVideo released FastH3. Today we bring it local: @NVIDIA DGX Spark 💚 and @Apple Silicon 🍎through MLX. 8x speedup!
FastH3 now supported on NVIDIA DGX Spark and Apple Silicon! over base @MiniMax_AI H3 for local generation!
This 15 seconds 1344×768 clip was generated on two NVIDIA DGX Sparks. Fully local. @NVIDIARTXSpark @NVIDIAAI
Vikram retweeted
Hanging out next Thursday 9/10 with my friends from NVIDIA Dynamo and SGLang.
We'll have tech talks on inference optimization at every layer of the stack.
Join us!
luma.com/basedynsgl
Vikram retweeted
Already running an inference engine? So where does NVIDIA Dynamo fit in?
In five minutes, we break down how Dynamo sits around engines like @sgl_project, @vllm_project and TensorRT-LLM to scale inference across GPUs and nodes.
Full video in the comments 🔽
Today, we are releasing Shadow engine recovery, a new feature in @NVIDIAAIInfra Dynamo that lets the engine restart up to 39x faster in k8s by using a warmed-up engine.
Benchmarking on GLM-5.2 with @nvidia B200 nodes showed that Shadow engine recovery reduced
(1/2)
When an LLM engine crashes, a cold restart can mean minutes of lost capacity.
Shadow engine recovery, a new preview feature in NVIDIA Dynamo, keeps a standby engine warmed up and ready to take over.
In our GLM-5.2 test, it restored capacity in 7.3 seconds, nearly 39x faster than a cold restart.
Read all about it: nvda.ws/45OhcA0
failover time from 283s (cold restart) to 7.3s, dramatically improving TTFT, decode rate, and SLA adherence after process failure.
All of these benefits are enabled using the GPU memory service discussed in our earlier blog: developer.nvidia.com/blog/nv…
Read more!
(2/2)
