Pinned Tweet
Today at PyTorch Conference 2026, we announced llm-d 1.0! ๐
Distributed inference is too complex to solve behind closed doors.
17 months in, llm-d is now a @CloudNativeFdn project powering fleets worldwide.
Hereโs whatโs inside 1.0 ๐งต๐
During the PyTorch Conf keynote, Abdullah Gharaibeh gave a live demo of a couple of llm-d features.
Watch live as llm-d routing and flow control provides real-time benefits for the user at scale.
youtube.com/watch?v=F-3YgAYyโฆ
llm-d retweeted
Live from #PyTorchCon 2026, Carlos Costa (@IBM, Abdullah Gharaibeh (@googlecloud), and Stephen Watt (@RedHat) present โOpen Source Inference at Production Scale,โ with a live @_llm_d_ demo underway.
๐ Stable v1 API:
A versioned configuration surface and hardened Endpoint Picker designed to give platform teams a dependable foundation for years to come.
No breaking surprisesโjust reliable, production-ready infrastructure.
To everyone who benchmarked a model, reviewed a PR, or ran early builds: thank you. You proved the future of inference is open.
๐ Read the 1.0 deep dive: llm-d.ai/blog/llm-d-v1.0-staโฆ
โญ Star on GitHub: github.com/llm-d/llm-d
Great open-source communities push enterprise hardware further. ๐ค
New work from IBM Research & Red Hat on llm-d:
โข 753B open model on 544 NVIDIA H100 GPUs
โข 5โ10x lower cost per token vs commercial APIs
โข Serves 1000s of concurrent agents
Blog: research.ibm.com/blog/runninโฆ
llm-d retweeted
This is awesome. Claude Code and Codex running on open models with @vllm_project, no OpenAI API in the loop.
Ask Qwen3.8 about a model released after its training cutoff and it has no idea. Give it server-side web search through the Agentic API and it finds the answer, with citations.
The Agentic API keeps the stateful parts, turn history and tool execution, outside vLLM so the engine stays a stateless inference server. Built in Rust, MVP just landed.
From @franciscojarceo (@RedHat_AI) and @Rxday000 (@EmbeddedLLM) ๐
llm-d retweeted
Every vLLM Office Hours runs with a live chat going the whole time. People ask real questions and get real answers, often from another attendee before a maintainer can get to it.
Three from this week ๐งต
1. Someone asked about the decode context parallelism tradeoff.
llm-d retweeted
vLLM Office Hours #56 recording is up:
- What's new in vLLM 0.27
- Running Codex and Claude Code CLIs on open models
- Speculators v0.6 and v0.7 update
- GuideLLM v0.7.0: benchmarking reasoning, tool calling, and agentic traces
Watch + see slides: youtu.be/dQAByb7N5QQ
llm-d retweeted
AI keeps evolving: new models, hardware, & deployment patterns are reshaping how orgs build intelligent apps.
@_llm_d_ shows what resilient AI infrastructure can look like.
Spoiler: itโs not owned by one company. Itโs strengthened by community. red.ht/4zuuMpM
llm-d retweeted
๐ฅ SPEAKER ANNOUNCEMENT | @vllm_project Bangkok Day
(Thailand's largest event dedicated to vLLM)
It is a real honor to announce this one.
As the organizer, I'm proud to introduce another speaker I'm genuinely excited about:
Jing Wen Ng
Specialist Solutions Architect, AI, Asia Pacific @RedHat_AI
And here's what makes it special: Jing Wen is flying to Bangkok to join us onsite, not virtually.
๐๏ธ Session: Scaling Distributed Inference with @_llm_d_
Red Hat is one of the teams behind llm-d, the project that takes vLLM beyond a single server and makes it scale across GPUs and nodes in production.
What you'll take home:
- What llm-d actually is, and why distributed inference is the next step once one vLLM instance stops being enough
- KV-cache aware routing and prefill/decode disaggregation, and what they do to your serving performance
- Running it for real on OpenShift AI, from single node to multi node deployments
- Hard-won lessons from enterprise customers across APAC in FSI, telco, healthcare and the public sector on what moves AI from experimentation to production
Why this session matters for Thailand:
With registrations now past 500, the two topics our community asked for most were KV-Cache and real production use cases.
This session sits right on top of both, delivered by someone who does this with enterprise customers across the region every week.
๐
Sunday, 6 September 2026, 2:00 PM to 7:00 PM
๐ JAS Green Village Ramkhamhaeng, 3rd floor (JAI Arena), Bangkok
๐๏ธ Free to attend. All sessions delivered in English.
Huge thanks to our partners @inferact , @EmbeddedLLM, @RedHat_AI, @tensormesh , and @agicafet for making the largest AI Inference event in Thailand happen.
"One vLLM Always vLLM"
#vLLMBangkok #vLLM #llmd #RedHat #AIInfrastructure #LLMServing #KVCache #Inference #AGICAFET
llm-d retweeted
๐ข ๐ง๐ต๐ฒ ๐ฆ๐๐ฎ๐๐ฒ ๐ผ๐ณ ๐ ๐ผ๐ฑ๐ฒ๐น ๐ฆ๐ฒ๐ฟ๐๐ถ๐ป๐ด ๐๐ผ๐บ๐บ๐๐ป๐ถ๐๐ถ๐ฒ๐: ๐๐๐ด๐๐๐ ๐๐ฑ๐ถ๐๐ถ๐ผ๐ป ๐ถ๐ ๐ผ๐๐!
Our goal with this newsletter is to give a clear, community-driven view of whatโs happening across the model serving ecosystem that our teams at @RedHat_AI are contributing to, including updates from projects like @vllm_project, KServe, @_llm_d_, @kubernetesio, and beyond.
๐ Check out the August newsletter here: inferenceops.substack.com/p/โฆ
๐ Subscribe to get future issues in your inbox: inferenceops.substack.com/
๐ Thanks to everyone who subscribed so far!
Kudos to all contributors to this edition! David Simmons, Francisco Arceo, Jooho Lee, Yuan Tang, Wentao Ye, Sasa Zelenovic, Vivek Karunai Kiri Ragavan
llm-d retweeted
Red Hat AI blog series part 3 is out now! This part focuses on how Red Hat's AI Inference on managed Kubernetes uses llm-d within the inference platform.
developers.redhat.com/articlโฆ
@RedHat_AI @_llm_d_
youtu.be/RbNfxFU8Qh8?si=x1sHโฆ
llm-d retweeted
Open source is helping change the economics of #AI.
Projects like @_llm_d_ are bringing open, collaborative innovation to LLM inference, helping organizations get more from the infrastructure they already have.
Learn how: red.ht/4xh75zF
#OpenSource #AI #GenerativeAI
AI is entering the era of large-scale, distributed, agentic systems. Applications coordinate multiple models, tools, and services, process millions of requests, and demand enormous amounts of computer capacity. buff.ly/pw23Cuo