Accelerating AI innovation with open platforms and community. The future of AI is open.
Joined May 2018
- Tweets2.7K
- Following2.1K
- Followers12.8K
- Likes1.8K
Pinned Tweet
Everything you need to start self-hosting an open LLM.
Run it on your own hardware. No API keys. No per-token bill. Nothing leaves your machine.
The full path with @vllm_project: batch inference in Python, an OpenAI-compatible API server in one command, and quantized models that cut an 8B from ~16GB of weights to a quarter of that while keeping 98-100% accuracy.
Walkthrough by @cedricclyburn.
Red Hat AI retweeted
During the PyTorch Conf keynote, Abdullah Gharaibeh gave a live demo of a couple of llm-d features.
Watch live as llm-d routing and flow control provides real-time benefits for the user at scale.
youtube.com/watch?v=F-3YgAYy…
Red Hat AI retweeted
🚨🚨🚨 vLLM is all you need to reproduce DeepSeek’s performance.
With Wide-EP and EPLB, vLLM can match DeepInfer’s decode performance within 6% and generate $39/gpu/hr in revenue at DeepSeek’s off-peak pricing, at 200 P90 TPS on the AgentX workload.
Specifically:
• Under DeepSeek’s decode configuration (DeepSeek-V4.1-Flash, 131K context, nextn=3), vLLM on GB200 achieves throughput within 6% of DeepInfer’s published results across batch sizes 8–256.
• On AgentX, vLLM pushes the Pareto frontier further on GB300, reaching up to 1M tokens/s/GPU and generating $38.96/gpu/hr in revenue at 196 P90 TPS, based on DeepSeek’s off-peak API pricing.
• Also, vLLM extended low-latency kernel support for DeepSeek-V4.1-Flash, achieving up to 826 TPS per user on AgentX.
Hola, Madrid. vLLM and llm-d are coming to town: Red Hat and @nvidia are hosting an evening on open source inference, with @vllm_project, LLM Compressor, vLLM-omni, safe agents, and @_llm_d_ on Kubernetes.
12 November | MOA Institute | Register:
luma.com/gl1d2loa
Red Hat AI retweeted
Red Hat researchers, in collaboration with Purdue, developed VEX-Bench—a benchmark for assessing large language models (LLMs) in software supply chain vulnerability triage. Traditional tools often generate up to 70% false alerts. Discover the findings: red.ht/4debJqd #EMNLP2026
Amazing ~4-minute walkthrough from @LegareKerrison of how @NVIDIAAI's OpenShell sandboxes autonomous agents using declarative security policies, policy as config, not code buried in the app. youtube.com/watch?v=sXXYJQ-3…
Red Hat AI retweeted
Today at PyTorch Conference 2026, we announced llm-d 1.0! 🎉
Distributed inference is too complex to solve behind closed doors.
17 months in, llm-d is now a @CloudNativeFdn project powering fleets worldwide.
Here’s what’s inside 1.0 🧵👇
When does vLLM beat llama.cpp? What happens when a model outgrows one GPU? Can you trust an agent benchmark? Where do MCP and agent skills fit?
Red Hat AI folks have been answering these questions on @IBM's Technology channel on YouTube.
A 🧵 of deep dives for your watch list:
Want to become an AI engineer? @cedricclyburn covers the foundations: Python, APIs, RAG, embeddings, agents, observability, and deployment.
youtube.com/watch?v=XN3xNJvW…
That's the watch list. Our experts show up regularly on IBM Technology with explainers on inference, agents, and open-source tooling. Full channel:
youtube.com/@IBMTechnology/v…
Red Hat AI retweeted
▲ Google, Red Hat and IBM Devs just released llm-d 1.0, turning a fleet of vLLM and SGLang servers into one coordinated inference pool on Kubernetes.
The key idea is cache-aware routing. The router tracks which replica already holds each agent's prefix and sends the next step there. When GPU or TPU memory fills, KV cache spills to host RAM, SSD or Managed Lustre instead of being evicted.
Google reports Gemini's prefix cache hit rate doubled from 35% to 70%, and a 17,000-node GKE benchmark scaled throughput 2.9x. It is a trade-off though: every number is self-reported, and RL time-slicing still leaves roughly 30% of accelerator time idle.
Apache 2.0, CNCF Sandbox, runs on NVIDIA, AMD, Intel GPUs and Google TPUs......
Full analysis: marktechpost.com/2026/10/20/…
Repo: github.com/llm-d/llm-d
Docs: llm-d.ai
Technical details: cloud.google.com/blog/topics…
@GoogleAI @googleaidevs @RedHat @RedHat_AI @IBM @IBMData @IBMResearch
Meta's Muse Spark Open, 816B, runs on Red Hat AI Inference Server (preview). The detail worth knowing: its DSpark speculator drafts exactly 7 tokens, and there's a real reason it stops there. Suffix decay, explained:
Reflection used @vllm_project's LLM Compressor to quantize Beam 1.0, their 501B open MoE, into an FP8 checkpoint. Always good to see a major open model reach for it to ship production weights. That's what building it in the open is for. Nicely done, @reflection_ai:
huggingface.co/reflection/Be…
Red teaming a model means attacking it on purpose: prompt injections, jailbreaks, unsafe outputs, before real users or attackers find them. Your chatbot works great in the demo. Red teaming is how you find out where it breaks.
A beginner path to run your first scan, a 🧵:
The path walks the full loop: run a probe, read the model's failure rate, decide if it needs guardrails, then design and apply targeted ones and scan again until it clears your safety bar. It's the real red-team workflow, scaled down to one model for learning purposes.
Best part: nothing to install, no hardware of your own. All you need is a free @RedHat Developer Sandbox account. You run it in JupyterLab on OpenShift AI, scanning Granite 3.1B.
Start here and let us know what you are thinking:
developers.redhat.com/learn/…
