Building LLMKube: K8s inference across NVIDIA, AMD, and Apple Silicon. Data stays yours. AI/ML Ops Director @ Ashley. Ham Radio: N7CPM
Gig Harbor, WA
Joined January 2018
- Tweets858
- Following586
- Followers319
- Likes2.8K
Pinned Tweet
Last month, Pluto Security reported a Hugging Face campaign that shipped crypto miners as small Qwen2 test checkpoints. No poisoned weights, no pickle trick: just Python that runs under trust_remote_code. The second wave's flagship scanned clean on the hub.
Chris Maher retweeted
Be careful what you wish for, America…
“The rest of the world is ready and willing” to build data centers - ARM CEO
Deck's done and I'm pumped! At #KubeCon I'm sharing how I got Kubernetes running LLM inference on Apple Silicon: what worked, what broke (a kernel panic included), and patterns you can take home to your own cluster. See you there!
Apache-2.0, every check. Go, air-gap buildable. Try it in your browser, nothing uploaded: socair.ai/try
github.com/defilantech/socai…
Source, Pluto Security's write-up: pluto.security/blog/hugging-…
I pointed DeepSeek Harness's Creator Mode at DeepSeek V4.1 Flash on my three DGX Sparks.
423 steps later it had built its own plugin, then said it never looked at the page. The page was broken.
Second pass: 7 minutes, one parenthesis.
llmkube.com/blog/deepseek-ha…
LLMKube is almost a year old (first public commit: Nov 17, 2025), and 0.10.16 is out today.
Huge thanks to the 18 contributors who've landed code alongside me. 121 releases and 252 stars later, this is way more than a solo project.
github.com/defilantech/LLMKu…
NVIDIA just released an NVFP4 quant of DeepSeek V4.1 Flash 1017. It's 400 GiB versus 475 for the original, so I tested whether it should replace what our 3x DGX Spark (GB10) ring runs.
Short answer: not on GB10.
All of the 76 GiB saving comes from the Engram tables, which my ring already reads from NVMe. The experts that sit in GPU memory get bigger, about 5.3 GiB more per GPU, because NVFP4 carries more scale bytes than MXFP4.
I benchmarked the MoE kernels on one GB10 with V4.1 shapes, every b12x recipe for both formats:
- Best NVFP4 recipe: 5-7% slower than best MXFP4 at every batch size, from 1 token up to 4096-token prefill chunks
- FP4 activations (W4A4) don't help, because the MoE is limited by weight-read bandwidth even at 4096 tokens
- The recipe that keeps today's activation precision is 29-48% slower
So the ring stays on the native MXFP4 weights. On hardware built for big-batch FP4, like the GB300s NVIDIA tested on, it's a different story.
Kernel-level numbers on synthetic weights, not an end-to-end run. The gap tracks the bits per weight (4.5 vs 4.25), so I don't expect a full boot to flip it.
My Kubernetes operator ignored a spec change for 7 minutes on a managed AKS cluster.
Pod Running. Probes green. Zero errors.
A manual restart, and ~10 seconds later the change landed.
The liveness probe was the Kubebuilder default. It only proves the HTTP server is up.
That check is controller-runtime's healthz Ping. It returns OK whenever the process can answer HTTP. An operator whose watches have stopped delivering events still answers HTTP. So Kubernetes saw a healthy pod while nothing was being reconciled.
I had seen the same signature twice in my lab and blamed the cluster. Seeing it on a managed control plane is what made me fix it in the operator.
The fix in LLMKube 0.10.4 is a write the operator has to see come back:
• every 60s the leader updates a Lease
• it has to see that update arrive through its own informer
• 5 minutes unseen, liveness fails and the pod restarts
• liveness only, not readiness, so the webhooks keep their endpoints
What it does not catch: informers are per resource type. If one type's informer dies while the Lease informer is fine, the canary stays green. There is an open issue to alert on that. And it has not caught a real stall yet, so treat it as a backstop, not a proven fix.
The signal that was there every time: metadata.generation ahead of observedGeneration. Check it on whatever your operator reconciles. For LLMKube:
kubectl get inferenceservices -A -o custom-columns='NAME:.metadata.name,GEN:.metadata.generation,OBSERVED:.status.conditions[?(@.type=="Available")].observedGeneration'
OBSERVED stuck behind GEN for minutes means nothing is reconciling.
The full write-up, with the timeline, the design choices, what the canary misses, and a checklist for anyone who writes an operator:
llmkube.com/blog/operator-gr…
The full write-up, with the timeline, the design choices, what the canary misses, and a checklist for anyone who writes an operator:
llmkube.com/blog/operator-gr…
This comment totally made my day! As LLMKube turns a year old next month, comments like this from people using this tool makes it all worth it! Join in and help make LLMKube your own! llmkube.com
