Building better local LLM inference on consumer GPUs. Creator of BeeLlama.cpp (1.1k+ stars on GitHub). Benchmarks, quantization, agents.
Belgrade
Joined June 2012
- Tweets550
- Following49
- Followers367
- Likes55
Pinned Tweet
Currently my main project is BeeLlama, a fork of llama.cpp with more than 900 stars on GitHub as of now.
It started very personally. I wanted to run Qwen 3.6 27B on my single RTX 3090, natively on Windows, and run it fast. And what can make it fast? Right, speculative decoding. But at that point, llama.cpp supported neither MTP nor DFlash.
So this May I forked it and made DFlash work there, with more and more tok/s gains in every subsequent release. It also shipped TurboQuant/TCQ for extreme KV cache quantization.
Since then, upstream llama.cpp has implemented a full set of speculative decoding options, and TurboQuant proved outdated in my own benchmarks. So BeeLlama moved on from those features and evolved.
Today, as of v0.4.x, its main additions are KVarN (variance-normalized KV cache quantization by Huawei) and KV Cache Precision Tail (storing recent tokens in full precision while quantizing the rest).
The feature set changes, but the idea stays the same. Local LLMs should be available to as many people as possible, not just owners of 4×3090 setups and entire fleets of DGX Sparks.
Quantization is a compromise, but the alternative is not being able to run the model at all. So the answer is to make quantization more efficient and minimize the losses, allowing people to experience local LLMs in the best shape possible on any half-decent hardware.
I'm constantly working on improving BeeLlama further and developing its unique features, while closely following upstream llama.cpp for improvements of their own.
So thank you to everyone using it, and stay tuned for more.
Expanded Bartowski + SINQ to include all recipes at 5-bit and above.
I'd say Bartowski's recipes at 5-bit and below are a bit weak, losing to Unsloth in both mean and median KLD. Stock Bartowski doesn't catch up to Unsloth in mean KLD until 6-bit, but SINQ already does at Q5_K_M.
At 6-bit, Bartowski still cleanly dominates Unsloth in mean KLD, even with all the additional recipes included in the comparison, especially with SINQ. Stock Q6_K_S performs particularly well for its size, with SINQ gains being smaller for this recipe.
Q6_K_S + SINQ is also very close to Unsloth in terms of median KLD, while being much better in mean KLD: 0.007257 vs 0.011163, which is 35% lower, and dominates Unsloth up until UD-Q8_K_L which has 0.007235.
It might actually be a very strong quant in the ecosystem overall, in terms of quality per model size. A great option if you have 32–48 GB of VRAM.
More results for Qwen 3.8 27B: Bartowski + SINQ has better mean KLD at Q6_K than Unsloth's UD-Q8_K_L!
At 6-bit and 8-bit, the dynamic becomes very interesting. In terms of mean KLD, stock Bartowski is already a bit ahead of Unsloth, which appears to be more "median-oriented".
SINQ tends to affect extreme errors the most. As a result, Bartowski + SINQ has better mean KLD at 6-bit than Unsloth does at 8-bit! See graph 1.
But if you look at median KLD, Unsloth is actually ahead of Bartowski even with SINQ, all the way up to Q8_0. See graph 2.
This might be a conscious decision in how the recipes are optimized, or just a side effect of the authors' methodologies.
So, why the difference? Extreme errors, P99.9 and max KLD. They affect the mean quite significantly, naturally, but not the median.
UD-Q6_K_L ends up having 240.91% higher P99.9 KLD compared to Bartowski Q6_K + SINQ. That's a lot. And unlike max KLD, which can be pretty random, P99.9 is more of a stable metric. See graph 3.
Which metrics are more important is debatable. One more consideration is top-1 agreement, which follows median KLD, with Unsloth being ahead. See graph 4.
Top-1 is a useful metric because it puts things into practical perspective. It's basically how often the quantization agrees with BF16. But the graphs make the differences look a bit dramatic. Consider the raw numbers:
- Unsloth UD-Q6_K_M = 98.1776%
- Q6_K + SINQ (w16 / 0.75) = 98.1786%
- Unsloth UD-Q6_K_L = 98.3856%
The difference doesn't look nearly as scary as 240.91% higher P99.9 KLD, but I can't just say it has no impact either. Still, maybe at this point reducing outliers actually has a greater effect on the output?
Of course, the best option would be not to choose at all, but to combine Unsloth with SINQ. :) Unfortunately, that's not currently possible due to their quantization process being only partially open-source.
My DMs are fully open-source tho. ;)
4
301
More results for Qwen 3.8 27B: Bartowski + SINQ has better mean KLD at Q6_K than Unsloth's UD-Q8_K_L!
At 6-bit and 8-bit, the dynamic becomes very interesting. In terms of mean KLD, stock Bartowski is already a bit ahead of Unsloth, which appears to be more "median-oriented".
SINQ tends to affect extreme errors the most. As a result, Bartowski + SINQ has better mean KLD at 6-bit than Unsloth does at 8-bit! See graph 1.
But if you look at median KLD, Unsloth is actually ahead of Bartowski even with SINQ, all the way up to Q8_0. See graph 2.
This might be a conscious decision in how the recipes are optimized, or just a side effect of the authors' methodologies.
So, why the difference? Extreme errors, P99.9 and max KLD. They affect the mean quite significantly, naturally, but not the median.
UD-Q6_K_L ends up having 240.91% higher P99.9 KLD compared to Bartowski Q6_K + SINQ. That's a lot. And unlike max KLD, which can be pretty random, P99.9 is more of a stable metric. See graph 3.
Which metrics are more important is debatable. One more consideration is top-1 agreement, which follows median KLD, with Unsloth being ahead. See graph 4.
Top-1 is a useful metric because it puts things into practical perspective. It's basically how often the quantization agrees with BF16. But the graphs make the differences look a bit dramatic. Consider the raw numbers:
- Unsloth UD-Q6_K_M = 98.1776%
- Q6_K + SINQ (w16 / 0.75) = 98.1786%
- Unsloth UD-Q6_K_L = 98.3856%
The difference doesn't look nearly as scary as 240.91% higher P99.9 KLD, but I can't just say it has no impact either. Still, maybe at this point reducing outliers actually has a greater effect on the output?
Of course, the best option would be not to choose at all, but to combine Unsloth with SINQ. :) Unfortunately, that's not currently possible due to their quantization process being only partially open-source.
My DMs are fully open-source tho. ;)
All the GGUFs are still being uploaded to HF: huggingface.co/Anbeeld/Qwen3…
The asymmetrical internet speed is an invention of Dario Amodei (aka the devil) to slow down (literally) development of open-source models.
1
53
First results for SINQ with Qwen 3.8 27B: turning Bartowski's good IQ4_XS into great IQ4_XS, with 13.12% lower mean KLD!
SINQ builds on the recipe's quality: Bartowski lags quite far behind Unsloth at 3-bit, and that gap remains, but at IQ4_XS the difference is now minimal.
I just made every GGUF quantization more accurate.
Introducing the first implementation of SINQ in the llama.cpp ecosystem, available in BeeLlama.cpp v0.5.0 Preview.
SINQ = Sinkhorn-Normalized Quantization, from the authors of KVarN. Take any existing GGUF quantization recipe, add SINQ to it, and its error compared to BF16 drops.
Here's a simple illustration. I took the Qwen 3.5 9B GGUF collection from the highly reputable Bartowski, whose quantization is fully open-source, and reproduced it with SINQ added.
Result: every single recipe achieved lower KLD, with drastic improvements for higher-bit quants. See graph 1.
I also compared the results against Unsloth's GGUF collection for the same model, which cannot be fully reproduced due to its partially proprietary quantization process. See graph 2.
While stock Bartowski and Unsloth trade blows at most model sizes, Bartowski + SINQ dominates most of the Pareto frontier, with a particularly large gap at 6 and 8 bits.
Disclaimers:
- The Bartowski + SINQ collection went through a manual selection process, with A/B testing of SINQ configurations to identify the best settings for each recipe. Automated selection algorithms are WIP.
- SINQ requires an inference engine that supports it, so these GGUFs can only run with BeeLlama.cpp v0.5.0 at the moment.
- Some GGUF recipes might benefit less from SINQ, or possibly not benefit at all, if their existing optimizations conflict with or supersede SINQ.
- Qwen 3.5 9B was chosen for its balance between research iteration speed within hardware constraints, and baseline model quality. Quantizations of larger models, such as Qwen 3.8 27B, will follow later.
GGUFs are currently being uploaded to HF: huggingface.co/Anbeeld/Qwen3…
Based on provisional results, at 6-bit and 8-bit the impact of SINQ will be much more transformative, so stay tuned for more!
5
72
GGUFs are now available on Hugging Face!
Includes all 24 quantization recipes, the SINQ width and strength settings used for each, and full KLD results.
Everything is open-source and easy to reproduce.
huggingface.co/Anbeeld/Qwen3…
I just made every GGUF quantization more accurate.
Introducing the first implementation of SINQ in the llama.cpp ecosystem, available in BeeLlama.cpp v0.5.0 Preview.
SINQ = Sinkhorn-Normalized Quantization, from the authors of KVarN. Take any existing GGUF quantization recipe, add SINQ to it, and its error compared to BF16 drops.
Here's a simple illustration. I took the Qwen 3.5 9B GGUF collection from the highly reputable Bartowski, whose quantization is fully open-source, and reproduced it with SINQ added.
Result: every single recipe achieved lower KLD, with drastic improvements for higher-bit quants. See graph 1.
I also compared the results against Unsloth's GGUF collection for the same model, which cannot be fully reproduced due to its partially proprietary quantization process. See graph 2.
While stock Bartowski and Unsloth trade blows at most model sizes, Bartowski + SINQ dominates most of the Pareto frontier, with a particularly large gap at 6 and 8 bits.
Disclaimers:
- The Bartowski + SINQ collection went through a manual selection process, with A/B testing of SINQ configurations to identify the best settings for each recipe. Automated selection algorithms are WIP.
- SINQ requires an inference engine that supports it, so these GGUFs can only run with BeeLlama.cpp v0.5.0 at the moment.
- Some GGUF recipes might benefit less from SINQ, or possibly not benefit at all, if their existing optimizations conflict with or supersede SINQ.
- Qwen 3.5 9B was chosen for its balance between research iteration speed within hardware constraints, and baseline model quality. Quantizations of larger models, such as Qwen 3.8 27B, will follow later.
I just made every GGUF quantization more accurate.
Introducing the first implementation of SINQ in the llama.cpp ecosystem, available in BeeLlama.cpp v0.5.0 Preview.
SINQ = Sinkhorn-Normalized Quantization, from the authors of KVarN. Take any existing GGUF quantization recipe, add SINQ to it, and its error compared to BF16 drops.
Here's a simple illustration. I took the Qwen 3.5 9B GGUF collection from the highly reputable Bartowski, whose quantization is fully open-source, and reproduced it with SINQ added.
Result: every single recipe achieved lower KLD, with drastic improvements for higher-bit quants. See graph 1.
I also compared the results against Unsloth's GGUF collection for the same model, which cannot be fully reproduced due to its partially proprietary quantization process. See graph 2.
While stock Bartowski and Unsloth trade blows at most model sizes, Bartowski + SINQ dominates most of the Pareto frontier, with a particularly large gap at 6 and 8 bits.
Disclaimers:
- The Bartowski + SINQ collection went through a manual selection process, with A/B testing of SINQ configurations to identify the best settings for each recipe. Automated selection algorithms are WIP.
- SINQ requires an inference engine that supports it, so these GGUFs can only run with BeeLlama.cpp v0.5.0 at the moment.
- Some GGUF recipes might benefit less from SINQ, or possibly not benefit at all, if their existing optimizations conflict with or supersede SINQ.
- Qwen 3.5 9B was chosen for its balance between research iteration speed within hardware constraints, and baseline model quality. Quantizations of larger models, such as Qwen 3.8 27B, will follow later.
BeeLlama.cpp v0.5.0 Preview
github.com/Anbeeld/beellama.…
Hot take: Open source has a much greater impact outside of The Bubble when most of the effort is concentrated on a few flagship projects, much like with commercial products.
Yes, Opus 5.5 can create a new inference engine for you daily, one for each model, but how much does that contribute to the ecosystem compared to implementing tensor split for Qwen 3.8 Flash Next in llama.cpp?
1
1
89
I see they finally abandoned what they are not good at (building harnesses) and pivoted to vlogging.
full recap of our week in Argentina
youtube.com/watch?v=D3Dgp9hE…
1
65
SweatyPanzer v1.3.0 brings first-person view! More immersion while driving, more precision while shooting.
Play now for free!
Browser: sweatypanzer.com/
Steam: store.steampowered.com/app/5…
68
Dead on arrival, judging by benchmarks.
Meet Mistral Large 4, aka Le Chonk.
• 1T parameters, natively multimodal. 49B active.
It is the best open weights model from US or Europe on aggregated benchmarks.
• State-of-the-art on critical workloads, including cyber defense, manufacturing and finance and it surpasses closed frontier models on visual grounding.
• Forged in Europe end-to-end and is deployable from Europe via our own Mistral Cloud infrastructure.
• Available to all via API today. Working with cybersecurity partners privately.
Open weights release end of October.
1
2
99
SweatyPanzer v1.0 is out! We cooked really hard on this one with @huggos_.
Try it now for free! Links in the replies ↓
Here's what changed, beautifully showcased in the trailer:
- All tank models have been remade to better match their historical appearance and details.
- Pershing and AMX-13 joined the roster, so there's a playstyle for everyone.
- More guns! The Sherman can now choose between the 76 mm gun and the 105 mm howitzer.
- Significant improvements to graphics, sounds, and map design.
- Even smarter bots that play as a team and try to flank you.
- Loads of other fixes and improvements.
1
60
