Alberto Cetoli (@[email protected])

The results are in: Trade-offs between accuracy & performance in LLM quantization After hundreds of thousands of evals and benchmarks from our research team at @neuralmagic, I'm excited to share our findings on LLM quantization—now available as a paper on arXiv: arxiv.org/abs/2411.02355 Some key insights: 1/ w8a8 quantization delivers up to 8x speedup on high-performance GPUs, making it ideal for larger models or server deployments. 2/ For smaller models, w4a16 quantization can reduce cost per request by up to 7x, especially for synchronous use cases. 3/ FP8 (w8a8) is essentially lossless in our benchmarks. INT8 (w8a8) & INT4 (w4a16) also retain very high fidelity. 4/ AWQ and GPTQ are close for academic benchmarks, but AWQ struggles with real-world performance. 📄 Full paper: arxiv.org/abs/2411.02355 Interested in reducing your model costs, boosting performance, or scaling efficiently? DM me—let’s chat! ✉️ And stay tuned—🚀 - New compressed model launch soon - Multi-modal model results incoming - More benchmarks, algorithms, and evaluations on the way #LLMs #Quantization #Optimization #GenAI #AI