Xerxes Rånby

Qwen3.8-35B-A3B is reportedly generating around 50 tokens/s on an RTX 3060 with just 12GB VRAM and 16GB system RAM. The same setup reportedly delivers around 550 tokens/s in prompt processing. Here’s the configuration: GPU: RTX 3060, 12GB VRAM System RAM: 16GB Quantization: Q4_K_M Generation: ~50 tokens/s Prompt processing: ~550 tokens/s Configured context: 262K tokens Runtime: llama.cpp Optimizations: Flash Attention and Q8 KV cache Running a model of this size on a 12GB GPU sounds unusual, but its architecture explains part of the story. Why a 35B model fits this setup Qwen3.8-35B-A3B uses a Mixture-of-Experts (MoE) architecture. It contains approximately 35B parameters, but only around 3B are active for each token. Instead of activating the entire model during every generation step, the architecture routes computation through selected experts. This reduces the computational workload compared with a dense model of similar total size. However, there’s an important distinction: having only 3B active parameters does not mean the other weights disappear from memory. The full model still needs to be stored and accessed. The solution is hybrid inference. The reported setup uses llama.cpp to distribute the workload between the GPU and CPU. Specifically, 26 MoE layers are offloaded to the CPU, allowing system RAM to hold and process parts of the model that cannot fit in the RTX 3060’s VRAM. Two additional optimizations help manage memory: Flash Attention: improves attention computation and can reduce memory overhead. Q8 KV cache: stores attention keys and values in 8-bit precision, reducing cache memory requirements compared with higher-precision storage. Q4_K_M quantization also reduces the model’s weight footprint. Together, these techniques make it possible to run a much larger model on hardware that would struggle to accommodate a conventional 35B dense model entirely in VRAM. One caveat: the reported 262K context is the configured capacity. It doesn’t guarantee the same speed or memory usage at every context length. Longer prompts increase KV-cache requirements and can affect performance, particularly on a system with only 16GB of RAM. But the real test is agentic coding. Generating 50 tokens per second is useful, but coding agents need more than generation speed. They have to inspect files, understand an existing codebase, call tools, modify code, diagnose errors and verify whether their changes actually work. The planned comparisons include: Ornith-1.5-35B Qwen3.8-27B Qwen3.6-35B-A3B Testing them through Hermes and Pi coding-agent workflows should reveal how well Qwen3.8-35B-A3B handles multi-step development, tool calls, debugging and recovery from failed attempts. If it can complete those tasks reliably while running locally on a 12GB GPU, it could become an interesting option for developers who want capable coding agents without depending entirely on cloud inference. The initial inference numbers are promising. What matters next is whether the model can turn that performance into reliable work.