CrypticSploit

Qwen3.8-35B-A3B is reportedly generating around 50 tokens/sec on an RTX 3060 with just 12GB of VRAM and 16GB of system RAM. The same setup reportedly processes prompts at around 550 tokens/sec, using Q4_K_M quantization and a configured 262K context window. Here’s the hardware and software configuration: GPU: RTX 3060, 12GB VRAM System RAM: 16GB Model quantization: Q4_K_M Generation speed: ~50 tokens/sec Prompt processing: ~550 tokens/sec Configured context: 262K tokens Runtime: llama.cpp with Flash Attention and an 8-bit KV cache How can a model with 35B parameters run on hardware with these memory limits? The answer starts with its Mixture-of-Experts architecture. Qwen3.8-35B-A3B contains roughly 35B total parameters, but activates around 3B parameters per token. Instead of performing computation across every parameter for each generated token, the model routes work through selected experts. This reduces the computation required during generation. However, active parameters and total model size are two different things. The full collection of weights still needs to be stored in GPU memory, system RAM, or a combination of both. That’s where this setup gets interesting. Hybrid inference makes the available memory go further. The reported configuration offloads 26 MoE layers to the CPU, splitting the workload between the RTX 3060 and system RAM. Q4_K_M quantization reduces the weight footprint, while Flash Attention and an 8-bit KV cache help manage memory use during inference. The trade-off is that CPU offloading can introduce additional latency. Performance depends on the CPU, memory bandwidth, prompt length, and how much computation stays on the GPU. There are two figures worth distinguishing: 550 tokens/sec is prompt-processing speed, not generation speed. These are separate stages of inference, and fast prompt processing doesn’t automatically mean equally fast text generation. Likewise, a configured 262K context window describes the intended capacity, not a guarantee that every prompt length will fit comfortably or run at the same speed. The real test is agentic coding Generating 50 tokens/sec is useful, but speed alone doesn’t tell us whether a model can function as a coding agent. An agent needs to inspect files, understand a repository, execute commands, diagnose errors, modify code, and verify its fixes. It must also recover when its first approach fails. That’s why the planned comparisons with Ornith-1.5-35B, Qwen3.8-27B, and Qwen3.6-35B-A3B are worth watching. Testing these models through Hermes and Pi coding-agent workflows could reveal differences in tool use, debugging, task completion, and recovery from failed attempts. If Qwen3.8-35B-A3B can handle those workflows reliably while running locally on a 12GB GPU, it could offer developers a practical alternative to relying entirely on cloud-based coding models. The reported performance makes this setup interesting. The next question is whether that speed translates into reliable results on real development tasks.