For researchers that want to know how this is done without going on AI-heavy details: ( affordable DIY at the bottom ) They took a model, and compressed it to be able to run on a single 4-GPU (still expensive for non-enterprise, so not for solo use - think $80/hour or $180,000 purchase). Their process / methodology: - Pick a model with the expert architecture - Choose a calibration dataset that matches the intended use ( So audited codebases v.s. reports etc) - Run the calibration data through the unmodified model and record two numbers per expert per token - Compute a single score per expert - Delete the lowest-scoring experts in each layer The expert's weights are dropped from the model file, and the router's output columns for those experts are removed so it can only choose among survivors. The router then renormalizes its weights over the remaining experts as it normally would. This is EXPENSIVE. BUT it's no different than the token game you play v.s. large scale security companies either way so no reason to give up. You can do the exact same process for a smaller model and have something you can run on your mac or a hetzner node. 🙂‍↔️🙂‍↔️ Affordable DIY plan: 1. Start from the unpruned GLM-4.7-Flash 2. Collect a few hundred to a thousand traces from your own audits 3. Run the Cerebras REAP repo over them on a rented A100 for an hour or two. Under $10 of compute. 4. Prune 25%~50%, quantize to GGUF, load in LM Studio or whatever. 5. Run locally on your mac