AI Engineering Current: vibe-deploying @pocketmodelai to prod Prev: OpenAI, Netflix, Uber
Latent Space
Joined July 2007
- Tweets8.5K
- Following1.9K
- Followers2.9K
- Likes16.5K
Pinned Tweet
Run DeepSeek V4.1 Flash 1017, a SOTA-level model, at 30 tokens a second on your old gaming PC.
Hardware:
- 2x RTX 3090 (24 GB each)
- 128 GB DDR4
- i7-12700K
- NVMe drive
Speed:
- decode 32–36 tok/s with --defer-rank 3 (27–29.5 without)
- prefill 383 tok/s on a 3K-token prompt
github.com/lxe/dsx
huggingface.co/lxe/DeepSeek-…
This is it. This is the missing model.
@Recursive_SI Flint is a modern MoE sized for the hardware people actually have: 40B total with ~2.5B active per token, so it fits a 24 GB Mac or a single consumer GPU and decodes incedibly fast
- 256 routed experts, 6 per token, plus a shared expert
- 262K context, reasoning and tool calling
- Apache 2.0,
- 125+ tok/s on an M4 Pro MacBook Pro
- people already have it running on a single RTX 3060 at ~46 tok/s
- I bet unsloth has dynamic quants for this already
Running it against Qwen3.6-35B-A3B in @pocketmodelai now; results soon.
Watch an eval of @Recursive_SI's Flint head-to-head with Qwen3.6-35B-A3B, on @pocketmodelai.
It researches both models, builds a JavaScript coding eval from MultiPL-E HumanEval, writes a code grader that runs each answer against its tests, and compares accuracy, speed and cost
Aleksey Smolenchuk retweeted
Watch an eval of @Recursive_SI's Flint head-to-head with Qwen3.6-35B-A3B, on @pocketmodelai.
It researches both models, builds a JavaScript coding eval from MultiPL-E HumanEval, writes a code grader that runs each answer against its tests, and compares accuracy, speed and cost
Introducing Flint, @Recursive_SI’s first model: a fast local coding model built with our automated research system, the Eureka Machine.
• Best-in-class fast local coding
• Up to 2x faster than comparable models: 125+ tokens/sec on a MacBook Pro, 150+ on RTX Spark
• Built from scratch in under 4 months
Private, unmetered intelligence on your own machine. Open weights available now on @huggingface, run it locally on your favorite framework or try it for free on @opencode or @pidotdev.
Aleksey Smolenchuk retweeted
Introducing Flint, @Recursive_SI’s first model: a fast local coding model built with our automated research system, the Eureka Machine.
• Best-in-class fast local coding
• Up to 2x faster than comparable models: 125+ tokens/sec on a MacBook Pro, 150+ on RTX Spark
• Built from scratch in under 4 months
Private, unmetered intelligence on your own machine. Open weights available now on @huggingface, run it locally on your favorite framework or try it for free on @opencode or @pidotdev.
In theory we can get exl3-like accuracy without speed tradeoffs for CPU kernels for architectures that lack AVX-512
Aleksey Smolenchuk retweeted
We are improving by the minute!! 🫡🫡
Aleksey Smolenchuk retweeted
Quantization is easy. Fast quantization kernels for every format × every GPU × every workload are not.
Today we're open-sourcing Humming with Ant Group: one kernel library for all of it — up to 5x kernel speedups, +14–43% E2E MoE throughput.
Blog + benchmarks: vllm.ai/blog/2026-10-21-humm…
Aleksey Smolenchuk retweeted
EXL2.9 DeepSseek Flash 4.1 1017 streaming on 128gb machines has been updated for future versions of mlx-serve.
Prefill tops out around 600tps while decode sits around 30-40tps. Its very usable.
Aleksey Smolenchuk retweeted
I need to start keeping track of all these
@catonooka custom engine?
@joaosump custom engine?
@lxe dsx custom engine (mine)
@superalesha's gpu unlock + custom engine?
@PeasantSmith / @needmorevram project maya
@antirez dwarf star / ds4
llama.cpp w/@UnslothAI pr
@coldniko indrasmirror xolotl
who else?
Aleksey Smolenchuk retweeted
DeepSeek V4.1 flash 1017 EXL3 2.95bpw - duo 3090 no NVLink
Update 4: prefill is now at 783 t/s
I am pretty happy with current number, should share the work after real coding work, and hopefully a good PR to Maya repo.
Custom engine for DeepSeek V4.1 flash 1017 EXL3 -> jump directly to Maya repo
Update 3:
DSV41 Speed Checkpoint
Decode
• Before: 10.4 t/s
• Now: 26-28 t/s (2.6×)
Prefill @6K
• Before: 305 t/s
• Now: 380 t/s
Prefill @25K
• Before: 166 t/s
• Now: 516 t/s (3.1×)
How we got here, all benched:
1. Restored the crashed rig + rebuilt the EXL3 pack locally (3 mixed-K fixes, zero downloads)
2. Ported 8 maya ideas into our vLLM custom code one-by-one; every one measured flat or negative (route bias ×5 configs, tier depth, EC profile, decay/hyst, prefill chunks, DMA). Env space exhausted; lessons documented
3. EP2 dual-GPU vLLM retest: dead (all-to-all latency)
4. Pivoted to the maya engine: built v1.1.19 for sm_86 in a glibc-2.39 container (fixes the memlock crashes), packed our local 197G+190G files, seeded their shipped expert profile
5. First time I can run dual-3090 DSV41 numbers: 17.5 cold → 28.2 warm decode
Aleksey Smolenchuk retweeted
on the other hand, am backing to optimizing for a single 3090 now, current figures are:
@UnslothAI UD-IQ1_M quant with almost 90% KLD (not bad) + dense parts (att, router, embeddings) at q6/q8
1M context window - decode is at 37.5 and prefill 2200 tk/s
that's the baseline
Aleksey Smolenchuk retweeted
Agreed DeepSeek-v4.1-flash is an amazing model.
and we fixed the speed, it runs at 615+ tok/s on Wally
here it is against DeepSeek's own api making changes to the 2048 game -:
llama.cpp's anti-AI contributions policy is hindering its own mission in the most ironic way
All inference engines, AI-written, or human-written, will converge on the same optimization techniques. There's no reason to lag behind through artificial restriction.
Think of AI-generated code as a compiler optimization. Arbitrarily nerfing your own codebase seems very silly to me.
Aleksey Smolenchuk retweeted
Are all the neolabs converging on the same thing?
@togethercompute started with inference and is now offering sandboxes and RL training
@FireworksAI_HQ AI started with inference and is now offering RL training
@PrimeIntellect started with RL training and is now offering inference and sandboxes
@modal started with cloud compute and sandboxes and is now offering inference and RL training
