asiai Benchmark (@asiai_dev) header
asiai Benchmark (@asiai_dev) profile picture

asiai Benchmark

@asiai_dev

Which engine is fastest on your Mac? Open-source benchmark & monitoring for Apple Silicon LLM inference. 7 engines, GPU power, thermal — no sudo needed.

asiai.dev
Joined March 2026
  • Tweets18
  • Following45
  • Followers3
12Media
Photo by asiai Benchmark (@asiai_dev)Photo by asiai Benchmark (@asiai_dev)Photo by asiai Benchmark (@asiai_dev)Photo by asiai Benchmark (@asiai_dev)Photo by asiai Benchmark (@asiai_dev)Photo by asiai Benchmark (@asiai_dev)Photo by asiai Benchmark (@asiai_dev)Photo by asiai Benchmark (@asiai_dev)Photo by asiai Benchmark (@asiai_dev)Photo by asiai Benchmark (@asiai_dev)
asiai — open-source Apple Silicon benchmark & monitoring for local LLM inference. Compare engines & models. Real numbers. pip install asiai asiai.dev
asiai — Multi-engine LLM benchmark for Apple Silicon

asiai — Multi-engine LLM benchmark for Apple Silicon

Compare LLM inference engines on Apple Silicon. Measure tok/s, TTFT, GPU power and thermal across 10 engines. Open source CLI.

asiai.dev
1
52
Qwen3.8-27B open weights are out. Downloading now. Bench starts the moment they land: M5 Max, one engine resident at a time, same build and same quant as our Qwen3.6-27B baseline. Anything else would measure the setup instead of the model. First numbers tonight in this thread.
Qwen 3.8 27B benchmark

ALT Qwen 3.8 27B benchmark

1
1
165
First cell done. Qwen3.8-27B on llama.cpp b10270, unsloth UD-Q5_K_XL, M5 Max, 65k context, n=3, reasoning off and verified: 21.9 tok/s warm, 21.5 cold start, 17.7 at a 55k-token prompt. Thermal throttling declared, as on our baseline. Full conditions on the card.
asiai speedtest card. Qwen3.8-27B UD-Q5_K_XL on llama.cpp b10270, Apple M5 Max 128 GB, agentic bench n=3. Hero figure: 21.9 tok/s sustained, warm cache. Bars per phase: first call 21.5 tok/s (TTFT 3497 ms), warm 21.9 (103 ms), cold+prefix 21.5 (229 ms), long context 17.7 tok/s at 56k tokens, minus 19 percent. Reasoning off verified, 100 percent valid outputs, throttled to 50 percent declared. All gates green except thermal. asiai v1.32.0, 2026-08-14.

ALT asiai speedtest card. Qwen3.8-27B UD-Q5_K_XL on llama.cpp b10270, Apple M5 Max 128 GB, agentic bench n=3. Hero figure: 21.9 tok/s sustained, warm cache. Bars per phase: first call 21.5 tok/s (TTFT 3497 ms), warm 21.9 (103 ms), cold+prefix 21.5 (229 ms), long context 17.7 tok/s at 56k tokens, minus 19 percent. Reasoning off verified, 100 percent valid outputs, throttled to 50 percent declared. All gates green except thermal. asiai v1.32.0, 2026-08-14.

2
1
193
Update: Qwen3.8-27B ships a native MTP draft head, and the GGUF carries it. With speculative decoding on (draft cap 4): 31.8 tok/s warm, 22.3 at 56k, TTFT 112 ms. Same machine, same quant, same pinned build. The 21.9 earlier was measured without MTP, as its card states.
Measurement card from asiai, the speedtest for local LLMs. Qwen3.8-27B UD-Q5_K_XL on llama.cpp b10270, Apple M5 Max 128 GB, native MTP speculative decoding enabled (draft cap 4). Headline: 31.8 tok/s sustained, warm cache. Bars: first call 31.1 tok/s (3321 ms), warm 31.8 tok/s (112 ms), cold 30.6 tok/s (343 ms), long context 22.3 tok/s at 56k. n=3 runs, reasoning off verified, 100% valid outputs, thermal gate declared (throttled to 50%).

ALT Measurement card from asiai, the speedtest for local LLMs. Qwen3.8-27B UD-Q5_K_XL on llama.cpp b10270, Apple M5 Max 128 GB, native MTP speculative decoding enabled (draft cap 4). Headline: 31.8 tok/s sustained, warm cache. Bars: first call 31.1 tok/s (3321 ms), warm 31.8 tok/s (112 ms), cold 30.6 tok/s (343 ms), long context 22.3 tok/s at 56k. n=3 runs, reasoning off verified, 100% valid outputs, thermal gate declared (throttled to 50%).

4
One honest detail: turning on the room AC moved the same run from 19.1 to 21.9 tok/s. Ambient temperature is a benchmark condition !😅
11
72+ tok/s on Qwen 3.6 27B on an M5 Max? We checked: same machine, every engine in isolation, n=5. It holds.
Youssof Al Toukhi (@Youssofal_) profile pictureYoussof Al Toukhi
@Youssofal_Jul 7
72+ TPS on Qwen 3.6 27B on a Macbook pro M5 max. MTPLX V2 out now! The fastest way to run models on MLX.
1
1
121
The numbers, isolated runs: 66.7 tok/s sustained, 73.6 at p90, 2.3x llama.cpp Q8 at half the energy per token. Still 43.6 at 64K context. This model runs a 24/7 local agent here: at that duty cycle, half the watts is the real headline. Full methodology: asiai.dev
asiai — Multi-engine LLM benchmark for Apple Silicon

asiai — Multi-engine LLM benchmark for Apple Silicon

Compare LLM inference engines on Apple Silicon. Measure tok/s, TTFT, GPU power and thermal across 10 engines. Open source CLI.

asiai.dev
12
70B model. Mac Mini M4 pro 64GB. Llama-3.1-70Bat 6.3 tok/s, measured with asiai. TurboQuant compresses the KV cache from 20GB to 4GB. Same quality, zero fine-tuning. One week ago this didn't fit in memory. Now it runs at 24 watts. asiai.dev/turboquant/ #LocalLLM #AppleSilicon
Benchmark card asiai: Llama-3.1-70B Q4_K_M running at 6.3 tok/s on Mac Mini M4 Pro 64GB with TurboQuant turbo3 KV cache compression. Single engine llamacpp, 188ms TTFT, stable, 44.3 GB VRAM, 24W at 0.3 tok/s/W, 3 runs.

ALT Benchmark card asiai: Llama-3.1-70B Q4_K_M running at 6.3 tok/s on Mac Mini M4 Pro 64GB with TurboQuant turbo3 KV cache compression. Single engine llamacpp, 188ms TTFT, stable, 44.3 GB VRAM, 24W at 0.3 tok/s/W, 3 runs.

82
Qwen3-Coder-30B on M4 Pro 64GB: LM Studio (MLX): 102 tok/s — 12.4W — 8.2 tok/s/W Ollama (llama.cpp): 69.8 tok/s — 15.4W — 4.5 tok/s/W Same model. Same chip. +46% speed, -20% power. The runtime is half the equation. #AppleSilicon #LocalLLM
Image from asiai Benchmark (@asiai_dev)
61
asiai v1.3.0 is out 🎬 New web dashboard — dark theme, live monitor, shareable benchmark cards. M4 Pro 64GB · Qwen 3-Coder 30B - lmstudio: 73.4 tok/s - omlx: 66.0 tok/s - ollama: 47.6 tok/s Try it: pip install asiai asiai.dev #AppleSilicon #LocalAI
94
Most people compare models. Few compare engines. The engine you pick can matter more than the quantization level. asiai bench --compare github.com/druide67/asiai
GitHub - druide67/asiai: Multi-engine LLM benchmark & monitoring CLI for Apple Silicon

GitHub - druide67/asiai: Multi-engine LLM benchmark & monitoring CLI for Apple Silicon

Multi-engine LLM benchmark & monitoring CLI for Apple Silicon - druide67/asiai

github.com
1
10
Most people check tok/s. Few check TTFT. On agentic workloads with multi-turn tool calls, TTFT compounds fast. 30ms vs 257ms — that's 8x. asiai measures both. Every run. Every engine. asiai.dev #AppleSilicon #LocalLLM
asiai — Multi-engine LLM benchmark for Apple Silicon

asiai — Multi-engine LLM benchmark for Apple Silicon

Compare LLM inference engines on Apple Silicon. Measure tok/s, TTFT, GPU power and thermal across 10 engines. Open source CLI.

asiai.dev
70
Qwen3.5-35B on M4 Pro 64GB: LM Studio (MLX): 72.6 tok/s — TTFT 30ms Ollama (llama.cpp): 30.4 tok/s — TTFT 257ms 2.4x speed. 8x time-to-first-token. Same model. Same Mac.
103
Load more