Cascadia is now serving Inkling (975B) from @thinkymachines across eleven Intel Core Ultra PCs, in a deployment with @IntelBusiness.
Quantized, Inkling sits at ~475GB in size, which we split into shards and distributed across the eleven machines. Each one runs a custom MoE engine built on top of OpenVINO.
Peak aggregate throughput reaches 60.29 tok/s at 88 concurrent streams.
We’re working on upstreaming this into Cascadia, so that anyone with the available hardware can try it out.
Follow the progress (and leave a star) here:
github.com/labscommunity/cas…
Cascadia retweeted
Your AI wouldn't be down if you were using @withcascadia rn 🤠
Cascadia retweeted
Weekends are for running @withcascadia on chromebooks…
Solid 4gb of RAM 🤠
Cascadia retweeted
Why choose between local and hosted if you want to use both together?
We built yeschef so Claude Code and Codex can spawn subagents that run locally.
Works out of the box with @withcascadia, @Ollama, @exolabs, etc.
Save on your Claude usage limits and put your GPU to use.
Cascadia retweeted
Your business already owns the AI compute, but it’s just sitting idle.
@TateBerenbaum and Cascadia.to work with Intel to help teams take advantage of this power potential by turning their existing Intel hardware into a private, on-prem, distributed inference network—one that you own and that can run a range of open-weight and local models without additional hardware spend or per-token cloud costs.
Learn more: ms.spr.ly/6014aOYks
Cascadia retweeted
The best place to find machines with 1.5TB of memory for <$10k?
2019 Mac Pro.
Using to test @withcascadia
Cascadia retweeted
Excited to share the first research paper behind @withcascadia
We discovered pipeline micro-batching and speculative decoding can make up for much of the added latency of sharding models over TCP. This paper breaks down how, along with some novel work on NPU continuous batching.
Cascadia retweeted
Replying to @jackwsmth @withcascadia
I did some follow-up testing with Cascadia!
Qwen3.8-27B didn’t work out of the box, so I had to export it to OpenVINO FP16, re-quantize it to INT4 with NNCF, and simplify the chat template for Minja before I could finally get it running.
On the Arc 140V, INT4_ASYM / g128 reached about 6.3–6.4 tok/s, while INT4_SYM / g128 reached about 6.6 tok/s. That’s noticeably faster than the llama.cpp baseline of about 4.4 tok/s even without speculative decoding.
Prompt Lookup caused the output to break down, and draft-model speculative decoding isn’t supported with this VLM-layout export.
Thanks for the interesting suggestion — this was a lot of fun to experiment with!
Cascadia retweeted
I’ve been trying to figure out the best way to apply NPUs to inference.
Because NPUs are better at parallelism (and therefore digesting prefill tokens), and CPUs are generally better at token-by-token decoding, you can take a hybrid approach to inference if you have a CPU + NPU chipset.
TTFT speedup was 35x on a 1B model, and 40x on a 3B model (compared to solely CPU inference). Merged into @withcascadia a few weeks ago :)
Run the most powerful AI models across Intel silicon.
Co-developed with @IntelBusiness.
People shouldn’t have to build data centers to access frontier AI.
We just need it to run better on commodity hardware.
So we teamed up with @IntelBusiness to build a solution. This is Cascadia, and now you can use it too.
Check it out here:
github.com/labscommunity/cas…
Announcement:
cascadia.to/blog/cascadia-ru…
5
236












