
Jack Smith retweeted
Open source project renders your Windows 11 desktop using the NPU rather than the GPU — ‘npu-driver’ can only run at 2 to 3 Hz, targets 30 Hz performance tomshardware.com/pc-componen…
Jack Smith retweeted
it should tell you something that the most bleeding edge ai-pilled technologists are not letting AI write for them
i love writing. i never let agents write for me. whether it's docs, or tweets, or shitposts, or even prompts (okay maybe that one's harder as agents manage other agents)
there's a simple reason for that: i still strongly believe that writing is thinking. it's a skill i hope to never give up. it's also important to me that whatever i say is always authentic, mistakes and all. my favorite posts are the ones that i just full send without any editing
i suspect it also helps with prompting. being able to clearly express what i want, and knowing the vocabulary i need to be able to direct my agents does seem to make for more effective usage of agents. in the same way that engineers have an advantage when we prompt agents to write code, because we are intimately familiar with the words that make up the discipline
knowing the right word incantations to get an agent to produce a desired goal almost feels like spellcasting. as models get more capable, and that input box looks more and more like a magic box that can output anything, words become the interface between you and the genie
there may come a time when we give up the keyboard in exchange for brain-computer interfaces. but until then, i'm writing every word.
Jack Smith retweeted
We ran @thinkymachines’ Inkling 975B across eleven Intel Core Ultra PCs using @withcascadia.
Peak aggregate throughput: 60.29 tok/s at 88 concurrent streams. Single stream clocked 5-10 tok/s.
Almost a trillion parameters on consumer hardware :)
Cascadia is now serving Inkling (975B) from @thinkymachines across eleven Intel Core Ultra PCs, in a deployment with @IntelBusiness.
Quantized, Inkling sits at ~475GB in size, which we split into shards and distributed across the eleven machines. Each one runs a custom MoE engine built on top of OpenVINO.
Peak aggregate throughput reaches 60.29 tok/s at 88 concurrent streams.
People blame technology for many problems, when in reality the problems are created by humans.
Greed, lack of empathy, etc. These are the things that make software (among other things) worse
1
73
Jack Smith retweeted
We just published a paper on @withcascadia’s architecture.
Distributed inference is hard, and it’s even harder P2P: you have to handle networking, discovery, routing, scheduling, identity, security, etc.
The reward is the ability for any number of PCs to pool their compute, and serve inference without relying on a centralized control plane.
Full paper linked below.
Weirdly, AI being incredibly good at programming doesn’t bother me at all.
I now spend that time getting to write, make art, and learn more
20
you haven’t lived until you’ve eaten buckwheat with milk and sugar
buckwheat is literally the most insane food out there, imagine rice but it has 11 grams of protein per 100g and a fuckton of vitamins AND it tastes a lot better(subjective obv). the fact that gymrats dont eat buckwheat en masse and instead eat white fucking rice is baffling to me
1
2
119
Jack Smith retweeted
Your business already owns the AI compute, but it’s just sitting idle.
@TateBerenbaum and Cascadia.to work with Intel to help teams take advantage of this power potential by turning their existing Intel hardware into a private, on-prem, distributed inference network—one that you own and that can run a range of open-weight and local models without additional hardware spend or per-token cloud costs.
Learn more: ms.spr.ly/6014aOYks
I wonder how many (nearly) identical pieces of software have been created by prompting an LLM
And by extension how you've just burned double the amount of tokens for something that could've just been a `git clone` or copy paste
Hmm
1
44
Jack Smith retweeted
Excited to share the first research paper behind @withcascadia
We discovered pipeline micro-batching and speculative decoding can make up for much of the added latency of sharding models over TCP. This paper breaks down how, along with some novel work on NPU continuous batching.
Jack Smith retweeted
Some cool facts about yesterday's launch video:
- We built the shelves ourselves
- This was running without wifi. Completely off-grid in the middle of Texas
- The sequential flashes you saw at the end were actual tokens being generated
- We clocked ~1 tok/s on a 70B param model via CPU across 38 Dell laptops
- Cost <$10k and planned/shot/edited in ~3days















