Hard Work and Guts
Austin, TX
Joined March 2025
- Tweets1.9K
- Following899
- Followers4.8K
- Likes7.3K
Pinned Tweet
2 months ago, AI was a chatbot that I occasionally used for random questions. Then I came across Ahmad’s posts about how it works and what it can do, and I knew immediately this was something I wanted to dedicate all my time and resource into and finally, my own sovereign AI.
I started out with the goal of having my own AI assistant that is 100% private, that I can access anywhere using text, voice and vision. Currently building out the second brain infrastructure that can continuously improve itself, with actual long-term memory so it doesn’t just end up being another glorified chatbot. Afterwards, tie it into genomics research and VCF automation, and design my private console to drive and monitor the entire infrastructure.
I’ll start sharing my journey in a way that isn’t just technical jargon, my hope is that it’ll inspire more people to get into AI to improve their everyday lives.
Addressing some questions on here at once.
Regarding overheating, you will absolutely see high temps. My 3090 is watercooled but unlock easily caps power at 340w on mine, and I cannot imagine that wouldn’t cause temp to spike even after 5 min inference. This is why I keep stressing watercooling, none of these things will matter if you do
Unlock can be permanent, def not something you have to tinker with once you have it set
Shit sorry, might've spoken too soon. There's some huge gains but its not translating to certain things.
It unlocked FP16/BF16 TFLOP in huge amount, 1.65x but doesnt impact token generation at all. What it does benefit (thanks bandit) is image generation, fine-tuning, and maybe concurrency. Will post full numbers when its done
When you look at the spec of modern GPUs such as Blackwells, you'll see that it has FP4/FP8 tensor cores which older GPUs lacks. This means when we load a NVFP4 model on 3090, it cannot compute in FP4 and needs to go through extra steps to unpack it. So how much does this matter? Been wondering for awhile so put it to test
Bottom line is, negligible when it comes to decode, big impact on prefill. Also matters quite a bit when concurrency gets involved, so keep all this mind when you are selecting your next GPU.
As requested, GPU list that won't require 2nd mortgage to buy. If I was starting from scratch, I'd buy these 4 that has strong memory + bandwidth balance
RTX 3090 24GB 936GB/s $1.6k
RX 7900 XTX 24GB 960GB/s $1.2k
MI50 32GB 1024GB/s $700
MI100 32GB 1229GB/s $1350
AMD/MI's need tinkering, but raw capability is there those willing to put in the work
RAM vs Gen5 vs Gen4 NVMe Engram Offload
Offloading to RAM gives significant increase in prefill and decent rise in decode over Gen5 NVMe. Gen5 outperforms Gen4 by good amount as well
Bonus data, Gen5 loads models roughly 20% faster than Gen4, but no benefits outside of that I can measure
How would you test a model for intelligence? During my swarm experiments with generational agents, saw some incredible behaviors by 300B tier models that 10-100B models didn’t show, which made me want to see what even larger models will do. While amazing to see, it sort of happens at unpredictable times so trying to come up with a controlled setting where I can test for complex behavior.
I understand ‘intelligence’ could mean many different things, which is why I’d greatly appreciate your feedback on how I can best test for it once I get TP8 up and running
Examples of how PCIe switch handle various GPU/model configurations
Dense model behind same switch, zero difference.
MoE model behind same switch, actually faster.
TP that combines GPUs inside + outside switch (goes through CPU) loses any switch benefits and risks collapse, avoid at all cost
Switch with 100 lanes supports 4 GPU at full x16 over a single PCIe slot, perfect for running multiple single models or TP2/4. Worth it if you’re out of slots and/or want to maximize moe TP performance, or want to shoot for proper TP8 which is whole another ballgame requiring 3 switches