Vik | Practical AI & Tech (@uctive) header
Vik | Practical AI & Tech (@uctive) profile picture

Private local AI server for small businesses: your data stays inside your company. https://nitter.gq/t.co/dnhFJ2Yoqh

Como, Italy
Joined December 2011
A quick progress update from my local AI experiments. Earlier today, I went down to the basement and found the lights on for absolutely no reason. I simply asked my AI agent, out loud, in natural language: “What’s going on? Figure out why this happened and make sure it doesn’t happen again.” No technical instructions, no commands, no configuration details. Just a normal conversation. And here’s the agent’s report.
1
1
1
54
Meanwhile, I gave my local AI agent another practical job: securing the basement where the servers now live. I plugged in a USB camera, wired up some lights and relays, and left all the software configuration to the agent. Now motion automatically triggers lighting and video recording, with footage sent to external storage. Even if someone steals the servers, the evidence survives. Dear burglars, please wear masks. 😂🥷 Although getting inside won’t be easy. With ChatGPT’s help, I finally found a suitable lock for our armored basement door — something I’d been unable to find for years. I’m also making the house increasingly energy-independent. The solar power system already has a 5 kWh battery connected. Another 15 kWh of backup batteries are waiting to be connected. Literally just a couple of wires left! And I’ve already built triple internet redundancy: 🛰️ Starlink — satellite internet. 📡 5G — via a fixed external antenna. ☎️ ADSL — good old copper telephone wires, our third backup connection. Unfortunately, fiber still hasn’t reached the outskirts of our little Italian village. So my increasingly intelligent home is becoming more energy-independent, connected, resilient, and autonomous. The next step is bringing everything together under local AI control — including reverse engineering devices, building custom integrations, and letting agents develop the software themselves. Fewer benchmarks, more real-world local AI. 😁
2
1
103
We slowed down our AI benchmarks for a couple of days. Why? We moved the servers to the basement… and suddenly the house got cold. 😂 Turns out, servers were part of our heating system! So I installed two heat pumps in the bedrooms, worked on our solar power infrastructure, and let a local AI agent handle some household automation. The bigger plan? Replace gas heating with modern heat pumps, integrate everything into a locally controlled smart home, and let AI optimize heating zone by zone. We’ll measure actual energy efficiency, log everything, and publish the results. This project is secondary for now, but it’s definitely happening. Meanwhile, we’re also experimenting with reverse engineering, connecting incompatible devices, and letting AI agents write custom software. So we haven’t really stopped testing local AI. We’ve just temporarily switched from tokens per second to degrees Celsius. 😂
2
2
174
BONUS 6/5 — My actual hardware downgrade: DDR5 → DDR4 Here’s the configuration I’m moving my entire home and business infrastructure to. BEFORE — CLARYEL01: * CPU: Intel Core Ultra 7 270K Plus, 24 cores * Motherboard: ASUS Pro Q870M-C * RAM: 64 GB DDR5 * Storage: 4× Lexar NM1090 PRO 2 TB, PCIe 5.0 * RAID 0: 30–35 GB/s read throughput * Additional SSD: Samsung 990 PRO 2 TB * GPU: none AFTER — P1005 (DDR4): * CPU: Intel Xeon E5-2690 v4, 14 cores / 28 threads * Motherboard: ASUS Z10PA-U8 * RAM: 64 GB DDR4 ECC * GPU: NVIDIA RTX 3050 8 GB * SSD: Patriot Viper VP4300L 1 TB NVMe * SAS controller: LSI SAS3008 12G * Storage: 2×20 TB SAS HDD, RAID 1 * OS: Ubuntu 24.04 LTS WHAT’S MOVING? Around 40 Docker containers and supporting services: Home Assistant, Zigbee, Mosquitto MQTT, ERPNext, Odoo, OpenProject, PostgreSQL, MariaDB, MongoDB, Qdrant, Meilisearch, SeaweedFS, Valkey, Redis, LibreChat, LiteLLM, Activepieces, Ollama, ClamAV, agent infrastructure, knowledge APIs, integrations, monitoring, document storage, backups, and business automation. Plus Hermes Agent, with 40 scheduled jobs. The primary AI inference engine remains on a separate machine. I’m moving from a modern 24-core DDR5 platform with extremely fast PCIe 5.0 storage to an older 14-core Xeon with DDR4 and a modest 8 GB GPU. This isn’t about saving money. I already own the hardware. I want to find out whether a €1,000-class local AI platform can reliably handle my actual home and business workloads. And I’m going to find out by using it myself, every single day.
4
2
11
1,225
5/5. I want to use what I’m going to sell Today, I’m migrating all my main services, databases, Home Assistant, Docker containers, RAG, accounting, and business workflows to DDR4. I’ve published the hardware configuration before, but I’ll share it again separately. The only significant difference from the entry-level customer system is my relatively expensive SAS controller and two 20 TB HDDs in RAID 1 for data storage. My four-drive PCIe 5.0 RAID 0 array, delivering 30–35 GB/s, will be gone. And I’m genuinely curious whether I’ll even notice the difference in everyday work. I don’t want to sell someone a €1,000 server while continuing to use an expensive machine myself because the cheaper one supposedly isn’t good enough. I want to live and work on the same class of hardware I’m offering my customers. If something breaks, I’ll be the first to experience it. If everything works beautifully, that’s far more convincing evidence than any synthetic benchmark. So at the end of 2026, I’m voluntarily moving from DDR5 to DDR4. Not to save money. To test my business idea in real life.
1
8
1,234
4/5. I’m no longer interested in just tokens per second I’ve conducted countless tests measuring generation speed, prefill, long contexts, power consumption, and memory utilization. But customers aren’t buying benchmark records. They need document processing, accounting, search across their own data, automation, AI agents, and dozens of other processes to work reliably. Preferably simultaneously, without constant maintenance. So I’ve decided to move beyond laboratory benchmarks and test these systems under real production workloads. In the coming weeks, I plan to start offering local AI servers for small businesses, starting at €1,000. I’ve chosen DDR4 as the foundation of the entry-level configuration. There will also be more powerful options, for example around €2,500, supporting faster models, more complex workloads, and more simultaneous users. But the €1,000 system must be genuinely useful, not merely capable of launching an AI model. That’s why I’m moving my entire home and business infrastructure onto this class of hardware.
2
9
1,293
3/5. Why did I choose DDR5 in the first place? When I started developing local AI servers for small businesses in winter 2026, the situation was completely different. Local models were larger, less capable, and much slower. GPUs were expensive, and building a practical system without one seemed like a reasonable way to reduce costs. My goal was a €1,500–2,000 system capable of running entirely without a GPU. Small local models would handle documents, automation, filtering information before sending requests to cloud AI, and other relatively simple tasks. Even CPU inference at around 15 tokens per second was acceptable for certain workloads. Customers could always add a GPU later. But everything changed in September and October. Strata arrived, opening new possibilities for distributing model workloads across VRAM, system RAM, and SSD storage. Meanwhile, RAM and GPU prices increased significantly. I started testing DDR4, including relatively old hardware. Then I went even further and tested DDR3. The results confirmed something important: for many practical AI workloads, an expensive modern platform simply isn’t necessary.
1
5
1,377
2/5. Eight months on DDR5 Since winter 2026, my home and business infrastructure has been running on a modern Intel-based server with 64 GB of fast DDR5 memory and four PCIe 5.0 NVMe SSDs. Each drive delivers up to 14 GB/s, and together in RAID 0, they achieve around 30–35 GB/s of actual read throughput. The server runs Home Assistant, AI agents, around 40 Docker containers, ERP, accounting, databases, a vector database for RAG, document storage, backups, and numerous other services. Essentially, it’s the digital brain of my home and business. Only the AI model runs on another machine because this server has no GPU. Everything works beautifully. Except for one thing: despite all these workloads, the CPU is practically idle. Utilization usually doesn’t even reach 5%. An enormous amount of computing power simply goes unused.
2
6
1,443
1/5. Why am I switching from DDR5 to DDR4 at the end of 2026? And it’s not about money. @coldniko Strata changed my understanding of what hardware is actually needed to run local AI. So much so that I’m now moving my entire home and business infrastructure from a modern DDR5 server with ultra-fast PCIe 5.0 NVMe drives to a much older DDR4 platform. I already own both DDR5 and DDR4 hardware. I’m not trying to save money. My current server works beautifully, and I’m completely satisfied with it. So why am I voluntarily downgrading? Here’s the story.
7
6
1
90
12,659
Don’t worry, I’ll probably publish a very negative post about local AI soon, once I’ve accumulated enough frustrations to write about. But that doesn’t mean local AI is unnecessary. Quite the opposite. It proves why we need it, while also reminding us that it’s a tool with both advantages and disadvantages. And there are an enormous number of advantages. As for people spending too much money on hardware, how is that different from any other hobby? Some people spend fortunes on cosmetics and clothes. Others collect watches, cars, or all sorts of things. Some enthusiasts pour incredible amounts of money into smart homes, solar panels, computers, and countless other projects. Everyone has their own obsession. What seems like an absurd expense to one person can be a source of genuine happiness to another. And if someone enjoys the process, I think that’s perfectly fine, even if they never achieve any practical result. As for people not sharing their negative experiences, that’s perfectly understandable too. Nobody wants to look foolish in front of others, and many are understandably afraid of criticism, ridicule, or online hate. I think all of this is completely normal. It’s simply human nature, and we see the same behavior in practically every field. Yet despite all these imperfections, people keep experimenting, learning, making mistakes, and sharing discoveries. And together, humanity keeps moving forward. The progress we’re witnessing is absolutely incredible.
Replying to @uctive
I would be there are many times more people they bought hardware and never did anything worthwhile. X is full of people going crazy but no one posts negative stories.
1
1
493
5/5 🏁 CONCLUSIONS & NEXT RESULTS What did we actually learn from this express benchmark? ✅ 1. A dense 27B model can run successfully on a laptop with 24 GB of VRAM. 🚀 2. Basalt v0.4.1 with DFlash 2 achieved up to 121 tok/s. ✅ 3. Performance remained reasonably high through 131K context tokens, with DFlash 2 reaching 86.6 tok/s at 131K. 📊 4. Plain decoding demonstrated remarkably stable generation speed throughout individual responses. ⚡ 5. DFlash 2 can accelerate significantly within a single response, even at long context lengths. 🐢 6. A 256K context is technically possible, but partial KV offloading into system RAM reduced generation speed to approximately 5 tok/s. 🔌 7. VRAM capacity alone is not enough. Real memory bandwidth, KV placement, and CPU→GPU interconnect performance can determine the practical limits of local inference. ━━━━━━━━━━━━━━━━ 🎯 THE 24GB VRAM CONTEXT LIMIT — NOW MUCH CLEARER We already know approximately how much context fits entirely into the GPU’s 24GB VRAM, based on the engine’s exact allocation requirements and available memory. 📦 Plain decoding: ~221K tokens 🚀 DFlash 2: ~173K tokens The approximately 2 GiB of additional memory required by DFlash 2 costs around 48K positions of GPU-resident context capacity. These are capacity estimates derived from measured allocation failures, not yet verified sustained-speed results at the exact boundaries. The performance transition around these limits is still being investigated. ━━━━━━━━━━━━━━━━ 🧪 TESTING IS STILL IN PROGRESS We are currently completing the final measurements: 🔹 ~221K context without DFlash — speed near the fully GPU-resident limit. 🔹 ~173K context with DFlash 2 — speed near its own VRAM capacity limit. 🔹 Targeted retrieval of control facts at 128K and 256K. 🔹 Final performance tables and four charts. 🔹 A complete technical report in Russian and an English README. The results already establish the approximate fully GPU-resident context limits. The remaining measurements will show how fast the model actually runs near those boundaries. We expect to add a follow-up post to this thread within the next couple of hours, as soon as the remaining tests and final report are completed. ━━━━━━━━━━━━━━━━ 🧪 FINAL REMARKS This was an express benchmark, not an exhaustive scientific investigation. Our objective was to measure actual performance under real hardware constraints, including both successful and unsuccessful results. We are publishing the methodology, scripts, and available raw logs so that anyone interested can: 🔎 Inspect the measurements 🔁 Reproduce the experiment 🛠️ Test alternative configurations 📊 Compare results on different hardware 💬 Challenge our conclusions Additional code-analysis and structured-JSON results are also awaiting final verification. 📌 BENCHMARK CONFIGURATION Engine: Basalt v0.4.1 Commit: d3b7574 Model: Qwen3.8-27B-QUASAR-NVFP4 GPU: RTX 5090 Laptop 24GB KV: int8 Concurrency: 1 📂 GITHUB — SCRIPTS & RAW LOGS github.com/ViktorIXAI/basalt… The final technical report and additional verified results will be added to the repository and this thread after testing is complete.
1
70
4/5 🔬 MEMORY BOTTLENECKS, CORRECTNESS & REAL-WORLD WORKLOADS 🔌 WHY PERFORMANCE COLLAPSED The GPU supports PCIe 5.0 x16 at 32 GT/s per lane, but the actual negotiated connection was PCIe 4.0 x8 at 16 GT/s per lane. Measured CPU→GPU bandwidth: 14.4 GB/s. Once a substantial part of the KV cache resides in system RAM, generating a token may involve transferring approximately 2.1–4.3 GiB of KV data across this connection. The bottleneck shifts from GPU computation to data transfer between RAM and the GPU. 🚀 DFlash 2 speedup at 131K: 3.2× 🐢 DFlash 2 speedup at 256K: 1.09× Speculative decoding can do very little to compensate for insufficient memory-transfer bandwidth. This illustrates why VRAM capacity, KV-cache placement, and actual interconnect bandwidth can matter more than theoretical GPU compute performance. ━━━━━━━━━━━━━━━━ 🔎 LONG-CONTEXT CORRECTNESS Our initial attempt to retrieve five control records from 131K and 256K contexts returned: ❌ 0/5 correct records. However, inspection of the benchmark script revealed that the follow-up retrieval request did not include the original long document. Therefore, those results cannot be considered a valid measurement of the model’s long-context retrieval capabilities. We are already performing targeted checks of the control facts at 128K and 256K using a corrected approach. These results will be included in the final report. ━━━━━━━━━━━━━━━━ 🔀 PARALLEL WORKLOADS — PRACTICAL EXPERIENCE One of our followers asked how Basalt performs when multiple requests run simultaneously. We did NOT benchmark that in this experiment. All measurements used a single concurrent request. However, we have been running Basalt in our own system for about a week. Subjectively, its behavior under parallel workloads is approximately the same as Strata. ✅ Tasks generally get completed. ⚠️ But fast responses should not be taken for granted with default settings. 📺 A SLIGHTLY AMUSING REAL-WORLD EXAMPLE On one occasion, the command “turn on the TV” took approximately 15 minutes to execute. The command passes through our Hermes agent, which uses a local model running on Basalt or Strata, and then forwards the action to Home Assistant. The computer was already heavily loaded with other tasks. Of course, this does NOT mean the model itself needs 15 minutes to turn on a television. This is a shortcoming of my own system architecture, including scheduling, task queues, and priorities. For now, I don’t consider it a serious problem. The system is experimental, and we are constantly modifying it. But it demonstrates an important practical lesson: Even relatively powerful local AI hardware does not automatically guarantee instantaneous responses under parallel workloads. ⚙️ Fast response times require deliberate architecture: • Task prioritization • Resource allocation • Proper scheduling • Separate handling of urgent commands These are system-design issues, not simply model-performance issues. And this is precisely why real-world experimentation is so useful.
1
97
3/5 📊 GENERATION DYNAMICS & THE REAL VRAM LIMIT We divided each generated response into four consecutive segments, each containing 25% of the output tokens, and measured generation speed separately. ⚠️ These measurements describe OUTPUT generation, not the processing of different sections of the input context. 🚀 DFLASH 2 — TOKENS/SECOND 66K: 79 → 99 → 106 → 100 131K: 77 → 90 → 95 → 89 256K: 4.6 → 5.2 → 5.7 → 5.2 📈 At 66K, generation speed increased by approximately 34% between the first and third quarters. 📈 At 131K, the increase was approximately 23%. One possible explanation is warm-up behavior in the speculative drafting and verification pipeline. This remains a hypothesis rather than a proven explanation. 🔄 PLAIN DECODING 131K: 27.3 → 27.0 tok/s from beginning to end. 256K: 4.7 → 4.8 → 4.7 → 4.7 tok/s. Almost perfectly stable performance, although at 256K the absolute speed was already very low. 💡 Average tokens per second can hide interesting performance dynamics. 🌡️ ACTUAL GPU TELEMETRY ⚡ Power: 173–174 W 🌡️ Temperature: 68–70°C 🕒 SM clock: ~1.96 GHz 🔒 Software power limit: ACTIVE ✅ Thermal throttling: NOT DETECTED 📦 Model weight loading: ~2.9 s 🚀 Server ready: 12–15 s The GPU was power-limited, not thermally limited. ━━━━━━━━━━━━━━━━ 🚨 THE 256K PROBLEM — AND THE VRAM CAPACITY LIMIT At 131K, the KV cache remained entirely in GPU memory. DFlash 2 delivered 86.6 tok/s. At 256K, the complete KV cache no longer fitted in VRAM. We now have a much more precise estimate of the maximum fully GPU-resident context, based on the engine’s actual memory-allocation failures. 📦 WITHOUT DFLASH (–spec 0) At 224,000 KV positions: • Required KV memory: 9,575 MiB • Available VRAM: 9,487 MiB • Estimated fully resident limit: ~221K tokens 🚀 WITH DFLASH 2 (–spec 16) At 224,000 KV positions: • Required KV memory: 9,575 MiB • Available VRAM: 7,421 MiB • Estimated fully resident limit: ~173K tokens ⚠️ DFlash 2 consumes approximately 2 GiB of additional VRAM for its block-drafting mechanism. That costs roughly 48,000 positions of GPU-resident context capacity. So the practical VRAM capacity estimates are: ✅ Plain decoding: ~221K tokens ✅ DFlash 2: ~173K tokens These values are derived from actual engine allocation errors, not GPU specification sheets. Performance at these boundaries is being measured separately. Beyond these limits, the KV cache must be partially offloaded into system RAM. At 256K, this resulted in a dramatic performance collapse. 📉 256K — PLAIN DECODING • GPU-resident KV: 196,608 positions • Prefill: 2,030 tok/s • TTFT: 126.4 s • Decode: 4.7 tok/s • Peak VRAM: 23,068 MiB • Peak RAM: 8.5 GiB 📉 256K — DFLASH 2 • GPU-resident KV: 131,072 positions • Prefill: 2,010 tok/s • TTFT: 127.9 s • Decode: 5.1 tok/s • Draft acceptance: 77% • Peak VRAM: 23,482 MiB • Peak RAM: 10.9 GiB ⚠️ From 86.6 tok/s at 131K to just 5.1 tok/s at 256K with DFlash 2. The reason is not mysterious. The bottleneck has moved from computation to memory transfer.
1
98
2/5 🖥️ HARDWARE, METHODOLOGY & PERFORMANCE We ran an express benchmark of a dense 27-billion-parameter model on an NVIDIA RTX 5090 Laptop GPU using Basalt v0.4.1. Five context lengths, plain decoding vs DFlash 2, prefill, TTFT, and generation speed across four quarters of each continuous response. All figures are actual measurements, not theoretical hardware specifications. 🔧 HARDWARE 🎮 GPU: RTX 5090 Laptop — 24,463 MiB VRAM 🧠 CPU: AMD Ryzen 9 9955HX — 16C/32T 💾 RAM: 64 GiB 💽 Storage: NVMe SSD ⚙️ NVIDIA driver: 615.71.09 ⚙️ CUDA: 13.0 🌡️ IMPORTANT CPU NOTE We deliberately limited the Ryzen’s maximum frequency from approximately 5.5 GHz to 4.3 GHz. The goal was to improve thermal comfort during prolonged laptop operation, particularly by reducing RAM module temperatures. We have never recorded critical overheating. We simply decided to eliminate uncomfortable heat. Actual median CPU frequency during testing: 4.33 GHz. This is a laptop. Maximum performance at any cost is not our objective. 🔌 MEMORY & INTERCONNECT • Supported: PCIe 5.0 x16 — 32 GT/s per lane • Actual: PCIe 4.0 x8 — 16 GT/s per lane • Measured CPU→GPU: 14.4 GB/s • Measured RAM read: 58.6 GB/s ⚙️ SOFTWARE • Basalt v0.4.1 • Qwen3.8-27B-QUASAR-NVFP4 • Dense 27B / NVFP4 quantization • Model package: 21.15 GB • GPU-resident weights: 18.4 GiB • KV cache: int8 • Temperature: 0 • Thinking: OFF • Concurrency: 1 Two decoding modes: A — Plain (–spec 0) B — DFlash 2 (–spec 16) 🚀 DECODE SPEED — TOKENS/SECOND Context Plain DFlash 2 4K 38.3 100–121 32K 35.0–35.1 105.5–113.1 66K 31.6 91.3–94.1 131K 27.0 86.6 256K 4.7 5.1 ⚡ DFlash 2 speedup: 4K → 2.6–3.2× 32K → 3.0–3.2× 66K → 2.9–3.0× 131K → 3.2× 256K → only 1.09× 📥 PREFILL — TOKENS/SECOND 4K → 5,581–7,417 32K → 5,577–5,739 66K → 4,439–4,524 131K → 3,182–3,249 256K → 2,010–2,030 ⏱️ TIME TO FIRST TOKEN (TTFT) 4K → 0.59–1.15 s 32K → 5.6–5.9 s 66K → 14.6–14.9 s 131K → 40.6–41.4 s 256K → 126.4–127.9 s ✅ Up to 131K, performance is quite satisfactory for our purposes. But average tok/s tells only part of the story. What happens during the generation of a single long response is even more interesting.
1
192