Arvind

Reimagined the @Zineps_ai landing page for #ReimagineZineps 🚢 Live: reimagine-zineps.vercel.app One file of hand-written HTML, CSS and JS: an interactive rate engine, an animated "intelligent layer" diagram, a Shipping AI replay and a Workflows canvas. Dark, mint, fast. @iBhanuDahiya call me 📞

35/35 The whole system in one diagram: streaming ingestion, hybrid retrieval, rerank, generation, caches, evals, and tracing. None of it is exotic; all of it is work. Build the boring parts and the demo becomes a product. That's the thread.

34/35 Failure modes to design for: lost-in-the-middle (models skim mid-context), stale index, retrieval drift after re-embedding, prompt noise from over-stuffing, silent empty retrievals. Each has a cheap detector. Add them before users find them.

33/35 Trace every request end to end: spans for rewrite, retrieval (log the chunk IDs!), rerank, and generation. When a user reports a bad answer next week, you replay the exact context that produced it instead of guessing what the index looked like.

32/35 Multi-tenant RAG is a spectrum. Shared index + metadata filter: cheap, one bug from a data breach. Namespace per tenant: the usual balance. Dedicated index per tenant: bulletproof isolation, highest cost. Getting this wrong isn't a bug, it's an incident.

31/35 Freshness is a feature users notice. Wire CDC from your source-of-truth DB into the embedding pipeline and upsert continuously. Then measure staleness (event time to searchable time) as a first-class SLO. "The index is stale" is the real RAG outage.

30/35 Quantization is how big indexes fit in RAM. 100M vectors at 768 dims: float32 307GB, int8 77GB (~zero loss), product quantization ~6GB (recall dip, rescore the top), binary ~10GB (needs rescoring). Compress in RAM, keep full vectors on disk.

29/35 The cost model at scale: embeddings are one-time per doc, index RAM/storage is always-on, rerank and LLM tokens are per-query, and the per-query column dominates. Shrink prompts before you shrink models: context tokens are the bill.

28/35 A p95 RAG request, roughly: query embed 10ms, ANN 20ms, rerank 120ms, LLM generation 2000ms+. Generation dominates everything. So stream tokens, parallelize retrieval, and stop micro-optimizing the vector search: it is not your bottleneck.

27/35 Cache the whole stack. Embedding cache: same text, same vector. Retrieval cache: hot queries skip ANN. Semantic answer cache: a similar-enough question reuses a whole answer. In real traffic the semantic cache absorbs a shocking share of load.