milo

Why KV cache stores K and V vectors but never Q? (a popular technical LLM interview question) LLMs are autoregressive so each token is predicted from every token before it, one at a time. This autoregressive nature has a direct consequence inside the model. A forward pass over <n> tokens produces <n> hidden states, but only the last one is projected to logits and is required to generate the next token. So to understand why KV cache just stores K and V vector, we must back track to see how exactly is the last hidden state produced. Let's walk through this with a 10-token prompt. 1) Prefill: All 10 tokens go through the model in one forward pass, in parallel (with causal masking), since the whole prompt is already known. At every layer, each of the 10 positions produces a query, a key and a value vector, and attention at each position runs against all positions up to it. This pass is compute-heavy, and it's why the first token takes noticeably longer than the ones after it. TTFT is mostly prefill. 2) The first output token: To generate the 11th token, only the 10th token's hidden state is needed. So this is projected from the hidden-dim to vocab-dim to generate logits over vocab. These logits then go through softmax and sampling to generate token 11. 3) Back-track the hidden state: The last hidden state is the last row of the feedforward block's output. The feedforward block is position-wise (it's applied to each row independently) so that row comes from the last row of the attention output before it. So now we need to see how the last row of attention is computed. 4) Attention matrix: QKᵀ for a 10-token prompt will give a 10 × 10 matrix. Row <i> will have the dot product of query <i> with every key. Row 10 is therefore Q₁₀·K₁, Q₁₀·K₂, all the way to Q₁₀·K₁₀. Notice that only Q₁₀ appears in it. Q₁ through Q₉ only belong to their corresponding rows 1-9, and those rows' hidden states we already discarded because they were never needed. The last row of attention goes through softmax and multiplies the full stack of value vectors, V₁ through V₁₀, to give the last row of the attention output. So the last hidden state depends on exactly three things: Q₁₀, every key, and every value. 5) Generating token 12: Token 11 is appended, and this time, we need row 11's hidden state to generate token 12. Mathematically, attention operation turns out to be Q₁₁ against K₁ through K₁₁, then multiplied by V₁ through V₁₁. K₁ through K₁₁ and V₁ through V₁₁ are bit-for-bit what prefill + first token produced since under causal masking, a token's key and value depend on that token and the ones before it, never on anything after, so appending token 11 cannot change anything at position 3. 6) The cache state: Overall, this implies that you just need to retain the keys and values at each decoding step, and compute only the new position's Q, K and V. Each decode step requires one query vector, which is never used again, so they are never cached across the decoding process. The visual below explains the entire process. That said, KV cache is only one of four separate caching layers in an LLM stack. The other three are prefix caching on the server, prompt caching billed by a provider, and a semantic cache that skips the model entirely. I wrote a full breakdown of all four caches in LLM serving that you should know as an AI engineer, with code for each. Read it below.

I have been fine-tuning LLMs for over 2 years now! Here are the top 12 techniques I'd learn if I were to fine-tune them: Bookmark this. 1. LoRA > Freezes the base weights and trains two low-rank matrices as the update, resulting in ~95-99% fewer params to fine-tune. 2. QLoRA > LoRA on top of a 4-bit quantized base model. 3. Prefix tuning > Prepends trainable vectors to keys and values at every layer, weights frozen. 4. Adapter tuning > Inserts small trainable modules between transformer layers. 5. Instruction tuning > Supervised tuning on (instruction, response) pairs so the model follows directions instead of just continuing text. 6. P-tuning > Optimizes continuous prompt embeddings through a small encoder, mainly for NLU tasks where discrete prompts are unstable. 7. BitFit > Trains only the bias terms, ~0.08% of params, and still rivals full fine-tuning on small-to-medium datasets. 8. RLHF/RLAIF > RLHF trains a reward model on human preference rankings, then PPO against it. The pipeline behind the first ChatGPT. > RLAIF swaps the human labeler for an LLM judging. RLHF-level quality at a fraction of the cost. 9. DPO (Direct Preference Optimization) > Skips the reward model and optimizes preference pairs directly with a classification-style loss. Simpler than PPO. 10. GRPO (Group Relative Policy Optimization) > Samples a group of responses per prompt and normalizes their rewards within the group. DeepSeek R1 ran on it. 11. RLVR (Reinforcement Learning with Verifiable Rewards) > Replaces the learned reward model with a checker or compiler returning verifiable scores. The free signal behind R1's math and code. 12. Federated fine-tuning > Tunes across decentralized clients that share only weight updates, never raw data. For when data can't leave the device. GRPO needs exactly one scalar reward per response. RLVR (13) produces that for free on math and code by running the answer through a checker or compiler. But tasks like a RAG answer, a support reply, or a summary have no gold label to match against. The usual fallback is a hand-written reward function scoring faithfulness, hallucination, and completeness. It takes days to calibrate, rewards the wrong behavior when the weights are off, and breaks every time you add a tool or edit the system prompt. RULER, implemented in OpenPipe's ART (open-source), solves this. During training, it passes the N sampled trajectories to a judge LLM, which ranks them relative to each other against the agent's system prompt and returns the scores. Relative ranking is more stable than absolute scoring, and GRPO normalizes within the group anyway, so the rankings feed straight into the pipeline like with RLVR. Here's the GitHub Repo: github.com/OpenPipe/ART (don't forget to star it ⭐ ) I wrote a full breakdown recently on how exactly this works, with the training loop and code. Read it below.

NVIDIA researchers did it again! They found a way to make KV cache transferable between models. The target model skips prefill entirely, and the conversion runs 2.7 to 25x faster than processing the context again. Let's understand why this is so important today. LLM APIs are stateless, so every turn sends the entire conversation back to the model. The model reads all of it again before writing a single new token, and all of it is billed as input. Prompt caching allows Anthropic and other providers to hold the KV cache for a stable prefix and bill a hit at roughly 10% of the base input rate, because the compute was already done once. The 90% reduction is one of the largest lever in LLM serving, which is why so much production work goes into keeping prefixes byte-stable. But the cache only works on the model that produced it. Keys and values are produced from that model's weights, so no other model can read them. In pratice, the constraint shows up in LLM routing. If the traffic is shifted to a different model for cost/capability reasons, the accumulated KV cache becomes invalid. As a result, the accumulated context has to be processed from scratch, and it's billed at full rate. NVIDIA's recent paper treats this as a representation problem. Prefill's only output is the KV cache, so to move KV between models, we need to convert one model's cache into the format the other expects. They first checked whether the conversion has any structure worth exploiting. They found that moving from Qwen3 14B to 32B, a plain linear regression from a single source layer reconstructed 56% of the variance in the target model's keys. The two models obviously may have different layer counts, so there is no natural one-to-one pairing between them. For each target layer they rank every source layer by how well it predicts that layer, then feed the top eight in together, which takes the reconstruction to 79%. The mapper itself has three parts: > Each target layer and head gets its own independent linear map, solved in one closed-form step rather than by gradient descent. > The cross-layer selection described above is the second part, and their ablation shows it carries the most weight of the three. > Keys also carry a position-dependent rotation from RoPE. They strip that rotation, fit the map in position-free space, then re-apply the target model's rotation at inference. Across six pairs from Qwen3, Llama 3.1 and Ministral 3, four retain 73 to 98% of the receiving model's standalone accuracy, and the conversion runs 3-25x faster than processing the context again. Prior work on cross-model KV reuse exists, but it either trains a neural adapter per pair or requires both models to be architecturally identical. This is probably the first version that is closed-form and training-free, so a lot of it is still open research. Every pair tested belongs to one family, so it works on Qwen to Qwen and Llama to Llama. Cross-family transfer is listed as future work. All six pairs mentioned above also happen to share KV head count and per-head dimension across scales. Mismatched head configurations are currently untested. The researchers scoped this to dense full-attention only, so sliding-window and attention-recurrent hybrids still need work. Here's the paper: arxiv.org/abs/2608.03893 Plenty of work is yet to be done. Still, the constraint being solved is genuine. Every model swap currently invalidates the full KV that was already paid for, and this is the first result showing that work might be recoverable without training anything extra. That said, all of this only matters because of what the KV cache is doing in the first place. I wrote a first-principles breakdown of it, covering why the model stores keys and values at all, why the cache grows with every token, and what generation speed looks like with and without it. Read it below.