David Balbin

Brilliant paper by NVIDIA. they found a way to make KV cache transferable between models. the target model skips prefill entirely, and the conversion runs 2.7 to 25x faster than processing the context again. let's understand why this is so important today. LLM APIs are stateless, so every turn sends the entire conversation back to the model. the model reads it again before generating a new token, and all of it is billed as input. prompt caching lets providers hold the KV cache for a stable prefix and bill cache hits at roughly 10% of the base input rate because that compute was already done. this 90% reduction is one of the biggest levers in LLM serving. but the cache only works on the model that produced it. since keys and values depend on that model's weights, another model cannot directly reuse them. this becomes a problem with LLM routing. if traffic moves to another model for cost or capability reasons, the accumulated KV cache becomes invalid. the entire context must be processed again and billed at full rate. NVIDIA's recent paper treats this as a representation problem. prefill's main output is the KV cache, so transferring it means converting one model's cache into the representation another model expects. they first checked whether this conversion had enough structure to exploit. for Qwen3 14B → 32B, a linear regression from a single source layer reconstructed 56% of the variance in the target model's keys. because models can have different layer counts, there is no natural one-to-one layer mapping. so, for every target layer, they rank the source layers by predictive power and use the top eight together. that pushes reconstruction to 79%. the mapper has three parts: > each target layer and head gets its own linear map, solved in closed form instead of with gradient descent. > cross-layer selection chooses the source layers that best predict each target layer. their ablation shows this contributes the most. > RoPE adds a position-dependent rotation to keys. they remove it, fit the mapping in position-free space, then apply the target model's rotation again at inference. across six model pairs from Qwen3, Llama 3.1, and Ministral 3, four retain 73 to 98% of the receiving model's standalone accuracy. conversion is also 3 to 25x faster than processing the context again. prior cross-model KV reuse methods either train a neural adapter for every pair or require architecturally identical models. this approach is closed-form and training-free, but there are still important limitations. every tested pair belongs to the same model family, such as Qwen → Qwen or Llama → Llama. cross-family transfer is still future work. the tested pairs also share KV head count and per-head dimension. mismatched configurations remain untested. and the work currently covers dense full-attention models only, not sliding-window or attention-recurrent hybrids. link to the paper: arxiv.org/abs/2608.03893 i wrote a first-principles breakdown of how KV caching works the article is quoted below.

Zip files for AI/ML projects! KitOps revolutionizes how teams package, version, and share AI/ML projects by creating a ModelKit, which is like a zip file containing the entire AI project—model, data, code, and settings. 100% open-source, Zero MLOps headaches.