William Morgan

By the way, building inference infrastructure is pure crazy town if you're coming at this from the service mesh perspective. Basically all the things we've been doing for years—load balancing, routing, proxying—are still major problems to be solve. BUT all the details are totally different and you have to revisit all your assumptions. For example, we've been load balancing requests in @Linkerd for a decade. We have always been latency aware: we measure the moving average of latency for each endpoint, we weight it exponentially to favor more recent data, and then we use the "power of two choices" algorithm to select the endpoint for a request very quickly. We do it in this particular way because Linkerd's load balancing is designed to handle a huge amount of traffic going to a huge number of very fast endpoints and to be responsive to changes in small #'s of milliseconds. So we have to be very efficient in how we make every decision. For inference, proper load balancing makes a HUGE difference but all the details are all completely different. Your endpoints now take, say, 3000ms or even 30s instead of 30ms—and that's just to get the first token. They can take *minutes* to complete the request. So requests are dramatically slower and there's a lot less of them. Your decision-making budget is just *luxurious* compared to where we were balancing microservices requests. We take advantage of this in Kinference to do KV-cache-aware load balancing. Inside Kinference we maintain a model of the KV cache content of each vLLM worker. This comes in as a set of block hash ids from each vLLM instance, over ZeroMQ. When a new prompt comes in (i.e. over HTTP), we tokenize it, chunk it into blocks, hash them, and use that compute the exact amount of prefix overlap with each vLLM instance we're tracking. Then we build an expected TTFT for each instance for that prompt and send it to the fastest one. Seems like a lot of work, right? And this whole process is actually pretty fast except for the tokenization step, where we literally call out to vLLM over the network (!!!) and say, hey, please tokenize this. This can take... 50ms? 80ms? More? From the Linkerd perspective this whole setup is utterly *insane* (really, you want to make an inline synchronous HTTP call to another system just to make a load balancing decision???) and the amount of time it takes is laughable ... but it doesn't matter! We have that time and more. And we do it because hitting the GPU with the right KV cache content can mean the difference between a 3s TTFT and a 30s TTFT. Or more! It's crazy! Again, these numbers are, like, all bonkers, absolutely bonkers from the service mesh perspective. We did a little benchmarking to show how nuts this all is. We compared balancing inference requests to a set of 3 vLLM pods, using Kinference on one hand and a regular Kubernetes service on the other. Even in this very basic setup where Kinference is spending live-request-path time interrogating other systems over HTTP calls, we were able to show a 95% reduction in median TTFT! And a 7 X increase in capacity! Obviously that is workload-specific but all these numbers are just bonkers. So there you have it: use Kinference and even without any of the fancy policies or model aliases or bells and whistles, you can drop your TTFT by probably astonishing amounts (depending on your exact workload) vs plain Kubernetes balancing. buoyant.io/blog/distributed-…