Julián Cortés

Most developers know that avg latency is misleading and they should use percentiles instead. They also know that if 90tile response time is 200ms, then 10% of the requests had response times higher than 200ms. Well, the last sentence is false. Or rather, true in theory. The only way to calculate true percentiles is by sorting and scanning all the data. Since this often means many millions of datapoints, we actually don't do that. We approximate. With histograms. How does that work? You create buckets of values: 0 to 0.05, 0.05 to 0.25, 0.25 to 1, 1 to 5, 5 to infinity. Then you accumulate: number of data points, sum of values and the number of data points that fall in each bucket. Now, if I ask you for the 90tile value, you start going over the buckets from lowest to highest latencies, and add the number of points in each bucket. Once you go past 90% of the data points, you know that your 90tile value is in this bucket. Then you give either the middle of the bucket or the upper bound as the estimate. Note that you don't know what % of the data is higher than the estimate. And also note that the quality of the estimate really depends on how the data ends up distributing between the buckets. It isn't uncommon for ntiles 90, 95 and 99 to all fall in the same bucket (if your distribution is "heavy tail"). And sometimes all or most of the data falls in one bucket and you can tell nothing. I recently had a case where we noticed the 99tile of response time for a specific request type was 10s. This was both high and suspicious - we have a 10s timeout in some places. But we couldn't find any request that hit these timeouts in our logs. Super weird! Until a smarter colleague realized (by looking at sample traces) that 10s was just the lower bound of the top bucket. The requests actually timed out at 60s. This allowed me to find the timeout they actually hit and fix the issue. So the lesson is: Averages are misleading. But percentiles are usually approximate and can be misleading too! Be careful not to trust the data blindly. #StatisticsSaturday