🎉 New Series Production Debugging Playbook Part 1/6 : Debugging High Latency Your API is responding slowly. Before increasing CPU, adding database indexes, or scaling your service, find out where the time is actually going. Here's how I'd approach this as a production engineer. 1. Start with latency percentiles Average latency can hide serious problems. If most requests take 100 ms but a small percentage take 2 seconds, the average might still look acceptable while some users experience a painfully slow application. • p50 (median): 50% of requests complete within this time. It represents the typical request. • p95: 95% of requests complete within this time. The slowest 5% take longer. • p99: 99% of requests complete within this time. The slowest 1% take longer. Compare these percentiles over time and break them down by endpoint, region, service version, or customer segment where relevant. If p50 stays stable but p99 spikes, investigate what makes a subset of requests slow. If all percentiles rise together, look for a broader bottleneck. 2. Trace a slow request Metrics tell you that latency increased. Distributed traces help you understand where the time is spent. Imagine a request passing through an API gateway, user service, product service, and database. A distributed trace breaks the request into spans, showing the duration and relationships between operations. Look for the slowest spans, repeated downstream calls, retries, and time spent waiting for dependencies. One important detail: don't simply add every span duration together. Parent spans often include the time spent in their child spans, and parallel operations can overlap. Use the trace timeline to understand the actual critical path. 3. Identify the real bottleneck Once you know which component is slow, investigate the reason. • Application: Check CPU saturation, thread or worker exhaustion, garbage collection, and slow external calls. • Database: Separate query execution time from connection acquisition waits. Inspect slow queries, execution plans, locks, and connection pool utilization. • Network or dependencies: Check DNS resolution, connection establishment, timeouts, retries, and downstream service latency. • Caching: Inspect cache hit rates, misses, eviction patterns, and cache response times. Don't assume that a slow database span means the SQL query itself is slow. The application might be waiting for a connection, a lock, or a response across the network. Use logs for request-specific context, metrics to understand system behavior, and traces to connect the stages. Correlate them using timestamps, trace IDs, and relevant deployment events. 4. Validate your hypothesis before changing anything Suppose p99 latency jumps from 300 ms to 2.4 seconds after a deployment, but the error rate remains stable. A trace shows that the request spends most of its extra time waiting for a database connection. Pool metrics confirm that all connections are occupied, while query execution time remains normal. Now you have evidence for connection pool exhaustion, not a slow SQL query. Investigate why the pool is exhausted, apply a targeted fix, and compare the same signals before and after. Verify that latency recovers without increasing database load or introducing new errors. The principle to remember: Observe the symptom. Locate the delay. Test the hypothesis. Verify the fix. Senior engineers don't debug by trying every possible solution. They use evidence to eliminate possibilities until the root cause becomes clear. Next in the series: Debugging 5xx Errors & Failed Requests.