API p95 Latency and SLOs in Prometheus: Aggregating Histograms with PromQL

Aggregate API latency across replicas with Prometheus classic histograms. Estimate p95 with histogram_quantile, measure the fraction of requests within 300 ms, and handle bucket boundaries, counter resets, and missing series.

If an API runs on several replicas, their individual p95 values cannot tell you the p95 across all requests. A busy replica accounts for more of the traffic than a quiet one. In Prometheus, combine the bucket counts across replicas and calculate p95 once from the combined distribution. See the introduction to latency percentiles for how to interpret the resulting quantile.

Check the buckets, request counts, and units

The examples use a classic histogram named http_request_duration_seconds. They assume instrumentation records one duration in seconds per request. Labels such as service, job, and instance depend on your setup.

SeriesValue
http_request_duration_seconds_bucket{le="0.3"}Cumulative number of observations at or below 0.3 seconds
http_request_duration_seconds_bucket{le="+Inf"}Unbounded bucket with the cumulative count of all observations
http_request_duration_seconds_countCumulative count of all observations
http_request_duration_seconds_sumCumulative sum of observed durations in seconds

The le boundary is inclusive. A 100 ms observation contributes to both le="0.1" and le="0.3", so adding those buckets would count it twice. To count observations in (0.1, 0.3], subtract the smaller cumulative count from the larger one. Use the same bucket boundaries and units across the replicas you plan to aggregate. See the Prometheus metric type documentation .

Calculate p95 across the service

histogram_quantile(
  0.95,
  sum by (le) (
    rate(http_request_duration_seconds_bucket{service="checkout"}[5m])
  )
)
  1. rate(...[5m]) estimates the per-second increase in each bucket counter over five minutes. Apply it before aggregation so Prometheus can handle resets in each individual series.
  2. sum by (le) adds the rates across replicas for each boundary. To return separate results per service, use sum by (service, le) and select the intended services in the metric selector. Check that your selector does not include multiple measurements of the same request.
  3. histogram_quantile(0.95, ...) estimates p95 for the combined distribution in seconds. Multiply by 1,000 if you want the output in milliseconds.

The five-minute range looks backward from each evaluation timestamp. A range that is too short relative to the scrape interval can leave some series without enough samples for rate. On low-volume services, percentiles can also fluctuate because they describe very few requests. Plot the request volume alongside them. See the function references for histogram_quantile and rate .

Work through two replicas

Assume the counters grow at steady rates between scrapes. These are the five-minute rate values.

ReplicaWithin 100 msWithin 300 msWithin 500 msTotal
A88 req/s90 req/s90 req/s90 req/s
B0 req/s2 req/s10 req/s10 req/s
Combined88 req/s92 req/s100 req/s100 req/s

The 95% rank corresponds to 95 req/s. It lies between the cumulative rates of 92 req/s at 300 ms and 100 req/s at 500 ms. Classic histogram interpolation yields 300 + (500 − 300) × (95 − 92) / (100 − 92) = 375 ms. If all eight requests per second in that bucket actually took 310 ms, the true p95 would be 310 ms and the histogram estimate would still be 375 ms.

The two replicas have estimated p95 values of about 97 ms and 488 ms. Their arithmetic mean is about 292 ms, and their request-weighted mean is about 136 ms. Both differ from the combined histogram estimate of 375 ms. The counts within each bucket are what you can add across replicas.

Measure a 300 ms SLO with a bucket ratio

For an objective like “99% of eligible requests finish within 300 ms,” divide requests within the threshold by all eligible requests. When the threshold aligns with a bucket boundary, you can compute this from the counter rates without estimating the distribution inside a bucket.

sum(rate(http_request_duration_seconds_bucket{service="checkout",le="0.3"}[5m]))
/
sum(rate(http_request_duration_seconds_count{service="checkout"}[5m]))

The example gives 92 / 100 = 0.92, or 92% over five minutes. If the SLO uses a 30-day window, sum increase(...[30d]) for good events and total events separately before dividing. A simple average of five-minute ratios gives quiet windows the same weight as busy ones. If only successful requests are eligible, apply the same definition and filter to numerator and denominator. If errors count as SLO failures, count all eligible requests in the denominator and successful requests within the threshold in the numerator.

Distinguish boundary mismatches, missing data, and zero traffic

  • Different boundaries across replicas. If only some replicas export le="0.3", the numerator omits the others and the compliance ratio is biased downward. Use consistent boundaries and keep cohorts with different boundaries separate during a rollout. If no replica exports the threshold bucket, the numerator is an empty vector.
  • Zero requests or missing series. When the counters exist but there are no requests, the ratio is 0/0, or NaN. The p95 is also NaN. An absent input series may instead yield an empty result. Plot request volume alongside the ratio and check scrape health with up or monitored target counts. Partial data loss can still leave a plausible-looking aggregate.
  • p95 plateaus at the largest finite boundary. If the quantile falls inside the +Inf bucket, histogram_quantile returns the largest finite boundary for a classic histogram. When that boundary is 500 ms, a result of 500 ms alone cannot tell you how far above it the slow requests were. Subtract the last finite bucket from the +Inf bucket to measure traffic beyond the boundary.

Include the SLO threshold as a boundary in the instrumentation. For p95 estimation, add useful boundaries around the range where the quantile typically lands to reduce uncertainty from interpolation. See the Prometheus guide to histograms and summaries .

FAQ

How do I calculate p95 across multiple replicas?

Apply rate to each bucket series, aggregate with sum by (le), and then run histogram_quantile. To return one value per service, use sum by (service, le).

How do I measure the fraction of requests within 300 ms?

With a 300 ms bucket boundary, sum the rate of buckets with le=“0.3” and divide by the summed rate of _count over the same population and window. For a metric in seconds, 0.3 is 300 ms.

What should I display when there is no traffic?

With zero observations, p95 and the threshold compliance ratio are undefined. Display request volume and scrape health separately so you can distinguish zero traffic from missing measurements.