# Incident Postmortem: Checkout Latency

*A blameless operational review with impact, timeline, causal analysis, and owned actions.*

**Incident ID:** INC-2026-0718  
**Severity:** SEV-2  
**Owner:** Priya Nair  
**Window:** 2026-07-18 14:07-15:02 UTC  
**Affected service:** Checkout API - eu-west  
**Status:** Resolved; follow-up actions in progress  

## Executive summary

A connection-pool limit introduced during a routine database proxy rollout caused elevated checkout latency in eu-west. Requests queued behind exhausted connections, and the retry policy amplified load. The team rolled back the proxy configuration and restored normal latency within 55 minutes.

This review is blameless: the failure emerged from interacting safeguards, incomplete load-test coverage, and an alert that measured errors but not queue time.

## Customer impact

- 8.7% of checkout requests exceeded five seconds.
- 2.1% of checkout attempts returned a timeout before payment authorisation.
- No duplicate charges, data loss, or security exposure occurred.
- Support received 43 related contacts; all affected carts remained recoverable.

## Timeline

| UTC | Event | Owner |
| --- | --- | --- |
| 14:07 | Database proxy configuration deployed to eu-west. | Release bot |
| 14:14 | P95 checkout latency crossed the service objective. | Monitoring |
| 14:19 | Support linked three timeout reports and paged on-call. | N. Brooks |
| 14:26 | Incident declared SEV-2; change freeze started. | P. Nair |
| 14:38 | Connection saturation identified in proxy metrics. | D. Okafor |
| 14:46 | Rollback approved and initiated. | P. Nair |
| 14:54 | Queue depth and P95 latency returned to baseline. | Monitoring |
| 15:02 | Recovery verified; incident moved to monitoring. | S. Rossi |

## Root cause and contributing factors

**Root cause:** The proxy pool was capped below the application's steady-state concurrent connection demand.

**Amplifier:** Retries used fixed intervals, aligning repeated requests and increasing queue pressure.

**Detection gap:** The page fired on error rate, while the first visible symptom was rising queue time and latency.

**Test gap:** Staging load tests used average traffic and did not reproduce the regional peak-concurrency profile.

## Corrective actions

| Action | Owner | Due | Priority | Status |
| --- | --- | --- | --- | --- |
| Set pool limits from measured regional concurrency with 30% headroom. | D. Okafor | 2026-07-22 | P0 | Done |
| Add queue-depth and checkout-latency burn-rate alerts. | S. Rossi | 2026-07-24 | P0 | In progress |
| Replace fixed retries with jittered exponential backoff. | T. Baxter | 2026-07-29 | P1 | In progress |
| Add peak-concurrency profile to pre-production load tests. | A. Bello | 2026-08-02 | P1 | Planned |
| Document rollback ownership in the service runbook. | P. Nair | 2026-07-25 | P2 | Planned |

## What went well

- Support reports were quickly correlated with telemetry.
- The change was reversible and rollback completed in eight minutes.
- Payment idempotency controls prevented duplicate authorisations.

---

*Fictional operational sample. Adapt severity definitions, owners, and retention rules to your organisation.*