# Runbook — CheckoutLatencySLOBurn

> Fixture runbook for the alerting rules in this category. Every host, dashboard
> and endpoint named here is invented; nothing in this document resolves.

**Alert:** `CheckoutLatencySLOBurn`
**Severity:** critical · **Team:** shop · **SLO:** `checkout-latency`
**Fires when:** p99 latency for `checkout-api` exceeds 1 s for 15 minutes.

## 1. Confirm the alert is real

1. Open the *Shop — Service Overview* dashboard (`shop-service-overview`) and check
   the **p99 latency** panel over the last 6 hours.
2. Confirm `up{job="shop-services"}` is 1 for every target. A partially scraped
   target produces a p99 computed from a fraction of the traffic.
3. Check the exemplars on the latency histogram. Each one carries a `trace_id`
   that opens the exact slow request.

## 2. Localise the latency

| Symptom | Likely cause | Next step |
|---|---|---|
| `payments-api` spans dominate the trace | Upstream PSP timeout | Check the payment provider status |
| `sql.(*DB).QueryContext` dominates the CPU profile | Query plan regression | Compare against the baseline profile |
| Latency rises with `service:http_requests:rate5m` | Capacity, not a defect | Scale out before investigating code |
| Only one instance is slow | Node problem | Cordon the node and drain |

## 3. Mitigate

- If a deploy correlates with the onset (see the **Deployments** annotation),
  roll back first and diagnose afterwards.
- If the payment provider is the cause, the circuit breaker should already be
  open; confirm `breaker.state` in the logs before manually disabling checkout.

## 4. Close out

- Record the burn against the 28-day budget from the error-budget export.
- If the budget is exhausted, feature work stops until it recovers.
- Add a regression test if the cause was a code change.

## Related fixtures

- `otlp-trace-error-exception-event` — a trace of the failure mode in step 2.
- `cpu-folded-stacks-before` / `cpu-folded-stacks-after` — the profile comparison.
- `slo-error-budget-burndown-csv` — the budget figures used in step 4.