Incident Runbook — Checkout Latency SLO Burn (md)
The runbook the alerting rules link to, written as a real one would be: confirm, localise, mitigate, close out, with a symptom table pointing at the trace and profile fixtures in this category. A realistic document for testing runbook linting, link checking and documentation pipelines.
# Runbook — CheckoutLatencySLOBurn
> Fixture runbook for the alerting rules in this category. Every host, dashboard
> and endpoint named here is invented; nothing in this document resolves.
**Alert:** `CheckoutLatencySLOBurn`
**Severity:** critical · **Team:** shop · **SLO:** `checkout-latency`
**Fires when:** p99 latency for `checkout-api` exceeds 1 s for 15 minutes.
## 1. Confirm the alert is real
1. Open the *Shop — Service Overview* dashboard (`shop-service-overview`) and check
the **p99 latency** panel over the last 6 hours.
2. Confirm `up{job="shop-services"}` is 1 for every target. A partially scraped
target produces a p99 computed from a fraction of the traffic.
3. Check the exemplars on the latency histogram. Each one carries a `trace_id`
that opens the exact slow request.
## 2. Localise the latency
| Symptom | Likely cause | Next step |
|---|---|---|
| `payments-api` spans dominate the trace | Upstream PSP timeout | Check the payment provider status |
| `sql.(*DB).QueryContext` dominates the CPU profile | Query plan regression | Compare against the baseline profile |
| Latency rises with `service:http_requests:rate5m` | Capacity, not a defect | Scale out before investigating code |
| Only one instance is slow | Node problem | Cordon the node and drain |
## 3. Mitigate
- If a deploy correlates with the onset (see the **Deployments** annotation),
roll back first and diagnose afterwards.
- If the payment provider is the cause, the circuit breaker should already be
open; confirm `breaker.state` in the logs before manually disabling checkout.
## 4. Close out
- Record the burn against the 28-day budget from the error-budget export.
- If the budget is exhausted, feature work stops until it recovers.
- Add a regression test if the cause was a code change.
## Related fixtures
- `otlp-trace-error-exception-event` — a trace of the failure mode in step 2.
- `cpu-folded-stacks-before` / `cpu-folded-stacks-after` — the profile comparison.
- `slo-error-budget-burndown-csv` — the budget figures used in step 4.
Specifications
- Headings
- 6
- Table Rows
- 4
- Ordered Steps
- 3
- Linked Fixtures
- 3
- Referenced Alert
- CheckoutLatencySLOBurn
- Everything Fictional
- true
Testing contract
Reference control- Scenario
- Run a documentation link checker and heading-structure linter over the file.
- Expected result
- Six headings nest correctly under one H1, the four-row symptom table parses as GFM, and every referenced fixture slug exists in this category.
What is a .md file?
Markdown (MD) is a lightweight plain-text markup language that uses simple punctuation conventions to denote headings, lists, links, emphasis, and code. It is designed to be readable as-is and to convert cleanly to HTML. It is widely used for documentation, READMEs, and content authoring.
How to use this file
Use an example Markdown file to test parsers and renderers, verify GitHub-Flavored Markdown extensions like tables and fenced code, and exercise HTML-conversion pipelines.
How to use this file for testing
“Incident Runbook — Checkout Latency SLO Burn (md)” is a deterministic Novus Examples fixture for Observability, Config testing, Templates. Structured and plain-text telemetry with known timestamps, levels, request identifiers, and error states for testing log ingestion, correlation, dashboards, and alert pipelines.
Documented properties for this file: MD · 2,077 bytes. Compare results against paired or grouped companions on this page when present (clean↔damaged, searchable↔scanned, or format twins) so scores stay reproducible across runs.
Download the file once, keep the path stable in CI or local scripts, and treat the spec table as the contract: dimensions, seeds, field lists, and roles are intentional. Corrupt or invalid samples are labelled as such — expect parsers to fail loudly rather than silently accept them.
Telemetry fixtures use fixed trace IDs, span IDs, and timestamps so ingestion is reproducible run to run. Point your collector, parser, or query layer at the file and assert the documented span tree, metric families, or severity mix; service and host names are invented.
Code examples
import markdown # pip install markdown
html = markdown.markdown(open("incident-runbook-checkout-latency.md").read())
print(html[:200])Related files
- yamlAlertmanager Config — Routing Tree, Inhibition and Mute Windows (yaml)An Alertmanager configuration with the routing behaviour that is hard to reason about without a fixture: a continue: true branch so a critical alert reaches two receivers, an inhibition rule suppressing warnings under a matching critical, and a weekend mute window.
- jsonDashboard JSON — Chained Template Variables (json)A dashboard driven by four template variables, one of which queries using the value of another — the chained dependency that must be resolved in order. Multi-select variables interpolate as regex alternations, which is why every query here uses =~ rather than =.
- jsonDashboard JSON — Five-Panel Service Overview (json)A dashboard document in the Grafana JSON model: five panels on the 24-column grid, each with a datasource reference by UID, a PromQL target and a unit. Three panels query the recording rules in this category, so the dashboard and the rule file can be validated against each other.
- confFluent Bit — Tail, Enrich and Forward Pipeline (conf)A Fluent Bit configuration in its INI-like section format: a tail input using the CRI multiline parser that reassembles the partial-line fixture, Kubernetes enrichment, a modify filter dropping two of the high-cardinality fields, and a throttle. Keys are space-aligned, not delimited.
- yamlOpenTelemetry Collector — Tail-Sampling Policies (yaml)Six tail-sampling policies including an inverted regex match that drops health checks and a composite policy with its own rate limit and ordered sub-policies. Policies are OR-ed and one match keeps the entire trace, which is the semantic most people get backwards.
- yamlOpenTelemetry Collector — Three-Signal Pipeline Configuration (yaml)A Collector configuration wiring traces, metrics and logs through separate pipelines, with named component instances, a filelog receiver that parses the CRI log fixture, OTTL statements performing the semantic-convention migration from the paired trace fixtures, and a redaction processor.
Generated by generation/observability.py. Free for any use, no attribution required — license.