jsonl44.6 KB
Flaky Detection — 40-Build Test-Run History (JSONL)
Forty builds of six tests, one JSON object per line, with three tests that fail intermittently and three that never do. This is the input shape a flake detector ingests; the CSV twin in this group holds the aggregate it should produce.
Preview — first 50 linesjsonl
{"build": 5001, "branch": "main", "test": "cart.PricingTest.appliesUnitPrice", "outcome": "passed", "durationMs": 1384, "runner": "ci-runner-05", "startedAt": "2026-07-01T01:14:22Z"}
{"build": 5001, "branch": "main", "test": "cart.PricingTest.appliesTaxToSubtotal", "outcome": "passed", "durationMs": 1151, "runner": "ci-runner-01", "startedAt": "2026-07-01T01:14:22Z"}
{"build": 5001, "branch": "main", "test": "cart.DiscountTest.stacksDiscountsInDeclaredOrder", "outcome": "passed", "durationMs": 1504, "runner": "ci-runner-04", "startedAt": "2026-07-01T01:14:22Z"}
{"build": 5001, "branch": "main", "test": "checkout.SessionTest.expiresSessionAfterTimeout", "outcome": "passed", "durationMs": 801, "runner": "ci-runner-02", "startedAt": "2026-07-01T01:14:22Z"}
{"build": 5001, "branch": "main", "test": "checkout.PaymentTest.capturesAuthorisedPayment", "outcome": "passed", "durationMs": 743, "runner": "ci-runner-06", "startedAt": "2026-07-01T01:14:22Z"}
{"build": 5001, "branch": "main", "test": "checkout.ReceiptTest.rendersTotals", "outcome": "passed", "durationMs": 1711, "runner": "ci-runner-04", "startedAt": "2026-07-01T01:14:22Z"}
{"build": 5002, "branch": "main", "test": "cart.PricingTest.appliesUnitPrice", "outcome": "passed", "durationMs": 703, "runner": "ci-runner-06", "startedAt": "2026-07-02T02:14:22Z"}
{"build": 5002, "branch": "main", "test": "cart.PricingTest.appliesTaxToSubtotal", "outcome": "passed", "durationMs": 1089, "runner": "ci-runner-02", "startedAt": "2026-07-02T02:14:22Z"}
{"build": 5002, "branch": "main", "test": "cart.DiscountTest.stacksDiscountsInDeclaredOrder", "outcome": "passed", "durationMs": 702, "runner": "ci-runner-02", "startedAt": "2026-07-02T02:14:22Z"}
{"build": 5002, "branch": "main", "test": "checkout.SessionTest.expiresSessionAfterTimeout", "outcome": "failed", "durationMs": 892, "runner": "ci-runner-03", "startedAt": "2026-07-02T02:14:22Z"}
{"build": 5002, "branch": "main", "test": "checkout.PaymentTest.capturesAuthorisedPayment", "outcome": "passed", "durationMs": 165, "runner": "ci-runner-05", "startedAt": "2026-07-02T02:14:22Z"}
{"build": 5002, "branch": "main", "test": "checkout.ReceiptTest.rendersTotals", "outcome": "passed", "durationMs": 1198, "runner": "ci-runner-06", "startedAt": "2026-07-02T02:14:22Z"}
{"build": 5003, "branch": "main", "test": "cart.PricingTest.appliesUnitPrice", "outcome": "passed", "durationMs": 909, "runner": "ci-runner-06", "startedAt": "2026-07-03T03:14:22Z"}
{"build": 5003, "branch": "main", "test": "cart.PricingTest.appliesTaxToSubtotal", "outcome": "passed", "durationMs": 1003, "runner": "ci-runner-04", "startedAt": "2026-07-03T03:14:22Z"}
{"build": 5003, "branch": "main", "test": "cart.DiscountTest.stacksDiscountsInDeclaredOrder", "outcome": "passed", "durationMs": 475, "runner": "ci-runner-03", "startedAt": "2026-07-03T03:14:22Z"}
{"build": 5003, "branch": "main", "test": "checkout.SessionTest.expiresSessionAfterTimeout", "outcome": "passed", "durationMs": 1329, "runner": "ci-runner-02", "startedAt": "2026-07-03T03:14:22Z"}
{"build": 5003, "branch": "main", "test": "checkout.PaymentTest.capturesAuthorisedPayment", "outcome": "passed", "durationMs": 52, "runner": "ci-runner-03", "startedAt": "2026-07-03T03:14:22Z"}
{"build": 5003, "branch": "main", "test": "checkout.ReceiptTest.rendersTotals", "outcome": "passed", "durationMs": 1656, "runner": "ci-runner-01", "startedAt": "2026-07-03T03:14:22Z"}
{"build": 5004, "branch": "main", "test": "cart.PricingTest.appliesUnitPrice", "outcome": "passed", "durationMs": 283, "runner": "ci-runner-05", "startedAt": "2026-07-04T04:14:22Z"}
{"build": 5004, "branch": "main", "test": "cart.PricingTest.appliesTaxToSubtotal", "outcome": "passed", "durationMs": 315, "runner": "ci-runner-04", "startedAt": "2026-07-04T04:14:22Z"}
{"build": 5004, "branch": "main", "test": "cart.DiscountTest.stacksDiscountsInDeclaredOrder", "outcome": "passed", "durationMs": 38, "runner": "ci-runner-01", "startedAt": "2026-07-04T04:14:22Z"}
{"build": 5004, "branch": "main", "test": "checkout.SessionTest.expiresSessionAfterTimeout", "outcome": "passed", "durationMs": 1314, "runner": "ci-runner-01", "startedAt": "2026-07-04T04:14:22Z"}
{"build": 5004, "branch": "main", "test": "checkout.PaymentTest.capturesAuthorisedPayment", "outcome": "passed", "durationMs": 849, "runner": "ci-runner-02", "startedAt": "2026-07-04T04:14:22Z"}
{"build": 5004, "branch": "main", "test": "checkout.ReceiptTest.rendersTotals", "outcome": "passed", "durationMs": 994, "runner": "ci-runner-01", "startedAt": "2026-07-04T04:14:22Z"}
{"build": 5005, "branch": "main", "test": "cart.PricingTest.appliesUnitPrice", "outcome": "passed", "durationMs": 717, "runner": "ci-runner-04", "startedAt": "2026-07-05T05:14:22Z"}
{"build": 5005, "branch": "main", "test": "cart.PricingTest.appliesTaxToSubtotal", "outcome": "passed", "durationMs": 1626, "runner": "ci-runner-04", "startedAt": "2026-07-05T05:14:22Z"}
{"build": 5005, "branch": "main", "test": "cart.DiscountTest.stacksDiscountsInDeclaredOrder", "outcome": "passed", "durationMs": 828, "runner": "ci-runner-02", "startedAt": "2026-07-05T05:14:22Z"}
{"build": 5005, "branch": "main", "test": "checkout.SessionTest.expiresSessionAfterTimeout", "outcome": "failed", "durationMs": 484, "runner": "ci-runner-05", "startedAt": "2026-07-05T05:14:22Z"}
{"build": 5005, "branch": "main", "test": "checkout.PaymentTest.capturesAuthorisedPayment", "outcome": "passed", "durationMs": 271, "runner": "ci-runner-05", "startedAt": "2026-07-05T05:14:22Z"}
{"build": 5005, "branch": "main", "test": "checkout.ReceiptTest.rendersTotals", "outcome": "passed", "durationMs": 892, "runner": "ci-runner-01", "startedAt": "2026-07-05T05:14:22Z"}
{"build": 5006, "branch": "main", "test": "cart.PricingTest.appliesUnitPrice", "outcome": "passed", "durationMs": 990, "runner": "ci-runner-06", "startedAt": "2026-07-06T06:14:22Z"}
{"build": 5006, "branch": "main", "test": "cart.PricingTest.appliesTaxToSubtotal", "outcome": "failed", "durationMs": 438, "runner": "ci-runner-05", "startedAt": "2026-07-06T06:14:22Z"}
{"build": 5006, "branch": "main", "test": "cart.DiscountTest.stacksDiscountsInDeclaredOrder", "outcome": "failed", "durationMs": 877, "runner": "ci-runner-04", "startedAt": "2026-07-06T06:14:22Z"}
{"build": 5006, "branch": "main", "test": "checkout.SessionTest.expiresSessionAfterTimeout", "outcome": "failed", "durationMs": 1209, "runner": "ci-runner-03", "startedAt": "2026-07-06T06:14:22Z"}
{"build": 5006, "branch": "main", "test": "checkout.PaymentTest.capturesAuthorisedPayment", "outcome": "passed", "durationMs": 722, "runner": "ci-runner-02", "startedAt": "2026-07-06T06:14:22Z"}
{"build": 5006, "branch": "main", "test": "checkout.ReceiptTest.rendersTotals", "outcome": "passed", "durationMs": 1286, "runner": "ci-runner-02", "startedAt": "2026-07-06T06:14:22Z"}
{"build": 5007, "branch": "main", "test": "cart.PricingTest.appliesUnitPrice", "outcome": "passed", "durationMs": 746, "runner": "ci-runner-02", "startedAt": "2026-07-07T07:14:22Z"}
{"build": 5007, "branch": "main", "test": "cart.PricingTest.appliesTaxToSubtotal", "outcome": "passed", "durationMs": 410, "runner": "ci-runner-05", "startedAt": "2026-07-07T07:14:22Z"}
{"build": 5007, "branch": "main", "test": "cart.DiscountTest.stacksDiscountsInDeclaredOrder", "outcome": "passed", "durationMs": 174, "runner": "ci-runner-01", "startedAt": "2026-07-07T07:14:22Z"}
{"build": 5007, "branch": "main", "test": "checkout.SessionTest.expiresSessionAfterTimeout", "outcome": "passed", "durationMs": 1491, "runner": "ci-runner-05", "startedAt": "2026-07-07T07:14:22Z"}
{"build": 5007, "branch": "main", "test": "checkout.PaymentTest.capturesAuthorisedPayment", "outcome": "passed", "durationMs": 782, "runner": "ci-runner-01", "startedAt": "2026-07-07T07:14:22Z"}
{"build": 5007, "branch": "main", "test": "checkout.ReceiptTest.rendersTotals", "outcome": "passed", "durationMs": 1066, "runner": "ci-runner-03", "startedAt": "2026-07-07T07:14:22Z"}
{"build": 5008, "branch": "main", "test": "cart.PricingTest.appliesUnitPrice", "outcome": "passed", "durationMs": 1692, "runner": "ci-runner-06", "startedAt": "2026-07-08T00:14:22Z"}
{"build": 5008, "branch": "main", "test": "cart.PricingTest.appliesTaxToSubtotal", "outcome": "passed", "durationMs": 1418, "runner": "ci-runner-01", "startedAt": "2026-07-08T00:14:22Z"}
{"build": 5008, "branch": "main", "test": "cart.DiscountTest.stacksDiscountsInDeclaredOrder", "outcome": "failed", "durationMs": 779, "runner": "ci-runner-06", "startedAt": "2026-07-08T00:14:22Z"}
{"build": 5008, "branch": "main", "test": "checkout.SessionTest.expiresSessionAfterTimeout", "outcome": "passed", "durationMs": 720, "runner": "ci-runner-04", "startedAt": "2026-07-08T00:14:22Z"}
{"build": 5008, "branch": "main", "test": "checkout.PaymentTest.capturesAuthorisedPayment", "outcome": "passed", "durationMs": 69, "runner": "ci-runner-03", "startedAt": "2026-07-08T00:14:22Z"}
{"build": 5008, "branch": "main", "test": "checkout.ReceiptTest.rendersTotals", "outcome": "passed", "durationMs": 1282, "runner": "ci-runner-05", "startedAt": "2026-07-08T00:14:22Z"}
{"build": 5009, "branch": "main", "test": "cart.PricingTest.appliesUnitPrice", "outcome": "passed", "durationMs": 1563, "runner": "ci-runner-06", "startedAt": "2026-07-09T01:14:22Z"}
{"build": 5009, "branch": "main", "test": "cart.PricingTest.appliesTaxToSubtotal", "outcome": "passed", "durationMs": 1387, "runner": "ci-runner-06", "startedAt": "2026-07-09T01:14:22Z"}… 241 lines total — download for the full file.
Specifications
- Seed
- 61200
- Records
- 240
- Builds
- 40
- Tests
- 6
- Fields
- build, branch, test, outcome, durationMs, runner, startedAt
- One Json Object Per Line
- true
- Line Endings
- LF
Testing contract
Reference control- Scenario
- Aggregate a per-build result history into per-test flake rates.
- Expected result
- 240 records are read; three tests show a non-zero failure rate and three show none, matching the summary CSV in this group row for row.
What is a .jsonl file?
JSONL (JSON Lines) is a text format where each line is a complete, independent JSON value, allowing records to be streamed and appended without parsing the whole file. It is not itself a JSON array and each line must stand alone. It is common in logging, machine learning datasets, and data pipelines.
How to use this file
Use an example JSONL to test line-by-line streaming parsers, append-and-resume ingestion, and batch pipelines that process one record per line.
How to use this file for testing
“Flaky Detection — 40-Build Test-Run History (JSONL)” is a deterministic Novus Examples fixture for JSON parsing, Time-series data, Error handling. Flat, deeply nested, JSON Lines, and intentionally invalid JSON for testing parsers and error handling.
Documented properties for this file: seed 61200 · 240 records · LF. Compare results against paired or grouped companions on this page when present (clean↔damaged, searchable↔scanned, or format twins) so scores stay reproducible across runs.
Download the file once, keep the path stable in CI or local scripts, and treat the spec table as the contract: dimensions, seeds, field lists, and roles are intentional. Corrupt or invalid samples are labelled as such — expect parsers to fail loudly rather than silently accept them.
Test and coverage reports document their totals (suites, cases, passes, failures, skips, covered lines) in the spec table. Point your CI dashboard, coverage gate, or report converter at the file and assert those counts survive; format twins carry identical numbers so a conversion can be scored exactly.
Feed the file to your parser and assert it handles the documented quirks — quoted delimiters, embedded newlines, ragged rows, or invalid syntax; the valid↔invalid distinction is labelled in the title.
Code examples
import json
with open("test-run-history.jsonl") as f:
rows = [json.loads(line) for line in f]
print(len(rows), rows[0])Related files
- jsonCucumber JSON — Failing Step with Stack TraceOne step of one scenario fails with an assertion message and a stack frame; every other step passes. In cucumber.json a scenario has no status of its own — it is failed if any of its steps is — so this file catches reporters that look for a scenario-level field that does not exist.
- jsonCucumber JSON — Undefined, Pending and Skipped StepsThe three non-failure statuses BDD runners produce and dashboards routinely collapse into "failed": undefined (no step definition matched), pending (explicitly unimplemented) and skipped (never attempted). Whether the run is red depends on the runner's strict setting, which is exactly the decision this file forces.
- ndjsonCucumber Messages — Failing Run (NDJSON)The messages twin of the failing run: one testStepFinished carries status FAILED with both a human message and a structured exception object, and testRunFinished reports success false. Comparing it with the passing stream in the other group isolates exactly which envelopes a failure changes.
- jsonJest — JSON ResultsJest's --json output, where skipped tests are called pending and the totals appear as flat numTotal/numPassed/numFailed fields alongside per-file assertionResults. Timestamps are epoch milliseconds rather than ISO strings, which is the field most converters mis-handle. Reports the same canonical run as the JUnit dialect files: 12 tests, 9 passed, 1 assertion failure, 1 error and 1 skip.
- jsonMocha — JSON Reporter OutputMocha's JSON reporter, which lists every test in a tests array and then repeats each one in passes, failures or pending. Summing all four arrays counts the run twice — this file exists so that bug shows up on a 12-test suite instead of in production. Reports the same canonical run as the JUnit dialect files: 12 tests, 9 passed, 1 assertion failure, 1 error and 1 skip.
- jsonPlaywright — JSON ReportPlaywright's native JSON report, four levels deep: suites contain specs, specs contain tests (one per project) and tests contain results (one per attempt). Its stats use expected/unexpected rather than passed/failed, because a test annotated as expected-to-fail counts as expected when it fails. Reports the same canonical run as the JUnit dialect files: 12 tests, 9 passed, 1 assertion failure, 1 error and 1 skip.
Generated by generation/testing.py. Free for any use, no attribution required — license.