Browse testing · Novus Examples

pytest — JSON Report (pytest-json-report)

The pytest-json-report document, which records setup, call and teardown as three separate phases per test and keeps error distinct from failed in its summary. A test can pass its call phase and still error in teardown, which is why summing outcomes without reading the phases gives the wrong total. Reports the same canonical run as the JUnit dialect files: 12 tests, 9 passed, 1 assertion failure, 1 error and 1 skip.

File: JSON · Runners · UTF-8
Use case: JSON parsingConversion testing+1· Conversion set

Requirements Traceability Matrix with One Gap (CSV)

Thirteen requirements mapped to the test cases that cover them — and one, REQ-PAY-02, deliberately covered by nothing. A coverage-gap report that does not surface that row is not working, which makes this the answer key for exactly that check.

File: CSV · Plans · 13 rows
Use case: CSV parsingData import+1· Conversion set

Robot Framework — Checkout Suite

A complete Robot Framework suite with all four common sections, scalar, list and dictionary variables, a continued line using ..., and user keywords with arguments. Robot separates arguments on two or more spaces, so a parser splitting on a single space merges every keyword call into one token.

File: ROBOT · Runners · UTF-8
Use case: Code parsingEditor testing+1· Conversion set

Robot Framework — Data-Driven Test Template

The template form, in which the *** Test Cases *** header itself names the argument columns and each row becomes one test. Five tests share a single keyword body, and a parser that reads the header row as a test name reports six.

File: ROBOT · Runners · UTF-8
Use case: Code parsingEditor testing+1· Conversion set

Robot Framework — output.xml

Robot Framework's output.xml, in which every keyword call carries its own status element and a trailing statistics section restates the totals by tag and by suite. Robot 7 reports elapsed seconds rather than the endtime attribute older versions used, so schemaversion is what a reader must branch on. Reports the same canonical run as the JUnit dialect files: 12 tests, 9 passed, 1 assertion failure, 1 error and 1 skip.

File: XML · Runners · UTF-8
Use case: Conversion testingLog parsing+1· Conversion set

Robot Framework — Shared Keyword Resource

A resource file: keywords, variables and imports with no *** Test Cases *** section at all, meant to be imported by the suites next to it. Tooling that requires at least one test case rejects a perfectly ordinary resource, and keyword defaults such as ${qty}=1 are where argument parsers usually stop.

File: ROBOT · Runners · UTF-8
Use case: Code parsingEditor testing+1· Conversion set

TAP — Bail Out Mid-Run

A run abandoned after the third assertion with a Bail out! line, leaving five of the planned eight assertions unreported. The run is a failure even though only one assertion said not ok, and a consumer must not report 2/3 passing as a 67% pass rate.

File: TAP · Tap · UTF-8
Use case: Log parsingError handling+1· Conversion set

TAP — Comments and Diagnostic Lines

Free-form # comment lines interleaved with assertions, including a trailing summary block. Comments carry no result and must never shift assertion numbering, which is exactly what a parser counting lines instead of ok/not ok tokens gets wrong.

File: TAP · Tap · UTF-8
Use case: Log parsingError handling+1· Conversion set

TAP — Entire Plan Skipped

A whole suite declined in a single line: a zero-length plan with a SKIP directive and a reason. It is a valid, complete TAP stream containing no assertions, and a consumer must report it as skipped rather than as an empty pass or a parse error.

File: TAP · Tap · UTF-8
Use case: Log parsingError handling+1· Conversion set

TAP — Failures with YAML Diagnostics

Two failing assertions, each followed by an indented YAML diagnostic block carrying the expected and actual values and a file/line location. The block is where every useful failure message in TAP lives, and it is the part naive line-based parsers throw away.

File: TAP · Tap · UTF-8
Use case: Log parsingError handling+1· Conversion set

TAP — Nested Subtests (Indented)

Two subtests, each an indented TAP stream with its own plan, rolled up into two top-level assertions. Flatten it and you count seven assertions instead of two; ignore the indentation and the nested failure is counted twice.

File: TAP · Tap · UTF-8
Use case: Log parsingError handling+1· Conversion set

TAP — Plan Count Does Not Match

Every assertion in this stream says ok, but the plan promised ten and only eight arrived — the signature of a runner that crashed after its last reported assertion. A consumer that only counts not ok lines calls this a clean pass.

File: TAP · Tap · UTF-8
Use case: Log parsingError handling+1· Conversion set

TAP — Plan First, All Passing

The canonical TAP shape: a version line, a leading plan, then six passing assertions in order. Anything a TAP consumer does with a real stream it must do with this one first.

File: TAP · Tap · UTF-8
Use case: Log parsingError handling+1· Conversion set

TAP — SKIP Directives

Two assertions marked # SKIP with a reason, alongside a genuine failure. Skipped assertions are written as ok, so counting ok lines reports 5 passes when only 3 tests actually ran.

File: TAP · Tap · UTF-8
Use case: Log parsingError handling+1· Conversion set

TAP — TODO Directives

Two failing assertions marked # TODO, meaning they are known-unfinished work rather than regressions. A TODO failure must not fail the run, so a consumer that treats every not ok as a build breaker turns this green run red.

File: TAP · Tap · UTF-8
Use case: Log parsingError handling+1· Conversion set

TAP — TODO That Unexpectedly Passed

Assertion 2 is marked TODO but reports ok — an unexpected success, which means the feature landed and the marker is now stale. Harnesses such as prove report this as a bonus rather than a pass, and dropping the distinction lets stale TODOs accumulate forever.

File: TAP · Tap · UTF-8
Use case: Log parsingError handling+1· Conversion set

TAP — Trailing Plan

The same stream with the plan at the end, which is what a harness emits when it does not know the test count up front. A consumer that requires a leading plan rejects a perfectly valid stream.

File: TAP · Tap · UTF-8
Use case: Log parsingError handling+1· Conversion set

TAP — Version 14 Subtest Syntax

TAP 14 with an explicit version line and a pragma inside a subtest. Consumers pinned to version 13 either reject the stream outright or ignore the pragma, so this file separates the two behaviours.

File: TAP · Tap · UTF-8
Use case: Log parsingError handling+1· Conversion set

Test Case Matrix — Twelve Documented Cases (CSV)

Twelve test cases with level, priority, the requirement each covers, preconditions, steps and expected result — the table a test-management import expects. Eleven of the twelve are automated, and the requirement column joins this file to the traceability matrix next to it.

File: CSV · Plans · 12 rows
Use case: CSV parsingData import+1· Conversion set

Test Plan — Release Plan for a SAMPLE Service (Markdown)

A complete release test plan for the fictional Novus Checkout Service — scope, test levels, entry and exit criteria, a risk table and deliverables. Useful as a Markdown-rendering fixture with three tables and nested lists, and as a realistic input for documentation tooling that has to extract structure from prose.

File: MD · Plans
Use case: Editor testingConversion testing+1· Conversion set

TestNG — testng-results.xml

TestNG's own result document, with suite/test/class/test-method nesting, signature attributes and full stack traces in CDATA. TestNG has no error status, so the errored case is reported as FAIL and the header attributes read failed=2. Reports the same canonical run as the JUnit dialect files: 12 tests, 9 passed, 1 assertion failure, 1 error and 1 skip.

File: XML · Runners · UTF-8
Use case: Conversion testingLog parsing+1· Conversion set

xUnit.net — v2 Assembly Results

The xUnit.net v2 shape: an <assemblies> root holding one assembly, whose tests are grouped into collections rather than classes. Result values are Pass, Fail and Skip with capital initials, which trips converters doing a case-sensitive comparison against the lowercase values every other format uses. Reports the same canonical run as the JUnit dialect files: 12 tests, 9 passed, 1 assertion failure, 1 error and 1 skip.

File: XML · Runners · UTF-8
Use case: Conversion testingLog parsing+1· Conversion set

Explore the test library

Filter the library

142 results