How to Test Metadata Preservation and ONNX Inference with Novus Examples

It is easy to collect an image with EXIF tags or a small neural-network model. It is harder to turn either one into a regression test. A test needs a scenario, a known result, and a clear rule for deciding whether a difference is expected or broken.

The new Novus Examples testing-contract pages make that rule visible beside the download. Metadata fixtures identify what must disappear and what must remain equivalent. ONNX groups identify model inputs, tensor shapes and dtypes, the expected output, the opset, and the comparison tolerance. The files are still ordinary local downloads; the surrounding page supplies the contract that makes them reproducible.

That context also makes failures easier to review: teammates can inspect the same ground truth before deciding whether a changed result is acceptable.

Use the metadata-testing collection for tagged controls and the model inference testing collection for the ONNX suites described below.

A metadata scrubber can produce a file that looks right while changing more than intended. Re-encoding may alter pixels, audio samples, embedded document text, colour behaviour, or the container structure another application relies on. A screenshot comparison cannot distinguish a successful privacy operation from an accidental quality loss.

Each preservation family therefore has a full and stripped control linked as a pair. The wave covers six practical cases:

• PNG textual metadata and an embedded ICC profile;
• TIFF EXIF, XMP, and IPTC records;
• MP3 ID3 fields and cover artwork;
• FLAC Vorbis comments and a picture block;
• PDF Info and XMP properties;
• DOCX core and custom properties.

Start by reading the visible Testing contract section on the file page. It states the scenario, the expected result, and whether the artifact is a valid input, a recoverable edge case, an intentionally invalid fixture, or a reference control. The pair link keeps the full and stripped files together, while the specifications publish the fields that matter for that format.

A useful scrubber test has a removal assertion and a preservation assertion.

The removal assertion asks whether the targeted metadata is gone. Read the format with a library that exposes its native metadata model, then compare the remaining keys or blocks with the contract. Do not rely on the operating system's file-properties panel; it may hide fields that are still present or synthesize values from a different source.

The preservation assertion asks whether the file's meaningful content survived. The comparison depends on the medium:

• decode both PNG or TIFF images and compare their pixel arrays;
• decode audio to a common PCM representation and compare samples or a documented hash;
• extract PDF page text and rendered page dimensions;
• extract DOCX paragraph and table content rather than comparing ZIP bytes.

Container bytes are expected to change when metadata is rewritten. A byte-for-byte assertion would reject a correct result. Compare the decoded essence or document content instead, and keep container-level checks for the metadata structures you deliberately changed.

If the stripped file fails the preservation assertion, inspect the layer where the first difference appears. A pixel mismatch may indicate image recompression or colour-profile removal. An audio mismatch may reveal a transcode instead of a tag rewrite. A document-text mismatch may mean the tool rebuilt the package and dropped content it did not understand.

The Novus preview, specification table, group links, and related-file cards provide quick context, but your parser remains the source of the automated assertion. Download once, store the fixture with its stable filename in your test cache, and record the Novus page URL beside the test so a future maintainer can read the published contract.

An ONNX model on its own proves only that a file exists. A reliable inference contract needs:

1. the validated .onnx graph;
2. a JSON document containing named inputs, shapes, and dtypes;
3. a JSON document containing the expected outputs and numeric tolerance.

Novus ships those three artifacts as one linked group for four deliberately small models: static identity, affine transform, two-input broadcasting, and dynamic-batch reduction. Together they exercise the failures most often hidden by a large production model.

Identity isolates loading and tensor transport. Affine transform adds initializers and arithmetic. Broadcasting verifies that the runtime applies shape rules rather than relying on equal-sized arrays. Dynamic-batch reduction checks a symbolic batch dimension and an operation whose output shape differs from its input.

Every model is checked with the ONNX checker and evaluated with the ONNX reference evaluator before it enters the public catalog. The page's structured graph preview summarizes inputs, operators, and outputs without pretending a binary model is ordinary text. See the .ONNX format guide for the container-level explanation.

The exact adapter depends on your runtime, but the comparison pattern is stable:

import json
import numpy as np
import onnx
from onnx.reference import ReferenceEvaluator

model = onnx.load("model.onnx")
onnx.checker.check_model(model)

inputs_doc = json.load(open("input.json", encoding="utf-8"))
expected_doc = json.load(open("expected-output.json", encoding="utf-8"))

feeds = {
    name: np.asarray(spec["values"], dtype=spec["dtype"])
    for name, spec in inputs_doc["inputs"].items()
}
actual = ReferenceEvaluator(model).run(None, feeds)

for value, expected in zip(actual, expected_doc["outputs"]):
    np.testing.assert_allclose(
        value,
        np.asarray(expected["values"], dtype=expected["dtype"]),
        rtol=expected_doc["tolerance"]["rtol"],
        atol=expected_doc["tolerance"]["atol"],
    )

Adapt the JSON access to the manifest on the downloaded page rather than copying this snippet blindly. Assert names, shapes, dtypes, and opset before numeric equality. A runtime that silently casts float32 to float64 may produce similar numbers while violating the interface your application depends on.

When an inference test fails, classify it before changing tolerance.

A load failure points to protobuf parsing, unsupported ops, or an opset mismatch. A missing input or wrong shape is an interface failure. A dtype mismatch is a conversion-policy failure. Correct shapes with slightly different floating-point values are numerical differences and may belong inside the published tolerance. A completely different tensor usually indicates operator semantics, axis selection, or broadcasting.

Increasing tolerance should be the last step, not the first. Run the same grouped input against the reference evaluator and the target runtime, then localize the first operator whose output diverges. Because the Novus graphs are intentionally small, that investigation stays manageable.

Novus Examples is a static catalog, not a remote test API. Download the artifacts you need and run them locally or in CI. Pin the stable file URLs in a fixture lockfile, verify a checksum in your own repository if supply-chain repeatability matters, and fail clearly when an expected file is absent.

For metadata, run both removal and preservation assertions. For ONNX, validate the model, validate the input contract, execute it, and compare every output. Keep the reference and failure messages together so a future change explains what moved: metadata structure, decoded essence, model interface, or numeric result.

That is the difference between a sample and a test fixture. The sample demonstrates a format. The contract makes its result reviewable, automatable, and worth keeping.