Testing ASR pipelines with synthetic digit utterances

Real speech corpora are heavy, rights-encumbered, and awkward to ship in a CI cache. For loader and eval-harness smoke tests you often need something smaller: a handful of short WAVs with known transcripts.

Novus Examples ships synthetic tone-digit utterances under the audio asr subcategory. Each digit is a characteristic dual-partial tone — not a human voice — so the files stay tiny, deterministic, and free of biometric or copyrighted speech. Every utterance has:

• a clean WAV at 16 kHz mono PCM
• a noisy twin (~5–7 dB SNR)
• a JSON transcript with the expected text (zero one two three, etc.)

A minimal eval loop

1. Download a clean/noise/transcript triad (for example digits 0123).
2. Point your ASR runner at the clean WAV; assert the normalised transcript equals the JSON text field.
3. Re-run on the noisy twin and record word-error-rate (WER) or exact-match rate.
4. Keep the seed and sampleRate from the file specs in your CI logs so regressions are reproducible.

Normalise before you score

Most "the model got it wrong" results in a digit harness are really normalisation bugs. Before comparing anything to the JSON text field, pin your normaliser's behaviour on all of these:

• Numerals versus words. 0123 may come back as zero one two three, 0 1 2 3, or 123. Pick one canonical form and convert both sides to it; do not compare raw strings.
• Zero in particular. Recognisers return zero, oh, and o for the same digit depending on the language model. All three are correct transcriptions and all three fail an exact-match test.
• Punctuation and casing. Trailing periods and capitalised first words are inserted by post-processing, not recognition. Strip them before scoring or you are measuring the formatter.
• Whitespace. Collapse runs, trim ends. An extra leading space is a 100% failure on exact-match and a 0% failure on WER, which is a good reminder to record both.

Log the raw hypothesis alongside the normalised one. When a run fails, that pairing tells you within seconds whether the recogniser or your glue is at fault.

Read WER with the deletion caveat

Word error rate is (substitutions + insertions + deletions) / reference_words. On a four-digit utterance the denominator is four, so a single error is 25% — WER on short fixtures is coarse and jumps in big steps. Two consequences worth designing around:

• Set thresholds per-suite, not per-file. Aggregate across every utterance you run so the denominator is large enough to be meaningful.
• WER is not bounded at 100%. A model that hallucinates a long transcript over a short reference can exceed it, and a threshold like "fail above 100%" will never fire. Assert an upper bound on hypothesis length too.

The noisy twins exist to give this a gradient. Expect the clean WAVs at or near 0% and the noisy twins meaningfully worse; if both score identically, your noisy file is probably not reaching the model — check that the loader is not silently falling back to the clean path.

Pairing with the AI JSONL set

The ai category includes an ASR digit utterances JSONL that references the same clean WAVs. Use it to test dataset loaders that expect {id, audio, text, duration_ms} rows before you wire a real corpus.

Check the audio actually arrived

Before blaming a model, prove the bytes reached it in the shape it expects. Most harness bugs in this area are one of four things, and all are quick to rule out:

• Sample rate. The files are 16 kHz mono PCM. A runner that assumes 8 kHz or 44.1 kHz without resampling will play the audio at the wrong speed, and recognisers respond to that with confident nonsense rather than an error.
• Channel count. Mono in, mono expected. A pipeline that duplicates to stereo and then sums back can shift level by 3 dB, which matters for the noisy twins where SNR is the variable under test.
• Bit depth and container. Reading a WAV header wrong — or skipping it and treating the header bytes as samples — produces a click at the start and a small offset throughout.
• Duration. Assert the decoded length matches the documented duration. A file that decodes to zero samples fails silently in a lot of runners, and a 0% WER on an empty transcript compared against an empty hypothesis will even look like a pass.

What this does not measure

These fixtures will not tell you how well a model understands conversational English. They will tell you whether your preprocessing, resampling, file I/O, and scoring glue work — which is usually where CI breaks first.