Visual Diff Regression With A/B Image Twins
Calibrate pixel and perceptual diffs against documented A/B twins — 1px shifts, chrome adds, and colour tints with known change kinds.
Visual regression suites fail for boring reasons: anti-alias noise, a 1px layout shift, or a button label that changed from OK to GO. To debug the tool—not the product—you need fixtures where the change kind is documented and the baseline twin is bit-stable.
Start from a pair, not a screenshot dump
Novus ships diff-regression twins: each case has an A baseline and a B changed image, plus specs that name the change (pixel, button, colour). Your assertion should be:
- Diff(A, A) ≈ 0 (sanity).
- Diff(A, B) exceeds your threshold for that change kind.
- Masked / ignored regions still pass when chrome outside the ROI moves.
Browse the diff-regression subcategory and pin paths in CI so URLs and /files/... paths stay permanent.
Calibrate thresholds per change class
A single global SSIM cutoff will either flake on anti-alias or miss a one-pixel badge. Keep a tiny matrix:
| Change kind | Example suite member | Expectation |
|---|---|---|
| Pixel shift / crop | 1px-shift or 1px-crop twins | High pixel delta; tiny perceptual delta |
| UI chrome | button-add / corner-badge | Localized ROI fail |
| Colour | colour-tint / hue-shift | Perceptual meters catch more than MAE |
Anti-aliasing is the real enemy
Almost every flaky visual test traces back to text and curved edges rendering a fraction of a pixel differently between runs — a different GPU, a font-hinting change, a browser point release. The pixels genuinely differ, so a naive diff is correct to flag them and useless anyway.
Three defences, in order of how much they buy you:
- Compare with an anti-alias-aware algorithm. Tools in the pixelmatch family can classify a differing pixel as anti-aliasing when its neighbours bracket it in brightness, and exclude it. This alone removes most flake.
- Set a per-pixel colour tolerance and a total-pixel budget. One knob is not enough: a tight per-pixel threshold with a generous count budget catches a solid one-pixel badge while ignoring a thousand faintly-shifted edge pixels. A single global percentage cannot distinguish those.
- Compare a downscaled copy for layout, full-size for detail. Halving both images before diffing averages sub-pixel noise away while preserving anything structural. Run it as a second, looser assertion — not a replacement.
Use the Diff(A, A) ≈ 0 sanity check to calibrate all of this. If your own baseline does not diff cleanly against itself after a re-render, no threshold will save the suite.
When the diff fails, fail usefully
A red build that says "images differ by 0.4%" tells nobody anything. Whatever runner you use, make failures emit three artefacts: the baseline, the candidate, and a highlight image with changed pixels marked in a colour that appears nowhere in the design (magenta is the usual choice). Attach all three to the CI run rather than logging a number.
Store baselines per platform. A baseline captured on a developer's macOS machine will not match a Linux CI container — different font stacks, different rasterisers — and teams that discover this late usually respond by loosening thresholds until the suite stops catching anything. Capture baselines in the same environment that verifies them, and regenerate them through a reviewed commit, never an automatic "update snapshots" step that runs on failure.
Pair with codec ladders when compressors lie
If your pipeline re-encodes before diffing, also pull a codec ladder rung so JPEG q20 noise is not mistaken for a product regression.
Keep it SAMPLE
All scenes are synthetic still-life / geometric fixtures—no real brands or faces. Treat titles and descriptions as the contract, and leave corrupt samples out of “must match” golden sets unless you are testing failure modes on purpose.