Testing Image Segmentation with Ground-Truth Masks

A mask you can trust

Segmentation output looks convincing until you overlay it on the truth. The only way to know whether a model actually found the object — versus a blob that happens to sit roughly where the object is — is to compare its mask against a known mask for the same image. Without ground truth, every segmentation "looks fine."

Pair the image with its labels

The segmentation fixtures ship a colour scene together with its label map, both generated from the same source. Single-subject scenes give you a clean two-class mask (subject vs background); the fruit still-life gives you a real instance segmentation map with four distinct object classes. Run your model on the ground-truth image and score the predicted mask against the reference.

The metrics that matter

• IoU (Jaccard) — intersection over union between predicted and true masks. The workhorse for "how much overlap."
• Boundary F1 — precision/recall on the mask edge within a small tolerance. A model can win IoU on a chunky blob while getting every boundary wrong; boundary F1 is what catches soft or bleeding edges.
• Instance matching — for the multi-object scene, match predicted instances to ground-truth instances (Hungarian assignment on IoU), then report per-instance IoU and any missed or duplicated objects.

Report region overlap and boundary quality together. They fail independently, and a segmentation that's good at one and bad at the other is common.

Why mean IoU can lie

Two averaging decisions change the headline number more than most model changes do, and teams routinely compare figures computed different ways:

• Per-image mean versus dataset-wide. Averaging IoU per image and then across images weights a tiny object in one photo the same as a huge one in another. Accumulating intersection and union across the whole set first, then dividing, weights by area. Both are defensible; they are not the same number.
• Class imbalance. On a two-class subject/background mask, the background is most of the frame. Include it in a mean and you get a flattering score dominated by pixels no one cares about. Report foreground IoU separately — always.
• The empty-prediction edge case. If a model predicts nothing and the truth is also empty, IoU is 0/0. Whether you score that 1, 0, or skip it changes the aggregate, and libraries differ. Pin the convention.

A practical habit: record per-class and per-image IoU, not just the aggregate. When a score moves, the breakdown tells you whether one class collapsed or everything drifted slightly — and those have completely different causes.

Thresholds and soft masks

Most models emit per-pixel probabilities, not a binary mask, and the threshold you pick to binarise them is a free parameter that changes every metric. Sweep it rather than hard-coding 0.5: plot IoU across thresholds and record the best value along with the threshold that produced it. A model whose peak sits far from 0.5 is poorly calibrated, which matters if anything downstream consumes the raw probabilities.

Test the hard cases on purpose

Thin structures and fine edges are where segmentation quietly breaks. The potted-plant scene has thin leaves precisely so a mask that over-smooths shows up as a boundary-F1 drop. Keep a fixture like it in your suite as the edge-quality canary.

If your segmentation feeds a downstream generator (segmentation-conditioned synthesis), the conditioning set lets you close the loop: segment the output and compare with the input label map for a second, independent adherence check.

Reproducible by construction

The masks are derived deterministically from the same source image, so they're identical everywhere. That makes IoU and boundary F1 comparable across model versions and CI runs, and a regression reproduces on the exact same scene.

Get the fixtures

Start with the semantic-segmentation set, or explore the rest of the conditioning maps. Free to use, no attribution required.