Testing Structure-Conditioned Image Generation with Ground-Truth Maps

What a conditioning model actually promises

Structure-conditioned generators — the ControlNet family and its cousins — take a control map alongside a prompt and promise that the output will follow that structure: the edges in a Canny map, the geometry in a depth map, the pose in a skeleton. Testing that promise is hard for the usual reason: you rarely have the source the control map came from, so "did it follow the structure?" collapses into an opinion.

The fix is to start from a known colour image and derive every conditioning map from it. Then the source is your ground truth, and adherence becomes measurable.

Start from one image, derive every map

Each scene in the ControlNet conditioning set ships a colour ground truth plus the maps a guided model consumes, all derived from that exact image:

• a Canny edge map for edge-conditioned generation,
• a depth map for depth control,
• a normal map, a semantic segmentation map, a sparse scribble, and a clean line-art extraction,
• and, for the figure scenes, an OpenPose-style skeleton.

Because they all trace back to the same ground-truth image, you can feed a map to your model and score the result against the source it should reproduce the structure of.

Score adherence, not vibes

Pick a metric that matches the control type:

• Edge / line-art / scribble: re-extract edges from the generated image and compare with the input map (IoU of edge pixels, or Chamfer distance between edge sets). A faithful result reproduces the input edges.
• Depth: run a depth estimator on the output and compare with the input depth map — correlation or scale-invariant error tells you whether the geometry was respected.
• Segmentation: segment the output and compute mask IoU against the input label map.
• Pose: run a pose estimator on the output and measure keypoint distance from the input skeleton.

The point is that every control type has a round-trip check: apply the same extractor to the output and see how close it lands to the map you fed in.

Cover the map types, not just one

A model that nails Canny can ignore depth entirely. Test the matrix — edge, depth, normal, segmentation, scribble, line-art, pose — and record a score per type. A model that scores well across all of them is genuinely structure-aware; one that spikes on edges alone is really just an edge follower.

Sparse controls deserve special attention. A scribble gives the model far more freedom than a dense edge map, so scribble adherence and edge adherence measure different things. Keep them separate.

Separate adherence from quality

A round-trip score answers one question — did the output follow the map? It says nothing about whether the image is any good, and the two trade off directly. Turn conditioning strength up and adherence rises while the image gets stiff and artefact-ridden; turn it down and you get a beautiful picture that ignores the control.

So never report adherence alone. Sweep the conditioning scale across several values and record adherence and a no-reference quality measure at each point. What you want is the shape of that curve, not a single number: a good model holds adherence high while quality stays flat, and a weak one buys every point of adherence with visible degradation. Comparing two models at one arbitrary strength setting mostly tells you which one was tuned for that setting.

Pin the sampler, step count, and seed while you do this. All three move both metrics, and a comparison that varies them alongside the model is not a comparison of models.

Make it reproducible

Every map in the set is generated deterministically from the same source with a documented seed, so the pixels are identical on every machine. That means your adherence scores are comparable across model versions and across CI runs — which is what lets a conditioning benchmark double as a regression gate. When a score drops, the fixed source lets you reproduce the exact failing case locally.

Get the fixtures

Grab a scene and its maps from the ControlNet conditioning set, or explore related tasks: depth estimation, edge detection, normal maps, and pose estimation. Every file is free to use, no attribution required.