How to Test a Photo Restoration Tool

Restoration needs a ground truth

Photo restoration demos look convincing until you try to measure them. Without a clean reference, you cannot tell whether a model invented plausible texture or recovered the original. The same problem shows up for generative fill and content-aware fill: a filled hole can look fine and still be wrong.

Novus ships photo restoration fixtures as triplets — a clean reference, a damaged input, and a damage mask — so you can treat restoration as a controlled experiment.

Start with a crease or scratch case

A practical first pass:

1. Download the crease clean reference, the damaged input, and the damage mask.
2. Run your restorer on the damaged image (optionally conditioning on the mask).
3. Score the output against the clean reference with PSNR and SSIM.

Repeat for scratches, dust, water stains, torn corners, and cuts in the same suite. Documented seeds keep the damage pattern identical across machines and CI runs.

Score the hole, not the photograph

Whole-image PSNR is close to useless on restoration fixtures, and the reason is arithmetic: if a crease covers 3% of the frame, then 97% of the pixels are already identical to the reference before your tool does anything. A do-nothing pass-through will post an impressive score, and a genuinely better repair will move the number by a few tenths of a decibel.

Use the mask to split the measurement:

• Masked region only — the actual restoration quality. This is the number that should gate your build. Expect it to look poor next to whole-image figures; that is the point.
• Unmasked region — should be near-identical to the input, well above 45 dB. Restoration models that quietly re-tone or sharpen the entire image fail here, and nothing else will tell you.
• A narrow band straddling the mask boundary — where seams live. A repair can be excellent in the middle and leave a visible ring at the edge, which is exactly what a human notices first and what neither of the other two regions catches.

Plausible is not correct

The trap this fixture set exists to expose: generative restorers produce texture that looks right and is not the original. On a real damaged photo you cannot tell. Here you can, so measure it:

• Track LPIPS or a similar perceptual metric alongside PSNR. When perceptual scores improve while PSNR falls, the model is inventing detail rather than recovering it. That may be exactly what you want for a consumer photo app and exactly what you cannot ship in an archival or forensic pipeline — but it should be a decision, not a surprise.
• Watch for hallucinated structure in the larger damage cases. A torn corner gives a model enough room to invent a plausible continuation of an edge that runs the wrong way. Irregular tears are better at surfacing this than geometric holes, which is why both shapes are in the suite.
• Run the mask-free case too. Many tools accept an optional mask; feeding the damaged image without one tells you how good their own damage detection is, which is what an end user actually experiences.

Pair restoration with inpainting cuts

When you are testing object removal or generative fill specifically, use the inpainting fixtures. Each cut variant (centre rect, circle, corner, strip, irregular tear) has a matching mask and shares a clean reference. Feed the cut image and mask to your tool, then diff against the clean original.

Damage masks from the restoration set are also tagged for inpainting when you want real crease/tear shapes instead of geometric holes.

Related image suites

• Deblur testing — gaussian, motion, and shake blur with clean references.
• Super-resolution — downscaled (and JPEG-degraded) inputs against large clean sources.
• Background removal — including fur, mesh, and glass hard cases for removers and recomposite into visual templates.

Get the fixtures

Browse restoration testing and the rest of the Images library. Every file is free to use, no attribution required.