How I stopped screenshot tests from crying wolf

Why pixel diffs failed

The first version did what most screenshot tests do: take a baseline, take a new screenshot, and fail if any pixel is different. On a real mobile product that is a coin flip. Fonts anti-alias slightly differently between runs, devices and GPU drivers round colours differently, and a carousel or a blinking cursor is never in the same frame twice.

Every one of those produced a red build with nothing actually broken. After enough of them, people stop reading the failures, and then the real regressions slip through too. A test that fails for noise is worse than no test.

Compare what the screen looks like, not its pixels

A perceptual hash is a short fingerprint of how an image looks to a person. You shrink the screenshot until fine detail disappears, drop the colour, and keep only the broad structure: where things are light and dark. That structure becomes a 64-bit number. Two screenshots that look the same to a human produce the same, or almost the same, number, even when thousands of individual pixels differ.

Hamming distance as the verdict

To compare two fingerprints you count how many bits differ, which is the Hamming distance. Anti-aliasing noise flips a bit or two. A missing button, a broken layout or the wrong screen flips many. So the test passes when the distance is under a threshold and fails when it is over, and that threshold is the knob that separates noise from real change.

// Simplified sketch of the idea, not the production code.
long hash(BufferedImage img) {
    // 1. Shrink and drop colour: tiny details and anti-aliasing disappear.
    double[][] px = grayscale(resize(img, 32, 32));
    // 2. Keep only the low frequencies: the overall shape of the screen.
    double[][] freq = dct(px);              // top-left 8x8 block
    double median = median(topLeft(freq, 8));
    // 3. One bit per frequency: above or below the median.
    long bits = 0;
    for (int i = 0; i < 64; i++) {
        bits = (bits << 1) | (topLeft(freq, 8)[i / 8][i % 8] > median ? 1 : 0);
    }
    return bits;
}

int distance(long a, long b) {
    return Long.bitCount(a ^ b);            // Hamming distance: bits that differ
}

boolean looksSame(BufferedImage baseline, BufferedImage actual, int threshold) {
    return distance(hash(baseline), hash(actual)) <= threshold;
}

Tuning the threshold

Too low and the noise comes back; too high and real regressions pass. The way to set it is with data: run the suite many times against an unchanged build to see how far noise drifts, then check that known real changes land well above that. When a screen keeps failing for good reasons, that is a product bug to file, not a threshold to raise.

What changed

Flaky failures in the screenshot tests went down. More importantly, a red visual test started to mean something again, so people read the failures, and the regressions that do happen get caught before release instead of after.