The restore drill that failed politely

14 August 2026 · storage

The quarterly drill went green from top to bottom. It also would not have brought anything back, and it took an afternoon to work out why.

The setup

Nightly snapshots on the storage node, a daily push to an external disk that lives in a different room, and a monthly copy that leaves the building in a rucksack. The drill is meant to prove the middle one: pull a random dataset from the external disk onto a scratch pool and diff it against the live copy.

What the drill reported

snapshot listing        ok   (91 snapshots, newest 03:14)
transfer                ok   (412 GiB, 38 min)
checksum verification   ok   (0 mismatches)
mount as scratch pool   ok
sample diff             ok   (0 differing files)

Five green lines, same as the previous seven quarters. The problem is that all five describe the same thing from slightly different angles: they prove the bytes on the external disk match the bytes on the source, at the moment the drill ran.

What was actually missing

The dataset holding the database volume had been renamed in February when I split the media pool. The backup job takes an explicit list of datasets. The renamed one silently stopped matching, and nothing in the pipeline treats "a configured dataset produced no snapshots" as an error. It treats it as zero work, and zero work finishes successfully very quickly.

So the drill verified 412 GiB of holiday photographs with great rigour, and no part of it noticed that the only irreplaceable dataset had not been copied since winter.

The fix

Three changes, in the order they were made:

1. Expected-set check instead of configured-set check

The job now enumerates the datasets that exist on the source, subtracts an explicit ignore list, and fails when the result does not match what it copied. Adding a dataset now fails loudly until it is either backed up or deliberately ignored, which is the behaviour I assumed I had.

2. Freshness assertion per dataset

for ds in $(list_backed_up_datasets); do
  age=$(snapshot_age_hours "$ds")
  [ "$age" -lt 30 ] || fail "$ds stale: ${age}h"
done

Thirty hours rather than twenty-four, because the push sometimes waits behind a scrub and I would rather not train myself to ignore an alert.

3. The drill restores the thing that matters

Random selection sounded fair and turned out to be theatre. The drill now always restores the database volume plus one random dataset, and it starts the service against the restored copy, waits for its health endpoint, and runs one read query. That last step is the only one that would have caught this.

Cost of the change

What I would tell myself in February

Renaming a dataset is a backup change. It belongs in the same commit as the backup configuration, or it does not happen. The pipeline was not lying: it was answering the question I wrote, and I had written a smaller question than I thought.

← All notes