Proving the backups actually restore

What the pipeline does

Once a week, a scheduled job runs the whole recovery path end to end against both of the lab's backup stores. It verifies the backup chunks are intact, restores a machine from them, boots it, runs checks that the restored machine is really working, then tears the test copy down again. It does this for the secrets server and for the other machines that matter, so the claim is not "a backup file exists" but "this backup boots and works, as of this week."

A backup you have never restored is a hope, not a backup. The only way to know a restore works is to do the restore. Doing it by hand once a quarter is how you find out it is broken during the outage. Doing it automatically every week is how you find out on a Tuesday, with nothing at stake.

The 97 second restore

For the secrets server, the measured destroy and restore is 97 seconds: delete the running machine, restore it from the most recent backup, boot it, and confirm the service is answering. That number is real and it is the point of the whole exercise. If the worst day starts with the secrets server gone, recovery is a minute and a half of automation, not an afternoon of remembering how the thing was built.

The chronicle: twelve reruns to an honest check

The first version of this pipeline went green. That was the problem. A green check that is not actually checking anything is worse than a red one, because it tells you to relax. Getting from "green" to "green and true" took twelve reruns, each one peeling back a layer that had been quietly lying. Here is what each layer taught.

Permissions are an intersection, not a sum

The restore runs as a service identity that carries a scoped token, not as a full administrator. The first permission tests only ever tried actions that should be denied, saw them denied, and reported success. But a denial proves nothing about what is allowed. The real rule on the virtualization platform is that the effective permission is the intersection of what the user may do and what the token may do. A token can hold a right the user lacks, or the reverse, and the action only succeeds where both agree. The fix was to test the allows, not just the denies. If your test never tries the thing that must work, it is not testing access.

The gap you only find by running the real path

The restored machine has a guest agent the automation talks to. In a clean lab it just works, so nobody notices it needs a specific authorization to be reachable. The pipeline found the gap because it ran the real path against a freshly restored machine instead of a machine that had been up for days. The lesson repeats across this whole project: a lot of failures are invisible until you exercise the exact sequence the outage will exercise, from the exact starting state the outage starts from.

The assertion that drifted from the ruling

One check had drifted from what it was supposed to verify. It was asserting that a Linux service was running, on a machine that is a Windows domain controller, which has no such service and never will. Worse, the wrong assertion was hidden behind a 300 second retry loop: it would wait five full minutes for a thing that could never appear, then fail with a timeout that looked like a slow machine rather than a wrong test. Two bugs stacked. The check tested the wrong fact, and the retry masked that it was the wrong fact. The fix was to make each machine assert against its own type, and to fail fast on a "this can never be true" result instead of retrying it.

Teardown that confirms, not teardown that hopes

The last step deletes the test copy. The first version trusted the delete command's exit code and moved on. But an exit code says the command was accepted, not that the machine is gone. The teardown now confirms the machine no longer exists before it reports success. Same principle as the restore itself: check the state you care about, not the return value of the thing that was supposed to change it.

The audit that catches a silent check

The thread running through every one of those bugs is the same: a check that passes without actually checking. So I wrote a small audit that reads the pipeline's own assertions and flags the ones that can pass silently, the way the drifted assertion did. It is a test for the tests. The failure mode it guards against is the most dangerous one in all of this, because it does not announce itself. It just tells you everything is fine.

What green actually means now

After the twelve reruns, the run went green and 197 of 200 assertions flipped from failing to passing in one pass. The three that stayed were real, tracked, and understood, not swept under a retry. The value here was never the green result. It was the twelve rounds of finding out what "green" had been hiding, and building the checks and the audit so the next green means what it says.