Incident & Postmortem practice - Skills

What it is & why I use it

The most valuable thing a homelab can teach is not how to build systems - it is how to operate them when they break. So I treat real failures in AutomationLab as incidents: I mitigate, then I write an honest, blameless postmortem that ends in a concrete change to the system, not a note to "be more careful."

Blameless means the writeup asks why the system allowed the failure, not who typed the wrong thing. Every good postmortem produces at least one change that would have caught or contained the problem automatically.

How I use it in the lab

Detect. Monitoring and job status surface a failure - ideally before it has downstream effects. A silent failure that I find by accident is itself a finding.
Mitigate first. Restore the service or stop the bleeding before diagnosing root cause. The clock on impact matters more than being first to the "why."
Timeline. Reconstruct what happened and when, from logs and job history, so the story is grounded in evidence rather than memory.
Root cause, then fix. Trace to the underlying condition and ship a change - a guardrail, an alert, a code fix - that closes the gap.
Prevent recurrence. The fix lands as a PR like any other change, with the postmortem linked, so the reasoning is preserved next to the code.

Architecture

Diagram coming The incident loop: detect -> mitigate -> timeline -> root cause -> prevent, and where each step draws on monitoring and job history.

In action

A representative incident from 11 July 2026, told at a scrubbed level of detail.

Incident - backup job host-key mismatch A scheduled backup job began failing. The job connects to a target host over SSH, and after that host had been rebuilt, its SSH host key changed. The job was correctly refusing to connect to a host whose fingerprint no longer matched the one it had trusted - so it failed closed rather than trusting an unverified host, which is the safe behavior. The visible symptom was simply that backups had stopped succeeding.

Mitigation: re-established the trusted host key through the managed configuration path (not by hand-editing on the box), and confirmed the next run completed.
Root cause: host rebuilds rotate the SSH host key, but the trusted-key record the backup job relied on was not being refreshed as part of that rebuild - so a legitimate rebuild looked identical to an untrusted host.
Prevention: fold host-key refresh into the rebuild automation so a rebuilt host is re-trusted deterministically, and alert on backup success, not just hard failure, so a silent stall surfaces on its own.

Details are deliberately generalized: no hostnames, addresses, or key fingerprints appear here or in any screenshot, per the publishing checklist.

Lessons learned & gotchas

Gotcha "Fail closed" is correct behavior that looks like a bug. A security control doing its job - refusing an unverified host - reads in the moment like a broken backup. The fix is never to weaken the control; it is to make the legitimate change flow through automation so the control stays satisfied.

Alert on success, not only failure. A job that stops running entirely may never emit a failure. Absence of success is the signal you actually want.
Every rebuild has a trust tail. Rotating a host means rotating everything that trusted its old identity. If that is not automated, it becomes a future incident.

Impact

Same day

detect to fix

1 PR

prevention shipped

0

controls weakened

The practice is the point: each incident leaves the platform measurably harder to break the same way twice, and the reasoning is captured where the next operator - or the next me - will find it.

Code & further reading

Postmortem template (link coming) Blameless postmortems

Published postmortems are scrubbed to the same standard as the rest of this site - the lesson survives, the internal detail does not.