Automated server provisioning

What it does

A server is a fact recorded in five places at once: the address in address management, the machine on the hypervisor, the host in monitoring, the resource in infrastructure state, and the entry in configuration inventory. Build one by hand and any of those can silently disagree, or worse, silently agree with a machine that already exists. The failure this pipeline is built against is the one everybody has seen: a new host quietly reconciles a live one because nothing checked every system that already knew the name.

1. Validate input
2. Reachability pre-flight - refuse a target whose network path is untested
3. Name collision check - ask every system that could already know the name
4. Allocate an address from address management
5. Select a node by live free memory
6. Plan, in this machine's own infrastructure-state workspace
7. Human approval (60-minute timeout)
8. Apply - the roll-forward boundary
9. Register monitoring, minting a unique per-host key into the secrets manager
10. Configure with the agent (already trusted)
11. Register DNS
12. Commit the host manifest

The Windows path

Windows runs the same backbone and diverges where it must: it mints a per-host local-admin password into the secrets manager, resolves a template by tag rather than booting a cloud image, and waits for remote management to be stably up (six consecutive good checks) because first-boot init flaps the port and a single successful probe is a lie. Same spine, different limbs, one place where the difference lives instead of a second pipeline to keep in sync.

What broke, three times, and what I did about it

One: the gate could not see. The collision check asked the address manager for devices and virtual machines. But a host provisioned by this pipeline is recorded as an address with a DNS name, not as a device object. So the query came back empty, the API was perfectly healthy, and the gate said "no collision" about a host that was already alive. Fixed by also querying the address records, and by adding a deterministic scan of the configuration inventory: reading a file beats guessing from a naming convention.

Two: the gate could see, and nobody was listening. This one is my favourite. The detector correctly found the collision, printed it, and exited non-zero. The pipeline proceeded anyway. The stage ran the checker piped into tee so the report could be saved, and a shell pipeline exits with the status of its last command. That was tee, which always succeeds. The gate had been reading tee's exit code the entire time it existed. Fixed by redirecting instead of piping, and by adding a fail-safe so an unset or unparseable exit code can never read as clean. The same bug was sitting in the reachability pre-flight; it went in the same pass.

script {
    def rc = (env.COLLISION_RC ?: '').trim()
    if (!rc.isInteger()) {
        // Fail-safe: an unset/unparseable exit code must NEVER read as clean. Gate-or-fail.
        error("Name collision check produced no usable exit code (COLLISION_RC='${rc}'); refusing to proceed ungated.")
    }
    if (rc != '0') {
        // 2 = confirmed collision; 3 = unverifiable (fail-loud). Both gate; wording differs.
        def collision = rc == '2'

The fix for bug two, and the fail-safe it taught me to add.

Three: rejecting the plan leaked the address. Reject the approval and the reserved address stayed reserved. The release ran in the pipeline's failure handler, but a rejected approval does not fail a build, it aborts it. Fixed by moving the release to the handler that covers failed, aborted, and unstable alike, still guarded so that a machine which already exists keeps its address rather than having it yanked out from under it.

The rule worth stealing

Underneath all three is one rule, and it is the part worth stealing: an unreachable system is never "clean". If a system that could hold a conflicting name cannot be reached, that is not a pass, it is an unknown, and an unknown still stops the build.

def evaluate(hits):
    """Pure verdict over the gathered hits. 'collision' > 'error' > 'clean'."""
    if any(h["present"] is True for h in hits):
        return "collision"
    if any(h["present"] is None for h in hits):
        return "error"
    return "clean"


EXIT = {"clean": 0, "collision": 2, "error": 3}

Present means collision. Unknown means error. Only proven-absent is clean.