Jenkins

What it is & why I use it

Jenkins is the automation engine of AutomationLab. Every meaningful action - stand up a VM, register it for monitoring, configure it, decommission it, offboard a user - runs as a Jenkins pipeline. It is the place where the lab's many tools (the IPAM, Terraform, the secrets manager, the monitoring API, configuration management, identity) are orchestrated into one repeatable flow with approval gates and an audit trail.

I run it deliberately in the "boring, dependable" spot: the controller itself is config-as-code, the jobs are pipeline-as-code, and its secrets come from Vault at runtime. A controller I can delete and rebuild from git is a controller I actually trust.

How I use it in the lab

• Configuration as Code (JCasC). The controller's configuration - security realm, credentials wiring, jobs, agent settings - lives in git and is applied declaratively. There is no hand-clicked controller state to lose.
• Pipeline-as-code. Each job is a Jenkinsfile-<purpose> in the repo, reviewed like any other change. The provisioning flow, with its allocate -> plan -> approve -> configure -> register stages, is one such file.
• Secrets from Vault, not from Jenkins. The controller authenticates to Vault with an AppRole and pulls credentials at run time, so tokens are not stored in job config or printed in logs.
• Approval gates for irreversible actions. Destroying infrastructure or disabling an identity pauses for a human decision - and, for the most sensitive flow, enforces segregation of duties (below).
• Roll-forward semantics. Pipelines are written so a partial failure cleans up after itself rather than leaving half-built state, and a decommission reverses every step provisioning took.

Architecture

The pipeline worth calling out is the four-eyes deprovision control - offboarding an identity is the most destructive thing the lab automates, so it is the most guarded.

• Segregation of duties. The person who requests a deprovision cannot be the person who approves it. The pipeline captures both identities and refuses to proceed if they are the same.
• Approver allowlist. Approval is only accepted from an explicit allowlist of authorized approvers - not from "whoever happens to have a Jenkins login."
• Disable first, delete later. Offboarding disables the account immediately and defers destructive cleanup to a later sweep, so a mistake has a recovery window.

In action

Lessons learned & gotchas

• Segregation of duties must be enforced in code, not policy. "Don't approve your own request" written in a runbook is a suggestion. The same rule as a pipeline check that compares the two identities and aborts on a match is a control.
• Allowlist, don't blocklist, your approvers. Enumerating who may approve is safe by default; trying to enumerate who may not is a hole waiting to be found.
• A rebuildable controller changes how you operate. Because the controller is JCasC in git, I test config changes as PRs and can restore the whole controller from source. Once you have that, you stop being afraid to touch it - which is the point.
• Keep the Jenkins credential store thin. The more secrets Jenkins holds natively, the bigger the prize it becomes. Pulling from Vault at run time keeps the controller from turning into the crown-jewels box.

Impact

Jenkins is where the lab's guardrails actually live. Config-as-code makes the platform reproducible; pipeline-as-code makes every change reviewable; and hardening the highest-stakes pipeline to fail closed is the difference between automation that is convenient and automation you can trust with destructive actions.

Code & further reading

Published pipeline and JCasC code is scrubbed of internal addresses, hostnames, usernames, and credential wiring before it goes public.