Bill Gernert
Security & Identity

HashiCorp Vault

The secrets backbone of the lab - every pipeline and service authenticates with a scoped AppRole and reads only the paths it needs, nothing more.

What it is & why I use it

Vault is the single source of truth for secrets in AutomationLab: API tokens, per-host monitoring keys, service credentials, and the personal access tokens that automation uses to open pull requests. Nothing sensitive lives in a repo, a pipeline definition, or a container image - it lives in Vault and is fetched at the moment it is needed.

The reason it matters is not "encryption at rest" - it is identity and scope. Every consumer proves who it is, gets a short-lived token, and is authorized to read a deliberately narrow slice of the secret tree. A leaked credential is then a bounded problem, not a master key.

How I use it in the lab

  • AppRole for machines. Pipelines and services authenticate with the AppRole method - a role ID plus a secret ID - rather than a human logging in. The credentials are injected as environment or mounted secrets, never baked into an image.
  • KV v2 for secret storage. Secrets live in a versioned key-value store under a consistent <service>/... path convention, so every secret has an obvious, predictable home and a version history to roll back to.
  • Least-privilege policies. Each role is bound to a policy that grants read on exactly the paths that role needs. A monitoring agent cannot read PR tokens; the PR helper cannot read monitoring keys.
  • Vault-backed credential helpers. Small scripts fetch a secret at runtime and hand it straight to the tool that needs it - the secret is never written to disk and never printed. The PR-opening helper is the clearest example (below).

Architecture

A concrete, scrubbed example: the portable PR-token AppRole that lets the automation open pull requests from any box without a token file on disk.

  • A dedicated read-only policy grants read on a single KV path - the one holding the PR token - and nothing else.
  • A dedicated AppRole is bound to that one policy. Its role ID and secret ID are themselves stored as a cluster secret, so a pipeline can bootstrap an AppRole login over Vault's HTTP API using only its in-cluster identity.
  • The helper logs in, reads the one path, hands the token to the PR tool in memory, and exits. No secret is written to disk at any step.
Diagram coming The AppRole login path: cluster identity -> bootstrap login -> scoped policy -> one KV read -> token used in memory. Scrubbed of addresses and real secret paths.

In action

Screenshot coming The credential helper logging in via AppRole and reporting a successful read, with no secret material shown. Captured per the publishing checklist - no tokens, paths, or addresses.

Lessons learned & gotchas

Gotcha - KV v2 hides a "data" segment On a KV v2 mount, the CLI path and the API path are not the same. You read kv get <mount>/foo on the command line, but the HTTP API path is <mount>/data/foo - and the JSON response wraps the secret one level deep under .data.data. Every hand-rolled API client I wrote hit this: a 403 or an empty result that looks like a policy problem is often just the missing data/ segment. Worse, a policy written for <mount>/foo will not authorize the API call to <mount>/data/foo.
  • Scope the policy to one path, then prove it. It is tempting to grant a role a broad wildcard "so it just works." The discipline that pays off is granting exactly one path and then testing that the role can read it and cannot read a neighbour. Least privilege you have not tested is just a hope.
  • Policy paths must match the auth method's real request path. Because KV v2 inserts data/, the metadata and data planes are separate capabilities. Granting read on the data path does not let you list versions, and vice versa - decide which the consumer actually needs.
  • Keep secrets out of argv. A token passed on a command line leaks into process listings and shell history. Helpers fetch and pipe in memory; the token never becomes a visible argument or a file.
  • Separate role per consumer. One shared "automation" role that can read everything defeats the point. A role per job means revocation and rotation are surgical, and an incident stays contained to one blast radius.

Impact

0
secrets in git or images
1 path
the PR role can read
0
tokens written to disk

Vault is what lets the rest of the platform be config-as-code without being secret-as-code. Pipelines run from public-safe definitions, yet still reach real credentials at runtime - each one scoped so tightly that "what could this leak reach?" has a short, boring answer. That is the whole goal of a secrets manager, and wiring it up per-consumer is where the real work lives.

Code & further reading

Credential helper (link coming) Vault AppRole KV v2

Published helper code is scrubbed of real secret paths, role IDs, and addresses. The policies and paths shown here are illustrative, not the live values.