Bill Gernert
CI/CD & GitOps

Kaniko

Daemonless container image builds that run as an ordinary pod inside the cluster - no privileged Docker socket, and every build trusted only after its digest changes.

What it is & why I use it

Kaniko builds container images from a Dockerfile inside a Kubernetes pod, without a Docker daemon and without needing the privileged access that mounting the host's Docker socket would require. In AutomationLab it is how the custom images the pipelines depend on - like the provisioning agent image - get built and pushed to the registry.

I use it because the alternative, bind-mounting a Docker socket into CI, hands the job a root-equivalent foothold on the node. Kaniko keeps image builds inside the same least-privilege, scheduler-managed world as everything else the cluster runs.

How I use it in the lab

  • Builds as pods. A build is a one-shot pod: it checks out the context, runs the Dockerfile, and pushes the result to the registry, then completes.
  • Agent images for the pipelines. The tools a provisioning run needs - the IaC binaries, configuration-management collections, cloud SDKs - are baked into an agent image built this way, so pipeline runs start from a known, versioned toolset.
  • Registry push with scoped credentials. The push credential is provided to the build pod as a mounted secret, not embedded in the image or the manifest.
  • Digest-verified promotion. A build is only "done" once I have confirmed the pushed digest changed from the previous build - see the gotcha below, which cost me real time to learn.

Architecture

Diagram coming Build flow: git context -> Kaniko build pod -> registry push -> digest check -> pipelines consume the new image. Scrubbed of registry addresses and hostnames.

In action

Screenshot coming A build pod's log ending in a successful push, next to the registry showing the new digest. Captured per the publishing checklist - no registry addresses or credentials.

Lessons learned & gotchas

Gotcha - a "successful" build log can be a lie A one-shot Kaniko build pod can silently reuse a stale, already-completed pod. If the pod already exists, kubectl apply reports configured, not created, and does not re-run anything - so you happily tail logs from an old build and conclude the new one worked. Nothing errors. You just deployed the previous image and believed it was the new one.

The rule I now follow every time, without exception:

  • Delete the build pod before applying. Never rely on apply to replace a completed pod; remove it first so the new build actually runs.
  • Confirm apply reports created, not configured. That one word is the difference between a fresh build and a no-op.
  • Trust the digest, not the log. Verify the pushed image digest changed from the prior build. A green log proves a pod ran; only a changed digest proves this build produced this image.
  • Verify by digest everywhere downstream. The same principle carries into how the pipelines pin images: a tag can move, a digest cannot, so pinning by digest is what makes "which image ran?" answerable after the fact.

Impact

0
privileged Docker sockets
Digest
the real proof of a build
1 image
versioned pipeline toolset

Kaniko lets the lab build its own tooling images the same way it runs everything else - as scheduled, unprivileged workloads. The lasting lesson was less about Kaniko and more about verification discipline: a log that says "success" is not evidence the thing you intended actually happened. Checking the digest turned a class of silent, confusing failures into something I can no longer fool myself about.

Code & further reading

Build manifests (link coming) Kaniko

Published build manifests are scrubbed of registry addresses, hostnames, and credential wiring before they go public.