Bill Gernert
Platform & Orchestration

Kubernetes / K3s

The lightweight K3s cluster that runs the platform's own services - a real Kubernetes, sized for the lab, with tradeoffs made on purpose rather than by accident.

What it is & why I use it

K3s is a lightweight, fully conformant Kubernetes distribution. It is the service plane of AutomationLab: the CI controller, the GitOps engine, ingress, and the lab's own applications all run as workloads on it. Terraform makes VMs; Kubernetes is where the things that run on those VMs live.

I chose K3s over full upstream Kubernetes because it gives me the same API and the same mental model with a fraction of the operational weight - the right size for a lab that needs to be real Kubernetes, not a toy, without a control plane that dwarfs the workloads it runs.

How I use it in the lab

  • The platform's services run here. The automation controller, GitOps reconciler, ingress, and lab applications are all Kubernetes workloads, deployed and reconciled from git.
  • Control plane plus spread workers. A control-plane node runs the API and scheduler; worker nodes carry the workloads and are spread across separate physical hosts so a single machine failure does not take everything with it.
  • Declarative, GitOps-managed. What runs on the cluster is declared in git and reconciled continuously - the cluster is cattle, described in code, not a hand-tended pet.
  • Secrets and ingress wired in. Secrets come from the secrets manager and traffic arrives through the ingress behind the zero-trust edge - the cluster is a citizen of the wider platform, not an island.

Architecture

Diagram coming Control-plane node plus workers spread across hosts, workloads reconciled from git, ingress behind the edge. Scrubbed of node addresses and hostnames.

In action

Screenshot coming A node/workload view with the platform services healthy across the cluster. Captured per the publishing checklist - node addresses and hostnames redacted.

Lessons learned & gotchas

Gotcha - a stale kubeconfig makes every command fail identically One of the most time-wasting failures is not in the cluster at all - it is in the client config. When a kubeconfig points at a control-plane endpoint that no longer exists (a rebuilt or re-addressed control node), every kubectl and every tool that shells out to it fails with the same timeout, and it looks like the cluster is down when the cluster is fine. The tell is that everything fails the same way at once. The fix is to point the client at the live API endpoint - I keep an explicit API server override handy so tooling can reach the real control plane even when a config on disk is stale.
  • The single control plane is a deliberate SPOF. Spreading workers buys resilience for workloads, but one control-plane node means losing it stops scheduling and API access. That is an accepted lab tradeoff - the point is that it was chosen and documented, so nobody mistakes it for high availability it does not have.
  • Distinguish "cluster down" from "I cannot reach it." Client-side, DNS, and edge problems all masquerade as an outage. Before declaring the cluster broken, prove you can actually reach the API from a known-good client.
  • Lightweight is a feature, not a compromise. Because the distribution is the same API, everything learned here transfers to full Kubernetes - the smaller footprint just removes weight the lab does not need.

Impact

1
service plane for the lab
Spread
workers across hosts
GitOps
managed workloads

K3s is the substrate everything user-facing runs on. The lasting lesson is less about Kubernetes and more about honesty: name your single points of failure out loud, and learn to tell a real outage from a client that is simply pointed at the wrong place - because those two look identical right up until you check.

Code & further reading

Cluster manifests (link coming) K3s docs

Published manifests are scrubbed of node addresses, hostnames, and any secret references before they go public.