Bill Gernert
Observability

Zabbix

The lab's monitoring brain - hosts register themselves through the API with a unique encrypted key, and a classify-and-heal engine turns common failures into auditable, automatic fixes.

What it is & why I use it

Zabbix is the monitoring backbone of AutomationLab. Every provisioned host is registered into it automatically as the last step of the pipeline, so "monitored" is not a thing someone remembers to do later - it is part of what "provisioned" means. Beyond collecting metrics, it is the trigger source for the lab's self-healing.

I use it because it has a first-class API. That lets registration, templating, and remediation all be driven from code rather than clicked in a UI, which is the only way monitoring keeps up with infrastructure that is itself built from code.

How I use it in the lab

  • API-driven registration. The pipeline creates the host via the API, places it in the right host group, and attaches the standard agent template - no manual host creation.
  • Unique encrypted key per host. Each host gets its own pre-shared key, generated at registration and stored in the secrets manager; agent traffic is PSK-encrypted and required in both directions, so no host shares another's key.
  • Classify-and-heal engine. Triggers feed an engine that classifies a failure and applies an OS-aware remediation - for example, restarting a stuck service - and records every attempt and outcome so heals are auditable, not magic.
  • Tiered response. A fast Tier-1 path restarts common stuck services within minutes; anything it cannot safely fix escalates to a human with the context already gathered.

Architecture

Diagram coming Register -> monitor -> trigger -> classify -> heal -> record, with per-host keys drawn from the secrets manager. Scrubbed of addresses, hostnames, and key material.

In action

Screenshot coming The heal log showing a classified failure, the remediation applied, and the recorded outcome. Captured per the publishing checklist - no addresses, hostnames, or key identities.

Lessons learned & gotchas

Gotcha - the API is a machine door, not the front door The monitoring UI sits behind an identity-aware SSO proxy, which is correct for humans. But a script calling the API cannot complete an interactive SSO challenge - so automation has to reach the API endpoint directly, on the internal service path, rather than through the public, SSO-fronted hostname. Point a registration script at the pretty public URL and it gets an auth redirect page instead of JSON, and fails in a way that looks like a broken token. Machine-to-machine and human-to-UI are two different doors on purpose.
  • A unique key per host is worth the bookkeeping. Sharing one PSK across hosts is easier on day one and a disaster the day one host is compromised. Generating and storing a key per host at registration keeps a key leak scoped to a single machine.
  • A heal you cannot audit is a liability. Auto-remediation that does not record what it did, and whether it worked, erodes trust fast. Every heal writes an outcome, so the engine is accountable and I can see when it is masking a recurring problem instead of fixing it.
  • Register at the end, but never optionally. Making monitoring the mandatory final provisioning step is what prevents the classic "that box was never in monitoring" outage.

Impact

Auto
host registration
1 key
per host, encrypted
Minutes
Tier-1 auto-recovery

Zabbix is where the lab stops just watching itself and starts fixing itself. The combination that matters is automatic registration (nothing escapes monitoring), per-host keys (a leak stays contained), and audited self-healing (the automation is accountable). Together they turn monitoring from a dashboard you check into a control loop that keeps the platform healthy on its own.

Code & further reading

Heal engine (link coming) Zabbix API

Published automation is scrubbed of internal addresses, hostnames, key material, and secret paths before it goes public.