Prometheus - Skills - Bill Gernert

What it is & why I use it

Prometheus is the time-series metrics engine in AutomationLab. It scrapes instrumented targets and exporters on an interval, stores the samples, and evaluates alerting rules over them. It sits alongside the broader monitoring stack, feeding dashboards and firing alerts on the numeric health of the cluster and the workloads on it.

I use it because the pull model and the query language are a clean fit for infrastructure: targets expose metrics, Prometheus discovers and scrapes them, and rules turn raw series into "this is wrong, tell someone." Exporters let me bring things that do not speak Prometheus natively - like hypervisor stats - into the same model.

How I use it in the lab

Scrape targets and exporters. Cluster and host metrics come from instrumented endpoints and purpose-built exporters that translate a system's stats into the Prometheus format.
Alerting rules over the series. Rules encode what "unhealthy" means - saturation, error rates, a target that has stopped responding - and fire when the condition holds.
Feeds the dashboards. The series back the visual layer, so the same numbers drive both the at-a-glance view and the alerts.
Exporters for the non-native. Where a system does not expose Prometheus metrics itself, an exporter bridges it in - keeping everything in one query model instead of a special case.

Architecture

Diagram coming Targets / exporters -> scrape -> time-series store -> rules + dashboards -> alerts. Scrubbed of target addresses and hostnames.

In action

Screenshot coming A rule evaluating over a series, and an exporter's targets showing healthy scrapes. Captured per the publishing checklist - target addresses and hostnames redacted.

Lessons learned & gotchas

Gotcha - a down target goes absent, not to zero The alert that quietly does not work: you write a rule like "fire when the metric drops to 0," reasoning that a dead service is a zero. But when a scrape target is down, its series does not become zero - it stops existing. The rule has no series to evaluate, so it never fires, and the exact outage you built it for slips through silently. The fix is to alert on absence and scrape health - on the target being unreachable (its up signal going to 0) or on the series being missing - not on the value hitting a number that a vanished series will never report. "No data" and "zero" are different states, and conflating them is how a monitor misses the thing it exists to catch.

Alert on the scrape, not just the value. A target that has stopped responding is often the most important alert of all, and it lives in scrape health, not in the business metric.
Watch cardinality. A label with unbounded values (an id, a timestamp) quietly explodes the series count and the memory with it. Keep labels to bounded, meaningful dimensions - high cardinality is a slow-motion outage.
Rule for the state you fear, then test it by causing it. The only way I trust a "target down" alert is to actually stop a target in a safe window and watch it fire. A rule you have never seen trigger is a guess.

Impact

Pull

based scrape model

up == 0

catches a dead target

Exporters

bridge the non-native

Prometheus gives the lab a rigorous, queryable view of its own numbers - and the absent-versus-zero lesson is the one I would hand to anyone new to it. The most dangerous monitoring failure is not a wrong number; it is an alert that silently never fires because the thing it watches disappeared instead of changing value.

Code & further reading

Rules & exporters (link coming) Prometheus docs

Published rules and exporter config are scrubbed of target addresses, hostnames, and any secret references before they go public.