Bill Gernert
Observability

Grafana

The single pane over the lab - dashboards that pull from metrics, APIs, and JSON endpoints, including a hand-tuned Infinity data source.

What it is & why I use it

Grafana is the visualization and dashboarding layer for AutomationLab. It sits on top of the monitoring stack and turns raw signals into something an operator can actually read at a glance. I use it as the front door to the platform's health: cluster state, provisioning outcomes, self-healing activity, and service-level views all land here.

What makes it interesting is that not everything I want to chart comes from a time-series database. Some of it lives behind plain REST/JSON APIs, so I lean on the Infinity data source to pull JSON directly into panels - which is exactly where the sharp edges live.

How I use it in the lab

  • Metrics dashboards. Cluster and host health from the metrics backend, with alerting thresholds visualized alongside the raw series.
  • API-sourced panels. The Infinity data source pulls JSON from internal service APIs so operational data that never went into a TSDB is still charted.
  • Provisioning & healing views. Panels that summarize pipeline runs and auto-heal outcomes, so the lab's automation is observable, not a black box.
  • Access-controlled. Grafana is published behind the same zero-trust edge and SSO as every other app in the lab - no bare port on the internet.

Architecture

Diagram coming Data-source topology: the metrics backend and the Infinity JSON path into Grafana, and how dashboards are published behind SSO. Addresses and hostnames scrubbed.

In action

Screenshot coming A live dashboard combining a metrics panel with an Infinity JSON-sourced table. Captured with any internal identifiers cropped per the publishing checklist.

Lessons learned & gotchas

Gotcha - Infinity backend parser The Infinity plugin's backend parser is a Go implementation of JSONata, and it is not the full language. It handles a path-plus-filter root selector and plain top-level columns - but it does not support object-construction expressions, computed columns, or nested column definitions. Queries that work in a JSONata playground can silently return nothing here.
  • Prove it with live probes, not docs. The fastest way I found the real boundary was to fire small, deliberate probe queries at a known endpoint and watch what parsed and what came back empty. The behavior, not the documentation, is the spec.
  • Shape the JSON at the source when you can. If the parser cannot compute a column, the cheaper fix is to have the upstream API return the field already shaped, rather than fighting the query language to derive it.
  • Backend vs. browser parsing differ. Choosing the parser mode changes which expressions are legal. Pin the mode you are testing against so you are not debugging two engines at once.

Impact

1
pane over the lab
2
source types unified
SSO
on every dashboard

Grafana is where the lab stops being a set of independent services and starts being a system I can watch. The Infinity work in particular meant I could bring non-time-series operational data - things that only exist as API responses - into the same view as metrics, without standing up a whole extra pipeline to store it first.

Code & further reading

Dashboards JSON (link coming) Infinity data source

Exported dashboard JSON is scrubbed of internal endpoints and identifiers before it is published.