Monitoring
Every container on this box is covered - logs and metrics - with no per-service setup. That is not a claim about how thorough the configuration is; it is a consequence of how two of the three collectors find their targets.
Why it covers everything for free#
| Signal | Collector | Coverage |
|---|---|---|
| Logs | alloy, via Docker service discovery, into Loki |
every running container, any stack or project. Labels: container, stack (the compose project), stream |
| Container metrics | cadvisor into VictoriaMetrics |
CPU, memory and network, every container |
| Host metrics | node-exporter |
including the textfile collector, which is how metrics a host program writes reach the database with no scrape target of their own |
| App metrics | postgres-exporter |
|
| The edge | Traefik's own Prometheus metrics | request rate, latency and error rate per router, service and entrypoint |
alloy and cadvisor discover through the Docker socket rather than a
hand-kept target list. A container you start is covered the moment it starts,
and a container you delete stops being scraped without anyone editing anything.
Traefik's metrics are on an internal entrypoint with no host port -
VictoriaMetrics reaches it by name over devnet. That is why a new service only
has to join devnet to be visible; see Adding a service.
What you get in Grafana#
Grafana auto-provisions the Prometheus and Loki datasources, five dashboards, and eight alert rules in three groups:
| Group | Alert | Fires when |
|---|---|---|
| core | Instance down | a scrape target stops answering for 2 minutes |
| core | Host memory high | host memory above 90% |
| core | Host disk almost full | a filesystem above 85% full |
| cluster | Kubernetes cluster down | any kube* target down for 5 minutes |
| cluster | Kubernetes node NotReady | the kubelet reports NotReady |
| update | Update rolled back or failed | a component's last update job did not succeed |
| update | Automatic updates paused | the night job will not touch a component until a person clears it |
| update | Update waiting for over 14 days | a patch or minor has been on offer for a fortnight |
Everything is provisioned from files under monitoring/provisioning/, so a
dashboard edited in the Grafana UI is not persisted the way a provisioned
one is. Edit the file.
What you see first#
Box health (monitoring/dashboards/box-health.json) is the landing page, set
by GF_USERS_DEFAULT_HOME_DASHBOARD_PATH in monitoring/compose.yml. Before it,
Grafana opened on its stock "Welcome to Grafana" page - a tutorial carousel and an
invitation to add a datasource, on a box with two of them.
It is six panels in one screen, and the test each one had to pass was what question does this answer that nothing else here answers:
| Panel | The question only it answers |
|---|---|
| Alerts firing | is anything wrong that a rule already knows about |
| Scrape targets, as a timeline | has a target been flapping - instance_down needs two full minutes, so a target that bounces every ninety seconds pages nobody |
| Host headroom | how close to the memory and disk alerts are we, measured with those alerts' own expressions |
| Log lines per second | is the log pipeline alive - Alloy can stop shipping while every container stays healthy, and nothing alerts on it |
| Updates | pins drifted from what the repository pins, and components discovery could not check. Neither has an alert. bothy_update_config_drift - the image matches and the merged compose configuration does not - is exported beside them and is on no panel yet either |
| How current this is | how long since the host booted, and since the updater last wrote - the numbers to its left are read out of that file |
Three things are deliberately not on it, and the reasons are the useful part:
- "N of M containers running." M does not exist in metrics. The portal knows which services are meant to be up; Grafana only knows which ones emit. A fraction with an invented denominator is the failure dataviz records twice.
- Per-container CPU and memory. That is the Cadvisor dashboard and the portal Overview, and neither is one glance.
- A Loki dashboard. Explore already is the log tool, with a query bar and a label browser; a page of log panels is a worse Explore. The one durable Loki question - are lines still arriving - is a panel on the summary instead.
The dashboards, and which standard each is held to#
dataviz is the standard. Only two of the five can be held to all of it, and that split is explicit rather than accidental.
| Dashboard | Panels | Origin | Held to |
|---|---|---|---|
| Box health | 6 | written here | all of it |
| Traefik - edge | 8 | written here | all of it |
| Cadvisor exporter | 10 | grafana.com 193 | the mechanical rules |
| PostgreSQL Database | 35 | grafana.com 455 | the mechanical rules |
| Node Exporter Full | 140 | grafana.com 1860 | the mechanical rules |
Why the imports are not rewritten. An upstream dashboard carries knowledge
about its exporter that a hand rewrite throws away, and it is re-importable as a
clean replace only as long as nobody has hand-edited it. So the mechanical half
of the standard is a script - just dash-normalise - and the workflow is: drop
the new export in, run it, read the diff, commit. scripts/checks/dashboards.sh
runs the same rules with --check in CI, and a sixth dashboard that is filed as
neither hand-written nor upstream fails it, because the quiet way a standard
rots is a file nobody classified.
What the script changes, and why each one is a defect rather than a preference:
- Colour by entity, not by rank. 110 panels of Node Exporter Full used
palette-classic, which assigns a colour by a series' index - so filtering out one mount repainted every mount after it in the list. - Gaps are gaps. The Cadvisor import drew a missing sample as zero on its CPU and both memory panels, which is a cliff that never happened, on exactly the panels you open when a container is behaving oddly.
- A colour ladder nothing can move is removed.
Pressureispercentunitwithmax: 1and thresholds at 70 and 90 - 7000% and 9000% - so a PSI bargauge had been permanently green since the day it was imported.SpeedandMTUhad a single green step and no threshold at all. - One of everything else: one datasource syntax (the imports had three
spellings of the same datasource), one refresh interval,
timezone: browserso two dashboards mean the same instant, a shared crosshair, no datasource picker on a provisioned dashboard, and no__inputsenvelope once nothing is left for the import wizard to fill.
One rule refuses to auto-fix: a series drawn against a second y-axis. The fix is to split the panel, which is a judgement about what the two halves are for. It happens to be true of all five dashboards already; the check keeps it that way.
What is deliberately left broken in the imports, so nobody re-opens it:
- The old panel vocabulary.
singlestatandgraphare gone from Grafana, and Grafana 13 migrates them at load time - so the colours and units of those panels come out of the migrator, not out of the file. Writing a modernfieldConfigthere would either be a no-op that looks like a fix or a collision with the migrator. - Threshold numbers somebody typed that are nonsense anyway.
RootFS Totalturns red above 70 bytes. A typed number cannot be proven unintended, and a hand edit to an import is lost on the next re-import. - Grafana's own categorical palette rather than the validated five-slot one in colour. Grafana has no per-dashboard palette, and pinning a colour per series name rots the moment a route or a mount is renamed. The rule that is enforced is by-entity rather than by-rank, which is the one that causes misreading.
- PostgreSQL Database's Kubernetes lineage. Its
namespaceandreleasevariables are extracted by regex from labels a Helm-deployed exporter sets and this box's compose exporter does not, so both read as empty - and$instanceresolves only because an empty-string matcher happens to match an absent label. It works; it works by accident. Rewriting it is a rewrite, not a normalisation.
Retention, and the two limits that matter#
VictoriaMetrics: 15 days. There is no size ceiling, because VM has no
equivalent of Prometheus' --storage.tsdb.retention.size: it drops whole data
parts once they are entirely outside the window, so the disk can briefly hold more
than fifteen days. At roughly 20 MB a day that is not a problem, and
-storage.minFreeDiskSpaceBytes makes it stop writing rather than fill the
disk. Fifteen days bounds what any dashboard can honestly show - which is why
the update_stale alert looks back thirteen days and not thirty.
Prometheus held the same fifteen days behind a 3 GB ceiling, and the ceiling
existed because of a real incident: retention.time alone means the volume grows
as cardinality x 15 days, bounded by how much things happen to emit rather than
by anything anyone chose. cAdvisor ran with no flags, produced 4,147 series of
which ~400 were reachable, and that is most of why the volume reached 1.8 GB. The
flag trim in monitoring/compose.yml took that to 1,036 series, which is why the
missing ceiling is not the same risk it was.
Loki: 168 hours (7 days). Loki ships with no retention whatsoever - the
stock configuration defines neither a limit nor a compactor that would enforce
one - so both stanzas had to be added. Everything else in loki-config.yml is
the image default, verbatim, which is deliberate: it makes the diff against
upstream exactly "the retention block".
A scrape job for something that does not run is not harmless#
Two jobs were removed when their services were, and the reasoning is worth carrying into anything you add:
redis-exporterandkafka-exportercame out when those services did.- The
docker-daemonjob came out because nothing in the repository read a singleengine_daemon_*series, and because the daemon setting that produces them is opt-in - so the target sat permanently down on any box that had not been hand-edited.
A permanently-down target does not just add noise. It destroys the usefulness of the only question worth asking of a target list: are all targets up? Once the answer is routinely "no, but that one is fine", nobody reads it again.
So: deleting a service means deleting its scrape job in the same change. The same rule applies to backup steps, and this repository has broken that one twice
- see Backups.
Reaching it#
VictoriaMetrics and Grafana publish their own host ports; just urls prints the
table for the box you are on. Neither is behind single sign-on. VictoriaMetrics
carries HTTP basic auth on all endpoints as its boundary (-httpAuth.*), and
Grafana its own login, both on the shared DEV_LOGIN_* credential from .env.
What to look at first#
- Grafana's landing page, Box health, is built to be the answer to "is the box well?" in one screen. Start there rather than at a dashboard list.
just doctorcovers scrape targets alongside containers, ports, routes, DNS and disk, and is the fastest way to see that a collector has stopped. Troubleshooting reads its output.- Logs for one container are a Loki query on the
containerlabel, and because Alloy discovers by socket, the label is the container's real name with no configuration anywhere. Use Explore, not a dashboard: there is deliberately no log dashboard here, because Explore is the better instrument. - The Overview in the console reads the same database - the CPU, memory and network panels are queries, not a second collector.
Related#
- Adding a service to the stack - what joining
devnetgets you - Backups - what is and is not preserved (metrics are not)
- Troubleshooting -
just doctor, and what its sections mean docs/ARCHITECTURE.md- the collectors in their full context