Monitoring

Planned design. TractorStack is in development. This page describes how it is designed to work; nothing here is available to install yet. See the roadmap for the status of each part.

Planned Committed, and built after stage 1. Everything below is the planned design.

Observability: monitoring, logs and metrics, all running on your own hardware. Grafana is the dashboard and alerting tool our team already uses on the systems we run today, so it sits at the centre of the design. Around it, the plan is a small, well-known set of open-source tools: Prometheus collects metrics and Loki collects logs, and Grafana shows both. Nothing is sent to a hosted service, so the monitoring keeps working on a closed network and its data stays with the rest of yours.

Observability: monitoring, logs and metrics in one place

Prometheus scrapes metrics from each host and service at a regular interval and keeps a history you can query.
Loki stores logs and indexes them by labels such as host, service and stack. It can keep its data in S3-compatible storage, so the design points it at a bucket on RustFS.
Grafana, which we already use, shows dashboards for each layer and handles alert rules, with sign-in through your own accounts.

Ansible installs and configures all three from the same repository as everything else. In stages 1 and 2 they run as a Compose stack; from stage 3 they can move into the cluster and be deployed by Argo CD like any other app.

What is watched

Area	Examples of what the design checks
Proxmox VE hosts and VMs	CPU, memory, disk, cluster membership, VM state. On vSphere or bare metal, the equivalent host metrics
Operating system	Disk space, load, time sync, failed services, pending security updates
PostgreSQL and Patroni	Which node is the leader, replication lag, connections, a failover event
RustFS	Disk usage, request errors, whether the S3 endpoint answers
Backups	Last successful backup, last passing restore test, off-site copy age
Certificates	Days until expiry for every internal endpoint
Delivery	GitLab pipeline failures, Argo CD apps out of sync or unhealthy
Kubernetes, from stage 3	Node health, pods restarting, storage and ingress errors

Applications fit in without extra work if they follow the Kubernetes-ready rules: logs to standard output and metrics in Prometheus format.

A planned example

Illustrative only, not from a release. A Prometheus scrape job Ansible would write for the database nodes:

# Planned example only: prometheus.yml (fragment)
scrape_configs:
  - job_name: postgres
    static_configs:
      - targets: ["db1.example.internal:9187"]   # one entry per database node
  - job_name: patroni
    static_configs:
      - targets: ["db1.example.internal:8008"]   # Patroni REST API

From numbers to action

Dashboards are for looking; alerts are for being told. Each alert rule is planned to link to a runbook in Fixing common problems, so the person who gets woken up has first checks in front of them rather than a bare metric name.

Watching the watcher

If the monitoring host goes down, it can’t tell you. The design adds a simple check from a second place, such as the second host in stage 2, that raises an alarm when Prometheus stops answering. Retention periods and disk sizes for metrics and logs are set in the repository, so they are reviewed like any other change.

Limits

Monitoring shows symptoms. It doesn’t replace the restore tests in Backups, which prove recovery actually works.
Log volume grows quickly. Expect to tune what is kept, and for how long, in the first months.

Last updated 25 October 2026 · Questions or corrections: sales@infratractor.com