Docs / Guardrails
Alerts
Planned design. TractorStack is in development. This page describes how it is designed to work; nothing here is available to install yet. See the roadmap for the status of each part.
Planned Everything below is the planned design.
Alerts and notifications for runs and checks in TractorStack follow one rule: tell a named person about something they need to act on, and say nothing otherwise. A backup that ran fine last night is not news. A restore test that failed is, and it should reach someone who can fix it before the next backup depends on it.
Alerts and notifications: what triggers one
| Event | Why it matters | Urgency |
|---|---|---|
| A GitLab CI pipeline fails during apply | The system may be half changed | Now |
| The nightly test restore fails | You may not have a usable backup | Now |
| A backup did not run at all | Same as above, and easy to miss | Now |
| Drift found on production | The repository no longer describes reality | Next working day |
| A certificate expires within the warning window | Services stop talking to each other on the day it lapses | Escalates as the date gets closer |
| Disk or volume above its threshold | PostgreSQL and RustFS fail badly when a disk fills | Escalates with usage |
| A routine check fails to run | Silence is not the same as health | Next working day |
Service-level alerts, such as an application returning errors or a host going down, come from monitoring, as do Argo CD applications that stay out of sync or unhealthy from stage 3. The design sends both kinds through the same routing so people don’t have two places to watch.
Who gets told
Routing is planned to live in the environment’s repository: which events go to which people or groups, per environment. A small file is enough for most sites.
# planned example: alerts.yaml
routes:
- match: { environment: production, urgency: now }
notify: [ops-oncall-chat, ops-email]
- match: { environment: production }
notify: [ops-email]
- match: { environment: test }
notify: [daily-summary]Because the routing is code, a change to who gets woken up is reviewed like any other change, and the audit trail shows when it changed.
Email or chat
The design plans two delivery types to start with:
- Email, through your own mail relay. Works everywhere, including offline sites with an internal mail server.
- Chat, through a webhook to a team channel in a self-hosted or hosted chat tool, where your policy allows outbound traffic.
Alerts from monitoring, such as disk space, a Patroni failover or a host that stopped reporting, are defined as Grafana alert rules, since Grafana is already the tool we use for dashboards. Alerts from runs and checks come from the GitLab pipelines and the site runner.
Paging services and phone calls are left to the tools you already use. Where you have one, the plan is to send to it by webhook rather than build our own.
Avoiding alert fatigue
An alert that fires every day gets ignored, and then the one that matters gets ignored with it. The planned safeguards:
- One message per problem. A failing check alerts once, then updates the same thread or sends a reminder on a schedule, not every few minutes.
- A recovery message. When the problem clears, people are told, so nobody keeps chasing it.
- Thresholds that escalate. A certificate 30 days from expiry is a line in the daily summary; 7 days out it is an alert.
- A daily summary for everything that is worth knowing but not worth an interruption.
- Review the noisy ones. If an alert fires often and nobody acts, the threshold or the alert is wrong. Fix it in the repository.
Every alert should name what failed, where, and the first thing to look at, usually a link to the run log and the matching runbook in Fixing common problems.
Last updated 25 October 2026 · Questions or corrections: sales@infratractor.com