Docs / Grow in stages
Stage 2: Resilient
Planned design. TractorStack is in development. This page describes how it is designed to work; nothing here is available to install yet. See the roadmap for the status of each part.
Built for the database part. Planned for the rest of the stage, which follows stage 1.
Stage 2: high availability with a Patroni PostgreSQL cluster, object storage on its own disks, and copies off site. The goal is practical: one failed server shouldn’t mean an outage you have to fix by hand, or lost data. Applications still run on Docker Compose, from the same GitLab repository, and it is still something one or two people can run.
Stage 2: high availability with a Patroni PostgreSQL cluster
This is the part that already exists. InfraTractor runs a working multi-node PostgreSQL stack managed by Patroni. Patroni manages leader election and automatic failover across several PostgreSQL nodes, using a distributed configuration store such as etcd. Applications reach the current primary through a routing layer such as HAProxy or a virtual IP, so a failover doesn’t need a config change or a redeploy. Details of our reference layout will be published.
What else the stage adds
- RustFS on its own disks. Object storage moves off the database disks, so a full or failed disk affects one service, not both.
- Off-site copies. Backups and important buckets are copied to a second location you choose, for example a colocation facility.
- A site-loss runbook in the repository, under
runbooks/, rehearsed on a schedule.
OpenTofu creates the extra VMs, on Proxmox VE first or the infrastructure you already run, and Ansible configures them from the same roles as stage 1. Database nodes should sit on different physical machines, ideally on different power and network paths. Several VMs on one hypervisor protect against very little.
Quorum, and the two-server site
Automatic failover is only safe when the cluster can agree on a leader. The configuration store needs a majority of its members to make that decision, which is what stops a network split from leaving two primaries. The reliable layout is three or more store members.
If your site has only two servers, add a small third machine as a witness. It holds a vote in the configuration store but no database. Without it, losing either server can stop the cluster from choosing a leader at all.
What a failover looks like
- The primary stops answering and its leader lock expires.
- Patroni promotes the healthiest replica and the routing layer sends new connections to it.
- Applications see dropped connections for a short time and reconnect. They need retry logic for this.
- An alert tells the operator, who follows the first checks in Fixing common problems.
- Once repaired, the old primary rejoins as a replica.
Replication and backups are different things
Replication copies every change, including mistakes. If someone drops a table, the replicas drop it too within moments. Replication protects against a dead server; backups protect against bad data. Stage 2 keeps the nightly pgBackRest backups and test restores from stage 1, and the off-site copy is what you reach for when the site is gone.
Rehearsing it
The design calls for a planned switchover and a site-loss drill on a schedule, ideally in a test environment built from the same repository. Each rehearsal records how long it took and what the runbook got wrong, and the fixes go back through a GitLab merge request.
What it doesn’t cover
- It isn’t zero-downtime. A failover is short, but connections drop.
- It doesn’t make the applications themselves redundant. If they need rolling updates across several hosts, you are heading towards stage 3.
See also High availability and recovery, PostgreSQL, RustFS and Backups.
Last updated 25 October 2026 · Questions or corrections: sales@infratractor.com