High availability & recovery

Planned design. TractorStack is in development. This page describes how it is designed to work; nothing here is available to install yet. See the roadmap for the status of each part.

Built for PostgreSQL high availability with Patroni. Planned for everything else on this page.

High availability and disaster recovery on-premise are two different problems, and it helps to keep them apart. High availability keeps a service running when one machine fails. Disaster recovery brings it back when a whole site, or a whole dataset, is lost. TractorStack’s design covers both, and the database part of high availability already exists and works internally.

High availability and disaster recovery: what each covers

Event	Handled by	Status
A database server fails	Patroni automatic failover	Built
A hypervisor host fails	Proxmox VE cluster HA restarts its VMs elsewhere	Planned
A storage disk or node fails	RustFS on its own disks, plus copies	Planned
Bad data or a mistaken delete	Point-in-time restore from backups	In design
The whole site is lost	Rebuild at a second site from Git and off-site backups	Planned

The Patroni PostgreSQL cluster

InfraTractor already runs a multi-node PostgreSQL stack managed by Patroni. Patroni manages leader election and automatic failover across several PostgreSQL nodes, using a distributed configuration store such as etcd. Applications reach the current primary through a routing layer such as HAProxy or a virtual IP, so they don’t need to know which node is leading. If the primary fails, a replica is promoted and the routing layer follows it. Details of our reference layout will be published.

Automatic failover needs a quorum: the configuration store has to agree on who the leader is, so a network split can’t leave two primaries. The reliable layout is three or more members of that store. A small witness node can be one of them, which matters for two-server sites (see Stage 2).

Virtual machines and storage

Proxmox VE is the primary target. A Proxmox cluster can restart a failed host’s VMs on another node, and the plan is to configure that from the repository for the services that benefit. It doesn’t replace Patroni: a restarted VM is minutes of downtime, while a database failover is much shorter. If you run vSphere or another hypervisor, its own HA features play the same role; on bare metal, only the application-level redundancy applies.

RustFS is planned to sit on its own disks, apart from the database. RustFS is young, so the design doesn’t lean on its clustering alone: important buckets are also copied to a second location. From stage 3, applications on Kubernetes get redundancy from running several replicas, deployed by Argo CD from GitLab.

Second-site disaster recovery

Because everything is code, a lost site is rebuilt rather than restored by hand. The plan: OpenTofu recreates the VMs at the second site, Ansible configures them, the database is restored from the off-site pgBackRest copy, and buckets from their copies. Your GitLab repository must be mirrored there too, or the rebuild has no instructions.

RPO and RTO in plain words

RPO is how much recent data you can afford to lose. With asynchronous replication a failover can lose the last few transactions; synchronous replication avoids that at some cost to write speed. A site loss loses whatever wasn’t yet copied off site.
RTO is how long you can afford to be down. A database failover is short; a VM restart is longer; a second-site rebuild is hours, not minutes.

We won’t quote numbers until they are measured on your hardware during a pilot.

Rehearsals

The design schedules drills: a planned Patroni switchover, pulling a hypervisor host, and a full restore at the second site at least once a year. Each one records how long it took and what the runbook missed, and fixes go back through a GitLab merge request. When something does break, start with Fixing common problems, and see Backups and Monitoring.

Last updated 25 October 2026 · Questions or corrections: sales@infratractor.com