Docs / Make changes safely

Rollout waves

Exploring An idea we are working through. Nothing here is committed, and the details will change.

Rolling out one change across many sites is where small mistakes become large ones. A patch that is fine on one server can break twenty if it goes everywhere at once. Rollout waves are an idea we are exploring for TractorStack: apply the change to a few sites or environments first, check them, then move on in ordered groups, and stop the moment something looks wrong.

Who this is for

Most customers with one site and a couple of environments won’t need this; the normal pipeline already applies to test before production. Waves start to matter for:

• Managed service providers running the same stack for many customers, each in its own repository or environment. See teams and tenants.
• Multi-site organizations such as clinics, branch offices or factories, each with its own small stack built from the same stack kit.
• Larger single customers with several environments that should get the same OS patch or certificate change in a controlled order.

Rolling out one change across many sites, in waves

The rough shape we have in mind:

1. Wave 0, the canary. One internal or low-risk site. Often a lab or test environment that mirrors production.
2. Wave 1, a small group. A handful of real sites, chosen to cover the variety you actually run: different hardware, hypervisors or network setups.
3. Later waves, larger groups. The rest, in batches sized so that a bad change affects a number of sites you could repair by hand in a day.

Each wave uses the same reviewed change and the same approved version. What differs is only the list of targets, which comes from sites and environments in your repositories.

The check after each wave

A wave is only finished when its checks pass. The checks we are considering are the same ones a careful engineer would look at by hand:

• health and readiness endpoints answer on every site in the wave;
• a follow-up tofu plan and ansible-playbook --check show no remaining difference;
• no new alerts from monitoring during a short soak period;
• for database changes, replication is healthy and the next backup completes.

The soak period matters. Some problems, like a disk that fills slowly or a certificate that fails at renewal, only show up after a while.

Stop on failure, and what happens next

If any check fails, the rollout stops. Sites that already have the change keep it; sites in later waves stay untouched. Nothing is rolled back automatically, because an automatic rollback of infrastructure can do as much harm as the original change. Instead, an alert goes to the people responsible, and they decide whether to fix forward, revert in Git and roll the revert out, or skip a site. Every wave, check and decision is recorded in the audit trail.

Last updated 25 October 2026 · Questions or corrections: sales@infratractor.com