Docs / Guardrails

Drift checks

Planned Part of the routine checks that run across every stage. Everything below is the planned design.

Drift detection for on-premise infrastructure answers one question every night: does what is running still match what the repository says should be running? Someone fixes a firewall rule by hand at 2 a.m., a VM gets an extra disk from the hypervisor console, a package is upgraded outside the patch window. None of that is wrong in the moment, but if it never makes it back into Git, the next planned change can quietly undo it, or fail in a way nobody expects.

Drift detection: the nightly checks

The design runs two dry runs on the site runner each night, against the main branch of each environment’s repository:

• OpenTofu. tofu plan -detailed-exitcode for every OpenTofu root under tofu/. On Proxmox VE, the primary target, that covers VMs, storage and networks; on vSphere or another platform you already run, it covers what that platform exposes. After a clean apply, the expected result is “no changes”. Exit code 2 means the real VMs, disks or networks differ from the code. See OpenTofu.
• Ansible. The same playbooks used for real runs, with --check --diff. Any task that reports “changed” is a host that no longer matches its role: a config file edited, a service enabled, a user added. On bare-metal sites this is the main check. See Ansible.
• Argo CD, from stage 3. Kubernetes applications don’t wait for the night. Argo CD compares the cluster with Git continuously and marks an application OutOfSync when they differ. The nightly report includes that sync status for every application. The Kubernetes nodes themselves are installed and upgraded by Ansible, so the Ansible check covers them.

None of these checks change anything. They only read, compare and report. Argo CD can also be set to correct drift on its own; whether to allow that is a per-application decision written in the repository.

# planned example: nightly drift job on the site runner
tofu -chdir=tofu/site-a plan -detailed-exitcode -lock=false
ansible-playbook -i inventory site.yml --check --diff

What counts as drift

Not every difference is worth a message. The planned rules:

Ignore lists live in Git like everything else. Adding an entry is a change request, so nobody can hide drift by muting it on the side.

Reporting

Results go into the one-page status report: each environment, when it was last checked, and either “matches the repository” or a short list of what differs, with the plan or diff attached. Drift found on a production environment raises an alert. Drift on a test environment is shown in the report but doesn’t page anyone.

A clean result is recorded too. Knowing that a site matched its code last night is useful on its own, and it feeds the audit trail.

Fixing drift: by code, not by hand

When drift is found there are two honest answers, and both go through Git:

1. The manual change was right. Write it into the repository, open a merge request as a change request, get it approved. The next nightly check comes back clean.
2. The manual change was wrong or unknown. Re-apply the code through the normal GitLab CI pipeline (or let Argo CD sync it, for Kubernetes applications), which puts the system back to what was approved.

What the design avoids is a third answer: logging in and reversing the change by hand. That replaces one undocumented change with another, and leaves no record of why.

Last updated 25 October 2026 · Questions or corrections: sales@infratractor.com