Docs / Keep it running

Fixing common problems

Planned The runbooks below are planned. They show the shape we intend, not a finished set.

Troubleshooting common problems is easier at 3am with a short list of first checks than with a blank terminal. Each runbook is planned to live under runbooks/ in your GitLab repository, linked from the matching alert, and improved after every real incident through a merge request.

Troubleshooting common problems: how to use these

Start with the first checks, in order. They are cheap and usually find the cause. Write down what you did and when, because it goes into the incident note and the audit trail. If a fix needs a change to the infrastructure, make it through a change request, not by hand, or the next drift check will flag it.

A nightly backup failed

1. Read the pgBackRest log for the run, and check whether the RustFS endpoint answers.
2. Check free space in the backup bucket and on the database host.
3. Check that the backup credentials and encryption key are still valid.
4. Run the backup again by hand once fixed, and confirm the next restore test passes.

A restore test failed

1. Treat it as serious: the newest backup may not be usable.
2. Check whether the backup itself completed, or whether the scratch container failed to start.
3. Read which sanity query failed. A schema change can break a test that needs updating.
4. Try restoring the previous backup to see how far back you are covered.

A Patroni failover happened

1. Confirm which node is now leader and that applications reconnected through the routing layer.
2. Find out why the old primary dropped out: host, disk, network or the hypervisor.
3. Check the configuration store still has quorum.
4. Rejoin the old node as a replica only once its cause is fixed, then check replication lag.

A disk is full

1. Find what grew: database, write-ahead log, logs, images or a bucket.
2. On a database node, check that WAL archiving is working. A stuck archive fills disks fast.
3. On a Proxmox VE host, check thin-provisioned storage, which can fill before the VMs notice.
4. Free space safely, then grow the disk through OpenTofu so the change is recorded.

A certificate expired

1. Find which endpoint, and whether it came from the internal CA or cert-manager.
2. Renew it, restart or reload the service, and check clients reconnect.
3. Find out why the expiry warning didn’t reach anyone in time.

Drift was found

1. Read the tofu plan or Ansible --check --diff output to see exactly what differs.
2. Find who changed it and why, often an urgent manual fix.
3. Either bring the change into the repository or let the next apply undo it.

An Argo CD app is out of sync

1. Open the app’s diff in Argo CD and compare it with the last merge to GitLab.
2. Check for a failed sync, a missing image in your registry, or someone editing the cluster directly.
3. Fix it in Git and let Argo CD sync, rather than changing the cluster by hand.

A run or pipeline is stuck

1. Check the GitLab pipeline: is a job waiting for a runner, or for an approval?
2. Check the site runner is online and has disk space.
3. Check whether an OpenTofu state lock is still held by a run that died. Release it only when you are sure that run has stopped.

Last updated 25 October 2026 · Questions or corrections: sales@infratractor.com