A backup you have never restored is not a backup.

Two numbers decide everything else.

Before any backup tool is chosen, two figures have to be agreed in business terms. The recovery point objective is the maximum amount of data, measured in time, that the organisation can afford to lose. The recovery time objective is the maximum time it can be down before service is restored.

They pull on different levers. The recovery point objective drives how often backups are taken and whether transaction logs are shipped continuously. The recovery time objective drives architecture: whether there is a warm standby, how restoration is automated, and how much of the process is a person reading a document under pressure.

Setting them per system rather than once for the whole estate is the part most often skipped. A ledger and a public information page do not deserve the same answer, and pretending they do means overpaying for one and under protecting the other.

The restore drill.

A backup job that reports success has proved that a file was written. It has not proved that the file contains a usable database, that the restore procedure still matches the current schema, or that anybody on the team can run it without the person who wrote it.

HaveStack runs timed restore drills on a quarterly cadence, into a separate environment, against the written runbook. Every step is timed: provisioning, restoring the data, reconfiguring networking, validating application health. The measured total is what the recovery time objective is checked against, because a target nobody has timed is an aspiration.

Two rules make the drill worth running. It is performed against the runbook as written, so that gaps in the document surface rather than being filled in from memory. And correctness is signed off by somebody other than the person who performed the restore.

• Restore into a separate environment, never over the live one
• Follow the written runbook exactly, and fix the runbook where it fails
• Time every stage, not just the database restore
• Validate the application against the restored data, not just the row counts
• Independent sign off on correctness
• Record the result where the client can read it

Query performance decays quietly.

A query that was fast at ten thousand rows is a different query at ten million. Nothing broke; the data grew. Left alone, the first symptom is usually a user saying a page feels slow, which is the most expensive way to find out.

Slow query logging is enabled from the first day in production, not switched on during an incident. Queries crossing the agreed threshold are traced to the code that issues them, indexes are added or rebuilt where the plan justifies it, and the change is recorded. An index added without a recorded reason is an index nobody will dare remove later.

Every migration ships with the way back.

Schema changes are the class of change most likely to be irreversible and least likely to be rehearsed. A migration that adds a column is safe. A migration that drops one, rewrites a type or backfills fifty million rows is a small outage waiting for a bad afternoon.

HaveStack writes the rollback with the migration, applies both to a copy of production sized data before they touch the live system, and runs the migration inside a window agreed in advance. Where a change cannot be reversed, that fact is stated in writing before it is applied, and the backup taken immediately before it becomes the way back.

Replication is not failover until you have failed over.

A standby that is receiving data is doing half the job. The other half is promotion: the moment the primary is gone and something has to take its place, with the application following it. That path either has been walked or it has not.

Failover is exercised deliberately, replication lag is monitored against an agreed threshold rather than watched occasionally, and the promotion procedure lives in the same runbook as the restore. A team that has promoted a replica in a drill will do it in twenty minutes. A team that has only read about it will not.

Storage runs out on a schedule you can predict.

Disk exhaustion is among the most predictable failures in operations and one of the most common causes of unplanned downtime, because the growth curve is visible for months before it matters and nobody is looking at it.

Growth is tracked against provisioned capacity, a projection is included in the monthly report, and the threshold for action is set far enough ahead that expansion is a planned change rather than an emergency.

Written down, or it does not happen.

Maintenance that is not written down is a favour, and a favour stops the moment somebody is busy. These are the clauses that make the practice above enforceable.

Terms used here.

RPO

Recovery point objective. The maximum data loss, measured in time, that the organisation accepts. It sets backup frequency.

RTO

Recovery time objective. The maximum time from failure to restored service. It sets architecture.

PITR

Point in time recovery. Restoring to a chosen moment rather than to the last full backup, using continuously archived transaction logs.

Restore drill

A timed rehearsal of the documented recovery procedure into a separate environment, signed off by someone other than the operator.

Replication lag

How far a standby trails the primary. It is the practical floor on data loss during an unplanned failover.

References.

Published guidance this page draws on. HaveStack does not claim authorship of the practice described; it claims to follow it.

1. 01
   RTO and RPO in distributed databases. Cockroach Labs.
2. 02
   Recovery time and recovery point objectives: setting targets. Veeam.
3. 03
   Disaster recovery in site reliability engineering. SRE School.
4. 04
   Database disaster recovery runbook: RTO, RPO and PITR procedures. JusDB.

The rest of what stays under management.