# Backup and disaster recovery

![Backup and restore lifecycle](../../docs/architecture/generated/backup-restore.svg)
![Disaster-recovery dependencies](../../docs/architecture/generated/dr-dependencies.svg)

Each run writes an allowlisted namespace and node inventory, selected workload
objects from `hybrid-cloud` and `portfolio-dev`, and ConfigMap names and keys.
The script does not query Secret objects or export ConfigMap values, kubeconfig,
or database contents. It does preserve selected workload specs; literal values
in environment variables, commands, arguments, or annotations are not
independently scrubbed and could contain sensitive data if a workload stored it
there. Treat the encrypted archive as sensitive and inspect it before any wider
use. The archive is compressed, encrypted with age, uploaded, compared with the
remote object size, and removed from the pod. The script also attempts to write
structured status to Cloud Logging, but logging failure does not fail the
backup.

On 16 July 2026, a read-only check found the CronJob present and unsuspended;
its latest successful run was recorded at 14:17:53 UTC. An authorization check
confirmed that its ServiceAccount could not get Kubernetes Secrets.

Retention is 7 days for `daily/`, 28 for `weekly/`, 90 for `monthly/`, and 7
days for noncurrent versions. This is not a complete application or cluster
backup. It omits Secret and ConfigMap values, RBAC and ServiceAccounts, PDBs,
PVCs and storage contents, CRDs, GitOps bootstrap dependencies, and databases.
Those require authoritative configuration, application-consistent data backups,
documented restore order, and separate testing.

Verification is deliberately review-only:

```bash
BACKUP_BUCKET="${BACKUP_BUCKET:?set the private backup bucket}" scripts/verify-backup.sh
scripts/restore-backup.sh \
  "gs://${BACKUP_BUCKET}/daily/OBJECT.tar.gz.age" \
  "${AGE_IDENTITY_PATH:?set the offline age identity path}"
```

The restore script downloads the selected object, decrypts into a temporary
directory, lists archive members, and deletes the temporary data. It does not
extract files, run a Kubernetes server-side dry run, apply resources, restore
data, or test application consistency. The 15 July 2026 operational record
reports successful decryption and listing of a 4,973-byte test archive. Any
actual restore must extract into an isolated directory, review every object,
recover omitted dependencies from their authoritative sources, and verify the
application after a controlled apply.