# Restore validation

![Backup and restore lifecycle](../architecture/generated/backup-restore.svg)

## Preconditions

- Select an approved object and verify that its age and failure domain fit the
  intended recovery question.
- Use an isolated workstation or disposable environment with enough encrypted
  temporary storage.
- Keep the age identity offline and outside Git. Confirm who may approve any
  extraction, dry run, or application step.

## Procedure

1. Verify object presence and metadata through
   [backup verification](backup-verification.md).
2. Run the repository inspection helper:

   ```bash
   hybrid-cloud/scripts/restore-backup.sh \
     "gs://${BACKUP_BUCKET:?set the private bucket}/${BACKUP_OBJECT:?set the object}" \
     "${AGE_IDENTITY_PATH:?set the offline age identity path}"
   ```

   The helper downloads, decrypts to a temporary directory, runs `tar -tzf`,
   prints the member list, and removes temporary data. It does not extract or
   apply resources.
3. Compare the member list with the expected allowlist. Stop if the archive is
   unreadable, incomplete, unexpectedly broad, or contains sensitive fields
   that should not have been collected.
4. If deeper work is approved, extract into a new isolated directory and review
   every manifest for namespace, ownership, immutable fields, stale status,
   external dependencies, and required data that the archive does not contain.
5. Use a server-side dry run only against the authorized disposable target and
   only after review:

   ```bash
   kubectl apply --dry-run=server -f "${REVIEWED_MANIFEST:?set one reviewed manifest}"
   ```

6. Treat an actual apply, data restore, and application-consistency test as a
   separate approved drill. Verify storage, identity, configuration data, DNS,
   TLS, ingress, monitoring, and user-visible behavior in dependency order.

## Expected result

At the current portfolio boundary, the encrypted object can be downloaded and
decrypted and its archive table can be inspected. This proves archive
decryptability and content visibility—not extraction, API acceptance, data
restore, application consistency, RPO, RTO, or end-to-end recovery.

## Failure and rollback

- Stop on download, decryption, archive-integrity, allowlist, or manifest-review
  failure. Preserve only redacted diagnostics.
- Never apply a partially reviewed archive to compensate for a failed check.
- A dry run changes no resources; correct the reviewed copy or choose another
  backup. An actual drill needs its own rollback plan and disposable target.
- Escalate missing application data, credentials, CRDs, operators, storage,
  identity, or external-service dependencies to their owners before continuing.