# Operations

![Terraform ownership map](../../docs/architecture/generated/terraform-ownership.svg)
![Observability architecture](../../docs/architecture/generated/observability.svg)

## Static site

```bash
scripts/build-static-site.sh
GOOGLE_CLOUD_QUOTA_PROJECT="${GCP_PROJECT_ID:?set the authorized project}" \
  FIREBASE_TOKEN="$(gcloud auth print-access-token)" \
  npx --yes firebase-tools@14.16.0 deploy \
  --only hosting:"${FIREBASE_SITE_ID:?set the authorized Firebase site}" \
  --project "${GCP_PROJECT_ID}" --non-interactive
curl -fsSI "${FIREBASE_URL:?set the generated Firebase URL}/"
```

CI is dev-only. Configure its deployer service account, WIF provider, and Gitea
OIDC secrets before expecting the deployment step to run. The workflow is
configured to request the generated Firebase URL, apex, and `www` after deploy;
the workflow definition alone is not proof that its latest run succeeded.

Verify the public release independently:

```bash
curl -fsSI https://korab.space/
curl -fsSI https://www.korab.space/
curl -sS -o /dev/null -w '%{http_code}\n' \
  https://korab.space/projects/hybrid-cloud-backup-static-platform.html
```

On 24 July 2026, the apex returned HTTP 200 with Firebase Hosting headers,
`www` redirected to the apex, the project catalogue and a case-study route
returned HTTP 200, and an unknown route returned HTTP 404.

## Backup

Before applying, complete the
[rendering and JWKS prerequisites](../README.md#deploy): replace every
environment placeholder in the Kubernetes templates, refresh the k3s public
JWKS used by Terraform, and confirm that no rendered Secret value or private key
will enter Git. Do not apply the tracked placeholder templates directly. After
every completed Kustomize apply, refresh the dynamic private-registry Secret:

```bash
kubectl apply -k kubernetes
scripts/bootstrap-registry-auth.sh
job="backup-test-$(date -u +%Y%m%d%H%M%S)"
kubectl create job --from=cronjob/k3s-encrypted-backup "$job" -n hybrid-cloud
kubectl logs -f "job/$job" -n hybrid-cloud
backup_object="$(kubectl logs "job/$job" -n hybrid-cloud | jq -r 'select(.event == "hybrid_backup" and .status == "success") | .object' | tail -n 1)"
test -n "${backup_object}"
gcloud storage objects describe \
  "gs://${BACKUP_BUCKET:?set the private backup bucket}/${backup_object}" \
  --format='value(name,size,createTime)'
```

Job output uses one JSON line with the object name and encrypted size; failures
are also structured. Cloud Logging delivery is best-effort and must be checked
separately. `scripts/verify-backup.sh` checks only the latest `daily/*.age`
object, so it is not a substitute for following the exact `weekly/` or
`monthly/` path emitted by a selected Job. Never enable shell tracing. The
registry refresh is configured for every 30 minutes; backup is scheduled at
02:17 and 14:17 UTC. On 16 July 2026, the backup CronJob existed unsuspended and
reported its latest successful run at 14:17:53 UTC.

## Routine checks

```bash
kubectl get nodes
kubectl get kustomizations.kustomize.toolkit.fluxcd.io,helmreleases.helm.toolkit.fluxcd.io -A
scripts/verify-gke-removal.sh "${GCP_PROJECT_ID:?set the authorized project}"
scripts/cost-safety-check.sh "${GCP_PROJECT_ID}"
kubectl auth can-i get secrets --as=system:serviceaccount:hybrid-cloud:backup -n hybrid-cloud
gcloud logging read "logName=projects/${GCP_PROJECT_ID}/logs/hybrid-k3s-backup" --limit=10
```

The 24 July 2026 read-only check found all six nodes Ready, 12 HelmReleases and
one Kustomization Ready, and all 10 PVCs Bound. It also found a degraded Flux
Git source, an incomplete Longhorn path on the newest node, and mixed off-site
backup results. The earlier `kubectl auth can-i` denial remains dated
16 July. Aggregate GCP inventory on 24 July confirmed no VM, GKE, Cloud Run, or
Cloud SQL runtime, one active WIF pool/provider, zero user-managed
service-account keys in this project, and the configured storage/monitoring
controls. Full billing, notification delivery, and safe deletion of residual
snapshots, registry, and default-network resources still require separate
review.