# Cluster shutdown

![Cluster startup and shutdown](../architecture/generated/cluster-lifecycle.svg)

## Preconditions

- Obtain an approved maintenance window and identify the operator who can stop
  and restart each host through the local management plane.
- Record the intended replica counts and current workload owners.
- Confirm a recent backup check, healthy Longhorn replicas, and no restore or
  storage rebuild in progress.
- Assess the target nodes against PodDisruptionBudgets, placement, endpoints,
  storage locality, and control-plane quorum before changing state.

## Procedure

1. Capture the starting state without publishing node names or addresses:

   ```bash
   kubectl get nodes
   kubectl get pdb -A
   kubectl get endpointslices -A
   kubectl get --raw='/readyz?verbose'
   ```

2. Check storage and the latest backup result through their product-specific
   commands and [backup verification](backup-verification.md). Stop if replicas
   are rebuilding, degraded, or tied only to a target node.
3. Scale down only the applications named in the approved plan. Record every
   previous replica count so startup can restore it exactly.
4. Cordon and drain one worker at a time:

   ```bash
   kubectl cordon "${NODE:?set one approved worker}"
   kubectl drain "${NODE}" --ignore-daemonsets --timeout=15m
   ```

   Do not add `--force`, delete a PodDisruptionBudget, or discard local data to
   make the command succeed.
5. Confirm replacement pods and Service endpoints are Ready before stopping the
   drained worker through the approved host mechanism.
6. Stop the remaining workers. Before server shutdown, finish every
   Kubernetes/API-driven action and confirm that all three etcd members are
   healthy. Stop one control-plane/etcd server, leaving two members, then check
   API and etcd health once more. Abort and restart it if that check fails.
7. Record that stopping either of the two remaining servers is the intentional
   quorum-loss boundary: a three-member etcd cluster requires two members. Stop
   the second and final servers through the host-management plane in the
   approved order. Do not expect Kubernetes API checks after the second server
   stops; verify their power state out of band and record anything deliberately
   left running outside the cluster.

## Expected result

Approved applications are quiesced, workers drain without bypassing disruption
or storage safeguards, and the server sequence preserves quorum while API work
remains. Quorum is lost only at the documented full-shutdown boundary, after
which host state is checked through the management plane.

## Failure and rollback

- Abort on a blocked PDB, unschedulable replacement, lost endpoint, degraded
  storage, failed backup check, or unhealthy etcd/API response.
- Before a host is powered off, roll back by restarting scaled workloads and
  running `kubectl uncordon "${NODE}"`.
- After a host is powered off, restart that host and follow the startup runbook
  until it is Ready before attempting another shutdown step.
- After the second server stops, Kubernetes API unavailability is expected. If
  the sequence must be reversed, use the management plane to start enough
  servers to restore etcd majority, then follow the startup runbook.
- Escalate any quorum, storage-attachment, or data-consistency uncertainty. A
  controlled shutdown does not replace disaster-recovery planning.