# Cluster startup

![Cluster startup and shutdown](../architecture/generated/cluster-lifecycle.svg)

## Preconditions

- Use the shutdown record for server order, worker list, and previous workload
  replica counts.
- Confirm access to the local host-management plane, Kubernetes credentials,
  and the recovery contact if the datastore does not become healthy.
- Keep private hostnames and addresses out of shared notes.

## Procedure

1. Start the first approved control-plane/etcd server through the local
   mechanism. Wait for its host and k3s service before starting the next server.
2. Start the remaining servers one at a time. Stop the sequence on a host wake
   failure or k3s timeout.
3. Check the Kubernetes API and datastore before starting workers:

   ```bash
   kubectl get --raw='/readyz?verbose'
   kubectl get nodes
   ```

   Expect API readiness and etcd checks to pass, and the planned server nodes to
   report Ready with the correct roles.
4. Start workers one at a time. Wait for each node, CNI agent, CSI node plugin,
   and expected DaemonSets before continuing.
5. Check storage health and attachment state before restoring application
   replicas. Do not schedule a stateful workload while its volume is degraded or
   rebuilding without an approved exception.
6. Restore only the replica counts recorded by the shutdown plan. Remove any
   maintenance cordons deliberately:

   ```bash
   kubectl uncordon "${NODE:?set one recovered node}"
   ```

7. Verify Flux reconciliation, ingress, Services and EndpointSlices, DNS/TLS,
   monitoring, and the next backup schedule. Application owners confirm the
   service-level checks for their workloads.

## Expected result

All intended nodes become Ready, API and datastore checks pass, CNI and storage
components settle before applications scale, and public or internal service
checks match the recorded pre-shutdown state.

## Failure and rollback

- Stop on API timeout, role/address mismatch, etcd failure, degraded storage,
  unavailable CNI, unscheduled workload, missing endpoint, DNS failure, or TLS
  failure.
- Do not start more nodes or restore more replicas while the current stage is
  unresolved; preserve events and component logs first.
- If a newly started worker destabilizes placement or storage, cordon it and
  return workloads to the last healthy topology.
- Escalate datastore recovery rather than initializing a replacement cluster.
  Operator automation supplies repeatable checks; it does not authorize data
  loss or a new control plane.