Why kubectl drain is not enough

What the command actually establishes

kubectl drain marks the node unschedulable and asks the eviction API to remove eligible pods. The eviction API enforces current PodDisruptionBudget allowance; workload controllers and the scheduler, not drain itself, create and place replacements. Completion therefore does not establish that replacements became healthy, volumes reattached, or Services kept usable backends. DaemonSet, static, unmanaged, and local-data pods also need explicit interpretation. Flags that bypass a blocker change command behavior; they do not create spare capacity or redundancy. Drain also does not remove a control-plane server from etcd membership.

The misconception persists because command success is immediate and legible, while the relevant conditions live across controllers. Capacity, scheduling rules, storage topology, endpoint distribution, existing failures, and the node’s role must be correlated before the disruptive step begins. The decision must describe the intended outage and its duration, not merely whether a pod can be deleted now.

A maintenance decision path

1. Define the target role and intended outage.
2. Measure workload health and PDB allowance.
3. Test placement against real scheduler constraints.
4. Check volume attachment and replica locality.
5. Count surviving Service endpoints.
6. Preserve etcd majority throughout the outage.
7. Set abort, return, and post-change gates.

An illustrative, sanitized blocker

Example: illustrative and sanitized
Decision: BLOCKED
Target: control-plane server and etcd member
PDB allowance: 0 disruptions
Replacement placement: feasible
Storage attachment: not established
etcd: 3 members; 1 already unavailable
Planned outage: 1 of 3 members would remain

Aggregate CPU looks sufficient, yet two independent constraints stop the plan. The PDB allows no voluntary disruption, and taking a second etcd server offline would leave one available member where a three-member cluster requires two for quorum. The cluster may be serving with two members while one server is unavailable, but it has already consumed its single-failure tolerance. Drain would neither repair that server nor change membership. The operator should restore cluster health first and then reassess; deleting the PDB or casually removing an etcd member merely to make the command pass would change the system’s protection model. Membership changes require their own quorum-aware procedure.

Questions to answer before proceeding

• Which pods are unmanaged, local, static, or protected by a PDB, and what is their health now?
• Can every replacement satisfy taints, selectors, affinity, topology, host-port, pod-slot, and resource constraints?
• Can each persistent volume attach on an eligible node, with enough healthy replicas outside the maintenance failure domain?
• Will every required Service retain a ready endpoint away from the target, and has the replacement actually joined it?
• If this is a server node, what etcd members are healthy now, and does majority survive the entire planned outage?
• What observations stop the change, prove recovery, and confirm the node and displaced workloads returned as intended?