Korab CenajEngineering case studyAll case studies

Open engineering project · Assessed 12 July 2026

Kubernetes Maintenance Orchestrator

A Go CLI that assesses whether a Kubernetes node can leave service without violating placement, availability, storage, quorum, or recovery constraints—before an operator decides whether to run `kubectl drain`.

Project type
Infrastructure Tool
Domain
Reliability and operations
Language
Go
Status
Validated

Executive summary

Operational problem

Eviction success proves only that the API accepted disruption requests for eligible pods. It does not establish replacement capacity, PDB allowance, endpoint continuity, stateful quorum, volume locality, or a viable path for the node and its workloads to return after maintenance.

Implemented solution

Typed and dynamic collectors connect workloads, PDBs, scheduling rules, Services, stateful storage, control-plane roles, backups, and GitOps ownership in one dependency graph. The decision engine then produces blockers, warnings, a plan, and recovery gates. Any mutating action remains behind a separate guarded execution path with a different permission boundary.

Outcome

Operators receive an explicit SAFE, SAFE_WITH_WARNINGS, BLOCKED, UNSAFE, or UNKNOWN decision with reasons and recovery gates. The project makes “drain succeeded” a small part of a larger maintenance decision instead of the conclusion.

Technical and operational context

The Go decision model combines disruption policy, scheduler constraints, storage topology, quorum, and recovery state into an explainable preflight result while separating assessment permissions from action.

Architecture

client-go and optional CRD discovery feed workload, placement, storage, endpoint, quorum, and recovery models. Analyzers produce a stable decision and plan; execution remains isolated behind approval, UID checks, and bounded health gates.

  1. Kubernetes API
  2. Typed + dynamic collectors
  3. Dependency graph
  4. Safety analyzers
  5. Decision model
  6. Guarded plan / recovery report

Implementation

Typed and dynamic collectors connect workloads, PDBs, scheduling rules, Services, stateful storage, control-plane roles, backups, and GitOps ownership in one dependency graph. The decision engine then produces blockers, warnings, a plan, and recovery gates. Any mutating action remains behind a separate guarded execution path with a different permission boundary.

Key decisions and trade-offs

Assessment before execution

Inspect and plan use a read-only path. Mutation stays behind separate approval, object-UID checks, bounded health gates, and recovery behavior. This adds an operator step and a second permission boundary, but prevents a diagnostic command from cordoning or evicting during routine assessment.

Model capacity and dependencies together

Spare CPU cannot compensate for a blocked PodDisruptionBudget, a pinned volume, lost endpoint coverage, or reduced etcd quorum. The decision model therefore combines placement, availability, storage, and control-plane consequences instead of treating low utilization as permission to proceed.

Unknown is safer than optimistic

Missing permissions, optional APIs, or storage facts lower confidence and can produce UNKNOWN. That outcome is less convenient than a green decision, but it avoids approving maintenance when the tool could not inspect a dependency that determines whether the workload can return.

Security, reliability, and observability

Read-only preflight

Inspect and plan use a restricted path; cordon, eviction, and other mutations require a separately guarded executor and are not represented as run in the published assessment.

Availability before utilization

PDB allowance, endpoint continuity, storage locality, replacement placement, and etcd quorum are assessed together before a node is considered maintainable.

Recovery gates

Plans include object identity checks, bounded health conditions, abort criteria, and return-to-service observations instead of treating drain completion as the end state.

Infrastructure concepts demonstrated

Published evidence

Sanitized node-maintenance decision

Sanitized live output
Decision: BLOCKED
Confidence: HIGH
Automatic execution allowed: false
Blocker: PodDisruptionBudget allows 0 disruptions for a target workload.
Recommendation: restore healthy replicas or revise workload capacity; do not delete the PDB.

Validation

Checks shown here are project-specific; no portfolio-wide claim is applied automatically.

Evidence and links

Links are limited to public-safe source material and sanitized evidence. Credentials, state, plans, identifiers, private topology, and backup contents are intentionally excluded.

Lessons learned

Limitations

  • Scheduling analysis approximates relevant constraints but does not reproduce every scheduler plugin.
  • External host power control and out-of-cluster dependencies require separate verification.
  • The published assessment covered inspect, plan, and recovery reporting—not mutation.

Next improvement

Add controller-runtime reconciliation as a separate operator with approval policies and auditable maintenance-window status conditions.