Open engineering project · Assessed 13 July 2026

Kubernetes Disaster Recovery Validator

A Go validator that asks whether backup artifacts form a viable recovery path for an application, rather than merely proving that a backup job ran.

Project type: Infrastructure Tool
Domain: Data protection and recovery
Language: Go
Status: Validated

All projects

Executive summary

Operational problem

Replication, backup completion, and GitOps availability each cover only part of recovery. A stateful workload can still fail to return if storage, identity, DNS, bootstrap, or the backup target depends on the failed cluster.

Implemented solution

The validator correlates applications with PVCs/PVs, Longhorn, Velero, backup-target locality, Flux ownership, declared external dependencies, and recovery policy. A shared model classifies protection, simulates failure scenarios, detects bootstrap cycles, estimates bounded RPO/RTO ranges, and proposes an order for restoring dependencies instead of treating every backup artifact as equivalent.

Outcome

Reviewers can distinguish in-cluster copies, off-cluster backups, archive inspection, and an exercised application restore. Missing dependencies and circular bootstrap paths become explicit planning work instead of being hidden behind a successful backup status.

Technical and operational context

The validator separates availability from recoverability, traces dependencies across failure domains, preserves uncertainty in RPO/RTO ranges, and makes the level of restore proof explicit.

Architecture

Typed and dynamic discovery builds application and recovery-dependency models. Protection checks, RPO/RTO ranges, scenario simulation, cycle detection, and ordered plans share that model so findings point back to dependencies.

Application
Workload
PVC / PV
Longhorn
Velero backup
Backup target
Recovery plan

Architecture visual

Disaster-recovery dependency graph across cluster, GitOps, storage, DNS, certificates, and restore tests — Generic recovery dependencies and analyzer risk patterns; not a statement of current defects.
Read the diagram notes

Implementation

Connects stateful applications to PVC/PV, StorageClass, Longhorn, Velero, Flux, and target-locality evidence
Separates unprotected, replicated-only, backed-up, restore-inspected, and recovery-verified states
Simulates node, storage, namespace, control-plane, backup-target, and full-cluster loss
Produces a dependency-aware order for bootstrap, storage, data services, applications, ingress, and TLS

Key decisions and trade-offs

Use protection tiers

Replication, snapshot presence, backup, archive inspection, and an exercised application recovery are different states. A tiered model prevents a local copy from being described as disaster recovery. It requires more metadata and operator input than a boolean, but exposes exactly which step is missing.

Keep restore execution out of the live client

The assessment client can use restricted read access and still build a dependency-aware plan. Resource application and data restoration remain operator-approved runbook actions with different permissions and rollback needs. The tradeoff is that the tool can assess readiness but cannot certify a completed recovery.

Report RPO/RTO as ranges

Data size, dependency depth, operator steps, and missing drill measurements widen the estimate. A range communicates uncertainty better than an unsupported precise target. Until a timed drill exists, the result guides planning and comparison rather than promising a service-level objective.

Treat cycles as recovery findings

GitOps, identity, DNS, storage, or backup access can depend on the cluster they are expected to rebuild. The graph surfaces those bootstrap cycles and proposes ordering around them. Some external relationships still need manual input because Kubernetes cannot observe their control planes.

Security, reliability, and observability

Restore authority stays separate

The assessment client uses read access and produces a plan; resource application and data restoration remain operator-approved procedures with different credentials and rollback requirements.

Failure-domain reasoning

Protection is evaluated against node, storage, namespace, control-plane, backup-target, and full-cluster loss so an in-cluster copy is not mislabeled as off-cluster recovery.

Evidence tiers

Replication, snapshot, backup, archive inspection, and application recovery are distinct states. RPO and RTO remain ranges until timed drills provide stronger measurements.

Infrastructure concepts demonstrated

StatefulSets
PVCs and PVs
StorageClasses
Longhorn
Velero
Flux CRDs
Failure domains
Recovery ordering

Published evidence

Sanitized protection-matrix row

Sanitized live output

Application: namespace-3af2e4/application-cdfa20
Data: 1 PVC · replicated: true · snapshot: true
Off-cluster backup: false · restore tested: false
Protection: BACKED_UP_IN_CLUSTER
Estimated RTO: 55m–3h20m

Validation

Checks shown here are project-specific; no portfolio-wide claim is applied automatically.

Read-only application-protection matrix dated 13 July 2026
Source review of scenario simulations and dependency-cycle tests
Fake-client restore guardrail tests
No application restore performed by the assessment client

Evidence and links

Links are limited to public-safe source material and sanitized evidence. Credentials, state, plans, identifiers, private topology, and backup contents are intentionally excluded.

Lessons learned

Backup presence is one dependency in recovery, not evidence that the application can return.
Identity, DNS, storage, GitOps, and backup access can form bootstrap cycles that must be broken before an incident.
Precise RPO or RTO claims require observed, application-level drills; until then, ranges should widen with every unknown.

Limitations

RTO estimates use evidence-based ranges, not guaranteed restore times.
Backup metadata does not prove application-consistent recovery.
External DNS, identity, object-storage durability, and operator actions require evidence beyond Kubernetes objects.

Next improvement

Ingest signed restore-drill results so successful exercises can advance an application from backed up to restore-tested automatically.