Korab CenajEngineering case study Project catalogue

Self-managed engineering project · Private foundation complete · Production migration pending

Secure Always-On Git Platform

Designed and validated an always-on private Git foundation on a VPS with identity-based access, automated TLS, fail-closed activation, tested recovery, and verified reboot persistence.

Project type
Infrastructure Migration Case Study
Domain
Reliability and operations
Language
Ansible / Bash
Status
Validated

Executive summary

Operational problem

Infrastructure automation depended on a Git service hosted by a Kubernetes homelab that powers down nightly. The replacement foundation had to remain available independently without exposing administration, Git HTTPS, or private DNS to the public Internet.

Implemented solution

An Ubuntu VPS runs Gitea and PostgreSQL through root-managed Podman Quadlets. Tailscale identity policy controls access, Caddy terminates private HTTPS, Unbound provides split DNS, DNS-01 manages TLS, nftables restricts public access, and Ansible owns guarded activation, validation, rollback, and reboot-safe systemd ordering.

Outcome

The private foundation passed its final reboot without manual intervention: Git HTTPS, split DNS, administration, application health, database health, certificate state, policy evidence, and firewall controls persisted. Production repositories and databases remain on the source platform pending a separately controlled migration.

Technical and operational context

The project connects Linux administration, identity-based networking, container runtime design, TLS and DNS lifecycle management, fail-closed Ansible automation, service dependency diagnosis, rollback engineering, and evidence-led validation without overstating the unfinished production migration.

Constraints

Architecture

Identity-verified clients reach only the VPS private interface. Caddy forwards private HTTPS to loopback-only Gitea, which uses PostgreSQL on a private container network with no host database listener. Unbound answers the private Git name on the same identity-controlled interface, while public web delivery remains separate and unchanged.

  1. Identity-verified client
  2. Least-privilege tailnet policy
  3. Private VPS interface
  4. Split DNS and private HTTPS
  5. Caddy reverse proxy
  6. Gitea Podman Quadlet
  7. PostgreSQL private network

Implementation

An Ubuntu VPS runs Gitea and PostgreSQL through root-managed Podman Quadlets. Tailscale identity policy controls access, Caddy terminates private HTTPS, Unbound provides split DNS, DNS-01 manages TLS, nftables restricts public access, and Ansible owns guarded activation, validation, rollback, and reboot-safe systemd ordering.

Key decisions and trade-offs

Private identity instead of public exposure

Tailscale identity and explicit least-privilege grants protect administration, HTTPS, and split DNS. This reduces exposed surface area while making tailnet policy and provider-console recovery operational dependencies.

Host Quadlets instead of the source Kubernetes runtime

Root-managed Podman Quadlets make the target Git service independent of the nightly cluster lifecycle. The trade-off is a deliberately smaller single-host runtime rather than a claim of high availability.

DNS-01 for a private listener

Exact-name DNS-01 issuance allows a publicly trusted certificate without opening an HTTP challenge path or changing the existing public endpoint. Credential scope and renewal validation remain explicit controls.

Evidence-gated activation

Ansible and guarded scripts require policy, certificate, listener, health, rollback, and negative-test evidence before claiming success. The additional gates trade speed for reviewable failure boundaries.

Security, reliability, and observability

Identity and SSH hardening

A named non-root administrator uses ordinary OpenSSH through Tailscale. Remote root, password, keyboard-interactive, public SSH, and Tailscale SSH access remain disabled or refused.

Private listener boundaries

Caddy and Unbound bind only to the private interface, Gitea publishes loopback HTTP, and PostgreSQL has no host listener. Positive private tests are paired with public negative tests.

Validated TLS and policy lifecycle

Restricted DNS-01 issuance, certificate validation and renewal testing, least-privilege policy tests, and idempotent policy attestation keep external trust dependencies reviewable.

Rollback and recovery

Access-control changes use timed automatic rollback, configuration changes remain Ansible-managed, and an MFA-protected provider recovery console supplies an independent administration path.

Infrastructure concepts demonstrated

Published evidence

Sanitized completion evidence

Sanitized live output
private foundation: complete
reboot persistence: passed
project tests: 51 passed
Ansible syntax: 7 playbooks passed
final Ansible run: changed=0, failed=0
production data migration: pending

Validation

Checks shown here are project-specific; no portfolio-wide claim is applied automatically.

Challenges and troubleshooting

Configuration-valid service failed after reboot

Caddy passed configuration validation but started before its private Tailscale address existed. Service evidence isolated the startup race from proxy and certificate concerns.

Runtime dependency made explicit

A strict readiness helper and systemd drop-in now wait for the exact private interface state, order Caddy after Tailscale, and retry on failure. The existing Ansible role owns the full change and validation chain.

Current result

The completed private foundation is reboot-persistent and deny-by-default. It is ready for a separately approved backup, restore rehearsal, and controlled migration; no production repository or database move is claimed here.

Evidence and links

Links are limited to public-safe source material and sanitized evidence. Credentials, state, plans, identifiers, private topology, and backup contents are intentionally excluded.

Lessons learned

Remaining work

Limitations

  • The case study covers the completed private VPS foundation, not a completed production Gitea migration.
  • Kubernetes egress, Flux, runner, BuildKit, and CoreDNS integration remain future work.
  • The single VPS foundation is not presented as high availability or complete disaster recovery.
  • Private topology, endpoints, credentials, policy contents, and recovery procedures are intentionally omitted.

Next improvement

Create a consistent source backup and prove it through an isolated restore before authorizing any production cutover.