Self-managed engineering project · Private foundation complete · Production migration pending
Secure Always-On Git Platform
Designed and validated an always-on private Git foundation on a VPS with identity-based access, automated TLS, fail-closed activation, tested recovery, and verified reboot persistence.
Infrastructure automation depended on a Git service hosted by a Kubernetes homelab that powers down nightly. The replacement foundation had to remain available independently without exposing administration, Git HTTPS, or private DNS to the public Internet.
Implemented solution
An Ubuntu VPS runs Gitea and PostgreSQL through root-managed Podman Quadlets. Tailscale identity policy controls access, Caddy terminates private HTTPS, Unbound provides split DNS, DNS-01 manages TLS, nftables restricts public access, and Ansible owns guarded activation, validation, rollback, and reboot-safe systemd ordering.
Outcome
The private foundation passed its final reboot without manual intervention: Git HTTPS, split DNS, administration, application health, database health, certificate state, policy evidence, and firewall controls persisted. Production repositories and databases remain on the source platform pending a separately controlled migration.
Technical and operational context
The project connects Linux administration, identity-based networking, container runtime design, TLS and DNS lifecycle management, fail-closed Ansible automation, service dependency diagnosis, rollback engineering, and evidence-led validation without overstating the unfinished production migration.
Constraints
Administrative and private-service access had to remain unavailable from the public Internet.
The existing public proxy, public DNS, Kubernetes environment, and home Git service could not be changed during the private stage.
Policy, certificate, identity, health, rollback, and reboot gates had to fail closed.
Production repositories, packages, and database content were explicitly outside this completed foundation scope.
Architecture
Identity-verified clients reach only the VPS private interface. Caddy forwards private HTTPS to loopback-only Gitea, which uses PostgreSQL on a private container network with no host database listener. Unbound answers the private Git name on the same identity-controlled interface, while public web delivery remains separate and unchanged.
Identity-verified client
Least-privilege tailnet policy
Private VPS interface
Split DNS and private HTTPS
Caddy reverse proxy
Gitea Podman Quadlet
PostgreSQL private network
Implementation
An Ubuntu VPS runs Gitea and PostgreSQL through root-managed Podman Quadlets. Tailscale identity policy controls access, Caddy terminates private HTTPS, Unbound provides split DNS, DNS-01 manages TLS, nftables restricts public access, and Ansible owns guarded activation, validation, rollback, and reboot-safe systemd ordering.
Separated Git availability from the homelab power schedule with an always-on VPS foundation
Implemented named non-root administration, private identity policy, and public-access denial
Ran Gitea and PostgreSQL through root-managed Podman Quadlets with no host database listener
Automated DNS-01 TLS, split DNS, private reverse proxying, service health, and policy gates
Diagnosed and corrected a reboot-time service race through an Ansible-managed readiness check and systemd ordering
Proved rollback controls, positive and negative connectivity, reboot persistence, and final idempotence
Key decisions and trade-offs
Private identity instead of public exposure
Tailscale identity and explicit least-privilege grants protect administration, HTTPS, and split DNS. This reduces exposed surface area while making tailnet policy and provider-console recovery operational dependencies.
Host Quadlets instead of the source Kubernetes runtime
Root-managed Podman Quadlets make the target Git service independent of the nightly cluster lifecycle. The trade-off is a deliberately smaller single-host runtime rather than a claim of high availability.
DNS-01 for a private listener
Exact-name DNS-01 issuance allows a publicly trusted certificate without opening an HTTP challenge path or changing the existing public endpoint. Credential scope and renewal validation remain explicit controls.
Evidence-gated activation
Ansible and guarded scripts require policy, certificate, listener, health, rollback, and negative-test evidence before claiming success. The additional gates trade speed for reviewable failure boundaries.
Security, reliability, and observability
Identity and SSH hardening
A named non-root administrator uses ordinary OpenSSH through Tailscale. Remote root, password, keyboard-interactive, public SSH, and Tailscale SSH access remain disabled or refused.
Private listener boundaries
Caddy and Unbound bind only to the private interface, Gitea publishes loopback HTTP, and PostgreSQL has no host listener. Positive private tests are paired with public negative tests.
Validated TLS and policy lifecycle
Restricted DNS-01 issuance, certificate validation and renewal testing, least-privilege policy tests, and idempotent policy attestation keep external trust dependencies reviewable.
Rollback and recovery
Access-control changes use timed automatic rollback, configuration changes remain Ansible-managed, and an MFA-protected provider recovery console supplies an independent administration path.
Infrastructure concepts demonstrated
Linux service administration
Ansible roles and handlers
Podman Quadlets
systemd dependencies
Private DNS
DNS-01 ACME
TLS lifecycle
Least-privilege networking
Rollback and recovery testing
Published evidence
Sanitized completion evidence
Sanitized live output
private foundation: complete
reboot persistence: passed
project tests: 51 passed
Ansible syntax: 7 playbooks passed
final Ansible run: changed=0, failed=0
production data migration: pending
Validation
Checks shown here are project-specific; no portfolio-wide claim is applied automatically.
51 project tests passed
All seven Ansible playbooks passed syntax validation
Final Ansible reconciliation reported changed=0 and failed=0
Final reboot-persistence checks passed without manual Caddy intervention
Authorized private HTTPS, split DNS over TCP and UDP, and administrative access passed
Public SSH, private HTTPS staging, and public DNS-port negative tests passed
The existing public proxy, public DNS, Kubernetes environment, and home Git service remained unchanged
Challenges and troubleshooting
Configuration-valid service failed after reboot
Caddy passed configuration validation but started before its private Tailscale address existed. Service evidence isolated the startup race from proxy and certificate concerns.
Runtime dependency made explicit
A strict readiness helper and systemd drop-in now wait for the exact private interface state, order Caddy after Tailscale, and retry on failure. The existing Ansible role owns the full change and validation chain.
Current result
The completed private foundation is reboot-persistent and deny-by-default. It is ready for a separately approved backup, restore rehearsal, and controlled migration; no production repository or database move is claimed here.
Evidence and links
Links are limited to public-safe source material and sanitized evidence. Credentials, state, plans, identifiers, private topology, and backup contents are intentionally excluded.
Lessons learned
Configuration syntax can pass while runtime ordering still fails; reboot testing is a separate evidence gate.
Positive private connectivity is incomplete without public negative tests and an independent recovery path.
Idempotence and rollback are operational properties that need live validation, not documentation alone.
Remaining work
Deploy and validate Kubernetes egress integration
Integrate Flux, runner, and BuildKit access through the approved private path
Create a consistent production source backup
Rehearse an isolated restore and application validation
Perform a controlled production cutover with rollback evidence
Retire the home service only after explicit approval and post-cutover checks
Limitations
The case study covers the completed private VPS foundation, not a completed production Gitea migration.