Korab CenajIncident analysis All articles

DNS · TLS · Tailscale · Firebase · 5 min read

When split DNS made a Firebase site appear offline

The new Firebase release was healthy, public DNS pointed at Firebase, and an external fetch returned the complete page. The same domain still failed TLS from a tailnet-connected workstation. The contradiction was the useful evidence: the application was online, but not every client was taking the same path.

Start by proving which boundary disagrees

The Firebase-provided hostname returned HTTP 200, so the release itself was available. Independent public resolvers returned Firebase's Hosting address for the custom domain, and a remote fetcher loaded the updated portfolio. The workstation's default resolver returned a private tailnet address instead. That reduced the incident from “Firebase is offline” to “this resolver context selects a different endpoint.”

Public resolver     → Firebase Hosting address
Tailnet resolver    → private VPS address
Firebase hostname   → HTTP 200
Custom domain via public path → HTTP 200
Custom domain via private path → TLS handshake failure

Testing only the public resolver would have missed the user-visible failure. Testing only the local browser would have blamed a healthy deployment.

The TLS error was a routing clue

The private address belonged to a VPS running Caddy for an internal Git hostname. Caddy listened on the tailnet interface and had a valid certificate for that exact Git name, but no virtual host or certificate for the portfolio apex. When the browser sent the portfolio name as SNI, the private listener could not complete a matching handshake.

This was not a certificate-renewal problem and adding the portfolio to Caddy would have created a second delivery path unnecessarily. The portfolio was supposed to remain on Firebase; only the private Git hostname belonged on the VPS.

An Unbound zone type widened one record into every hostname

Tailscale routed queries for the domain to Unbound on the private VPS. The configuration comment said that one private hostname should differ from public DNS, but the implementation used a redirect zone and an apex record:

local-zone: "example.com." redirect
local-data: "example.com. 60 IN A <private-vps-address>"

For a redirect zone, the local answer applies across names beneath that zone. The apex, www, and private Git hostname therefore all received the VPS address. The configuration was syntactically valid and Unbound was healthy; its semantics were simply broader than the comment and intended ownership boundary.

Make the zone transparent and the exception exact

The correction allowed ordinary names to continue through the configured recursive path while retaining one explicit private record:

local-zone: "example.com." transparent
local-data: "git.example.com. 60 IN A <private-vps-address>"

The active file was backed up first. unbound-checkconf passed before the service was reloaded. Direct queries to the private resolver then returned Firebase for the apex and www, while the Git hostname remained on the VPS.

Validation must cover both positive paths

Operational lesson

“The site is online” is incomplete without a client and resolver context. DNS answers, SNI selection, certificates, and application responses are separate contracts. Comparing an affected path with a known-good external path exposed the first divergence quickly; inspecting the listener then explained the TLS symptom. The lasting fix restored ownership boundaries instead of adding another proxy route.

This was a self-managed portfolio incident, not an employer or customer system. Private addresses, credentials, and policy contents are intentionally omitted.