One Traefik per server, with routes written to a file

Every server that dockxo manages runs one Traefik container. It terminates TLS, owns ports 80 and 443, and routes each domain to the right app container. That part is ordinary. The unusual part is how Traefik learns about the routes.

The Docker provider and why we stopped using it

Traefik’s Docker provider watches the Docker socket and builds routes from container labels. It is the standard setup and it worked for the first months. Two things made us leave it.

First, labels are set when a container is created. A rolling deploy creates the new container, waits for it to pass a port check, and then wants to switch the domain over. With labels, the new container is routable the moment it starts, which is before it is healthy. Users saw a few seconds of 502s on every deploy.

Second, the panel wanted routes that are not containers: a load balancer that spreads one domain across the same app on three servers, a cache proxy in front of a static site, a maintenance page. You can express some of that in labels on a dummy container. It gets ugly fast.

What we do instead

The agent subscribes to Docker events on the server. When a container starts, stops or passes its health check, the agent rebuilds a single YAML file in Traefik’s dynamic configuration directory. Traefik’s file provider watches that directory and reloads without restarting.

The agent decides what goes in the file. A container that has started but not passed the port check is not in it. When the check passes, the file is rewritten with the new container and without the old one, in one write, so there is no moment where both or neither are routable. The old container is stopped after a grace period for in-flight requests.

Load balancers, cache proxies and maintenance pages are just more entries in the same file. The panel sends the agent a description of what the server should route; the agent merges it with what Docker says is running and writes the result.

The priority bug

One reason to write this post: for a while a deleted load balancer could shadow a real domain. The file kept a stale router for the balancer with a higher priority than the app’s router, and because both matched the same host, Traefik picked the stale one and returned a 404. It only happened if you created a balancer and then deleted it without deleting its domain.

The fix was boring: routers are now named after the thing that owns them, the file is regenerated from scratch on every change instead of edited in place, and priorities are computed from the length of the host rule the way Traefik does by default, so nothing can outrank a more specific match.

Certificates

Let’s Encrypt is handled by Traefik’s ACME resolver, one per server, with the certificates in a volume so they survive a Traefik upgrade. For wildcard domains the panel does the DNS challenge through Cloudflare and hands the certificate to the agent. Free subdomains under the panel’s own domain use a single wildcard certificate that the panel renews and pushes to every server.