Troubleshooting
Common issues when setting up and running sfp server.
Server initialization
| Symptom | Cause | Fix |
|---|---|---|
Docker registry credentials not found. Please set DOCKER_REGISTRY and DOCKER_REGISTRY_TOKEN. | Either DOCKER_REGISTRY or DOCKER_REGISTRY_TOKEN is missing. The CLI validates registry credentials on every init, dev and prod alike, before the tenant .env is written. The server.json sample in Step 1 must include both. | Add DOCKER_REGISTRY (e.g. source.flxbl.io) and DOCKER_REGISTRY_TOKEN to secrets.*, or export both as environment variables, then re-run init. |
After the failure above, a confusing ENOENT for a .env file that was never written | On a fresh install the failure aborts init before .env is created, and the cleanup path then reports the missing file. This only appears on a fresh install (an existing tenant already has a .env). | Not an additional defect — solve the registry-credentials error and re-run init. |
Tenant '<name>' already exists | The tenant directory already exists. | Re-run with --force (🔥 wipes tenant DB + volumes + dir) or use a different tenant name. |
General
| Symptom | Cause | Fix |
|---|---|---|
denied: denied from the registry login | Stale token, wrong registry | docker logout <registry> && echo "<pat>" | docker login <registry> -u <user> --password-stdin |
| Domain returns NXDOMAIN | DNS not propagated or A record missing | nslookup <domain>; verify A record points at server; allow up to 48h propagation |
| Need raw container logs | Compose project name matches tenant | cd /opt/sfp-server/tenants/<tenant> && docker compose -p <tenant> logs -f |
TLS / Certificates
| Symptom | Cause | Fix |
|---|---|---|
Caddy logs cert file not found | TLS cert not placed or not decoded | Re-run init with ORIGIN_CERT/ORIGIN_KEY secrets, or manually place origin.pem + origin-key.pem in {tenantDir}/certs/ |
| Let's Encrypt fails to issue cert | DNS not publicly resolvable or port 80 blocked | Verify dig <domain> returns the server IP from the public internet; open port 80 inbound for the ACME challenge. In cloudflare mode keep the admin subdomain DNS-only. |
| Certificate format error | Cert not PEM-encoded or not base64 | Certs must be PEM; .crt/.key are typically already PEM — rename and base64-encode: base64 -w 0 origin.pem. |
Services
| Symptom | Cause | Fix |
|---|---|---|
external volume "…" not found | Volumes not created yet | Use sfp server start (not raw docker compose up); it creates external volumes automatically |
Server healthy but curl returns 502 | App server not ready yet | Wait 30–60s after start; check sfp server logs --service worker; Caddy shows a maintenance page until the backend is up |
| Supabase containers not starting | Missing compose profile | sfp server start activates the supabase profile automatically for self-hosted mode; raw docker compose up does not |
VaultBootstrapService fetch failed | Supabase auth not ready at server boot | docker compose -p <tenant> restart server; auth containers need ~15s to initialize |
Update reports Migration container exited with code N | The supabase-migrations one-shot container failed applying migrations | Read docker compose -p <tenant> logs supabase-migrations; update already waits up to 2 minutes and tails 20 lines on failure |
SSH
| Symptom | Cause | Fix |
|---|---|---|
sfp server init: SSH connection error: Timed out while waiting for handshake then Docker is not installed or not available | Target port 22 unreachable (firewall, security group, bastion-only network). The "Docker" line is misleading — read the SSH line above it. | Open a local tunnel via the bastion: ssh -fNT -L 2222:<target-host>:22 <user>@<bastion>, then sfp server init --ssh-connection <user>@127.0.0.1:2222 --identity-file ~/.ssh/<key>; tear down with pkill -f "ssh -fNT -L 2222". |
sfp server init: getaddrinfo EAI_AGAIN <host-alias> | --ssh-connection was passed an ~/.ssh/config Host alias. sfp uses Node ssh2, which does not read ~/.ssh/config — Host, ProxyJump, IdentityFile, User are all ignored. | Pass a literal user@host[:port] plus --identity-file <path>. For ProxyJump-style routing, use the local tunnel recipe above. |
sfp server init: Failed to read private key: ENOENT <path> | --identity-file path missing or unreadable by the user running sfp. | Pass an absolute path: --identity-file /home/<user>/.ssh/<key>. ~/ expands; $HOME does not. |
sfp server init: No authentication method provided | Neither --identity-file nor --passphrase passed. sfp does not fall back to ssh-agent or ~/.ssh/id_rsa. | Always pass --identity-file <path> with --ssh-connection. No --use-agent flag exists. |
Proxy / Client IP
| Symptom | Cause | Fix |
|---|---|---|
sfp-server logs show wrong client IP (e.g. 127.0.0.1 or a proxy IP instead of the real client) | trustProxyHops is too low — sfp-server's Express layer reads X-Forwarded-For only up to the trusted hop count | Set TRUST_PROXY_HOPS=3 in .env for a Cloudflare → customer LB → Caddy → app chain (default 2 covers Cloudflare → Caddy → app). Range 0–10. Restart/recreate the stack. |
| Rate limiting applies to the wrong address | Same as above — Express-layer IP attribution and rate limits derive from the trusted X-Forwarded-For hop | Increase TRUST_PROXY_HOPS to match the actual proxy-chain depth. All inbound paths must have the same hop count. Note: admin-surface allowlists (ALLOWED_IPS) use remote_ip independently of this setting — see Security best practices. |
| Monitoring allowlists fail or audit logs attribute actions to a proxy IP | trustProxyHops too low, inconsistent hop counts, or proxies not forwarding headers | Verify every proxy preserves X-Forwarded-For/X-Forwarded-Proto/X-Forwarded-Host; then set TRUST_PROXY_HOPS=3 for Cloudflare → customer LB → Caddy → app and restart/recreate the stack. All inbound paths must have the same hop count. |
Webhooks
| Symptom | Cause | Fix |
|---|---|---|
| GitHub webhook delivery times out or reports connection refused | The webhook URL points to a private, VPN-only, or bastion-only host. GitHub.com cannot use SSH ProxyJump, an operator VPN, or a bastion path. | Expose a public HTTPS ingress that forwards privately to the server: public ALB / customer load balancer, managed reverse proxy / tunnel, or API Gateway with VPC Link / private integration. Verify the exact payload URL with GitHub Recent deliveries / Redeliver. See Webhook ingress for private servers. |
| Firewall rules would require chasing many GitHub source IP ranges | GitHub.com webhook delivery IPs change, and source IP is not the authenticity control | Do not make GitHub IP allowlists the primary design. Expose only the public HTTPS ingress, restrict the private backend to the ALB/proxy/tunnel/VPC Link path, and preserve signature validation. |
| Public ingress exposes the whole sfp UI/API when only webhooks should be public | Load balancer, tunnel, or API Gateway forwards all paths to Caddy | Add a path/method rule for only POST /sfp/api/repository/webhook; return 403/404 for other public paths unless the full UI/API is intentionally public. Keep the backend private. |
| Webhook reaches the load balancer but sfp-server sees HTTP or the wrong host | Load balancer / API Gateway is not preserving forwarded headers | Forward or set X-Forwarded-Proto: https, X-Forwarded-Host, X-Forwarded-For; then set TRUST_PROXY_HOPS to the trusted-hop count. See Proxy / Client IP. |
Webhook through API Gateway returns 503 Service Unavailable | VPC Link still provisioning, cannot reach the load balancer listener, or targets an internet-facing load balancer | Wait until the VPC Link is AVAILABLE; use HTTP API private integration to an internal ALB/NLB listener; allow the VPC Link SG to reach the internal load balancer; verify the internal target group is healthy. |
| Webhook through API Gateway returns 404 or never reaches the expected route | Private integration forwards the stage/base path to the backend | Use a custom domain/base-path mapping or request-path override so Caddy receives the same path as the payload URL. |
| Signature validation fails behind a proxy/ALB/API Gateway | Ingress changed the request body, dropped signature headers, or re-serialized the payload | Forward the raw body unchanged and preserve signature/event headers (GitHub: X-Hub-Signature-256, X-GitHub-Event, X-GitHub-Delivery). Remove mapping templates or body rewrites from webhook routes. |
| GitHub marks delivery failed even though it reached the server | The ingress or backend waited for workflow execution instead of acknowledging receipt | The webhook must return 2xx quickly. sfp-server dispatches to Hatchet and returns immediately; check ingress timeouts if the request waits for workflow completion. |
SELinux (Podman rootless on RHEL / Fedora)
| Symptom | Cause | Fix |
|---|---|---|
cannot apply additional memory protection after relocation: Permission denied or RELRO protection failed | Podman graphroot has SELinux label user_home_t (non-default home mount) | Follow the manual fix in Podman Support → 1.c |
restorecon: unable to set context ... Permission denied on overlay diff paths during init | Rootless UID remapping prevents restorecon from accessing mapped-UID files | Expected and handled — see Podman Support → 1.c |
Authentication
| Symptom | Cause | Fix |
|---|---|---|
| GitHub OAuth callback URL mismatch | Callback URL doesn't match domain | Must be exactly https://<your-domain>/auth/v1/callback — check the GitHub OAuth App settings |
| GitHub login fails at github.com with "The redirect_uri is not associated with this application" | The redirect_uri GoTrue sent doesn't match the OAuth App's registered callback — usually because AUTH_SUPABASE_EXTERNAL_URL in .env is a host other than your public domain (see below) | Set AUTH_SUPABASE_EXTERNAL_URL=https://<your-domain> (or leave it unset), recreate the auth service, and confirm the OAuth App callback is https://<your-domain>/auth/v1/callback |
Wrong OAuth redirect_uri domain
The redirect_uri GoTrue sends to GitHub/Azure is not built from DOMAIN or GOTRUE_SITE_URL — it is built from AUTH_SUPABASE_EXTERNAL_URL:
GOTRUE_EXTERNAL_GITHUB_REDIRECT_URI: ${AUTH_SUPABASE_EXTERNAL_URL:-https://${DOMAIN}}/auth/v1/callback
GOTRUE_EXTERNAL_AZURE_REDIRECT_URI: ${AUTH_SUPABASE_EXTERNAL_URL:-https://${DOMAIN}}/auth/v1/callbackIf AUTH_SUPABASE_EXTERNAL_URL is set to a different host than the one users reach, GoTrue tells GitHub to redirect to that other host, which the OAuth App doesn't allow. Check the live value:
docker exec <tenant>-supabase-auth-1 printenv GOTRUE_EXTERNAL_GITHUB_REDIRECT_URI
# must be: https://<your-public-domain>/auth/v1/callbackTo fix, leave AUTH_SUPABASE_EXTERNAL_URL unset (defaults to https://${DOMAIN}) or set it to your public domain — https://<your-domain> with no trailing /auth/v1 (compose appends /auth/v1). Then recreate the auth service and confirm the GitHub OAuth App callback URL. The same variable drives API_EXTERNAL_URL and the SAML entity ID — see SAML entity ID has the wrong domain.
Cloud Supabase
| Symptom | Cause | Fix |
|---|---|---|
init cannot reach Supabase Cloud | Project IP allowlist or rotated key | Verify allowlist covers the server IP; curl -X GET "<url>/rest/v1/" -H "apikey: <anon>" to probe; check the service key |
network is unreachable for IPv6 ([2a05:d014:…]) | Server has no IPv6 (common on Hetzner) | Switch SUPABASE_DB_URL to Session pooler (port 6543) from Supabase Dashboard → Connect |
| Same as above (alternatives) | Same | Buy the Supabase IPv4 add-on, or enable host IPv6 |
Security best practices
| Area | Action |
|---|---|
| Firewall (UFW) | sudo ufw allow 22,80,443/tcp && sudo ufw enable — SSH + HTTP redirect + HTTPS only |
| Host updates | Enable unattended security updates |
| Secret rotation | Rotate DOCKER_REGISTRY_TOKEN + Supabase service keys quarterly |
| Log shipping | Ship server logs to a central store (don't grep on the box) |
| Tenant backups | Snapshot <base-dir>/tenants/<tenant>/ — covers .env, generated compose, admin credentials.json |
| Database backups | Snapshot Supabase (cloud or self-hosted Postgres) on its own schedule — independent of tenant files |
| Admin dashboards | Restrict the admin surfaces (Hatchet 8080, Studio 3100, registry 4873) to admin IPs via ALLOWED_IPS in .env; in cloudflare mode these move to the DNS-only admin subdomain. Restart the stack after changes. |