Troubleshooting
Common issues when setting up and running sfp server.
General
denied: denied from Docker login
Stale token, wrong registry
docker logout <registry> && echo "<pat>" | docker login <registry> -u <user> --password-stdin
Domain returns NXDOMAIN
DNS not propagated or A record missing
nslookup <domain>; verify A record points at server; allow up to 48h propagation
Need raw container logs
Compose project name matches tenant
cd /opt/sfp-server/tenants/<tenant> && docker compose -p <tenant> logs -f
TLS / Certificates
Caddy logs cert file not found
TLS cert not placed or not decoded
Re-run init with ORIGIN_CERT/ORIGIN_KEY secrets, or manually place origin.pem + origin-key.pem in {tenantDir}/certs/
Let's Encrypt fails to issue cert
DNS not publicly resolvable or port 80 blocked
Verify dig <domain> returns server IP from the public internet; open port 80 inbound for ACME challenge
Certificate format error
Cert not PEM-encoded or not base64
Certs must be PEM format; .crt/.key files are typically already PEM — rename and base64-encode: base64 -w 0 origin.pem
Services
external volume "…" not found
Volumes not created yet
Use sfp server start (not raw docker compose up); it creates external volumes automatically
Server healthy but curl returns 502
App server not ready yet
Wait 30–60s after start; check sfp server logs --service app; Caddy shows a maintenance page until backend is up
Supabase containers not starting
Missing compose profile
sfp server start activates the supabase profile automatically; raw docker compose up does not
VaultBootstrapService fetch failed
Supabase auth not ready at server boot
Restart the server container: docker compose -p <tenant> restart server; auth containers need ~15s to initialize
SSH
sfp server init: SSH connection error: Timed out while waiting for handshake then Docker is not installed or not available
Target port 22 unreachable (firewall, security group, bastion-only network). The "Docker" line is misleading — read the SSH line above it.
Open a local tunnel via the bastion: ssh -fNT -L 2222:<target-host>:22 <user>@<bastion>, then sfp server init --ssh-connection <user>@127.0.0.1:2222 --identity-file ~/.ssh/<key>; tear down with pkill -f "ssh -fNT -L 2222".
sfp server init: getaddrinfo EAI_AGAIN <host-alias>
--ssh-connection was passed an ~/.ssh/config Host alias. sfp uses Node ssh2, which does not read ~/.ssh/config — Host, ProxyJump, IdentityFile, User are all ignored.
Pass a literal user@host[:port] plus --identity-file <path>. For ProxyJump-style routing, use the local tunnel recipe in the row above.
sfp server init: Failed to read private key: ENOENT <path>
--identity-file path missing or unreadable by the user running sfp.
Pass an absolute path: --identity-file /home/<user>/.ssh/<key>. ~/ expands; $HOME does not.
sfp server init: No authentication method provided
Neither --identity-file nor --passphrase passed. sfp does not fall back to ssh-agent or ~/.ssh/id_rsa.
Always pass --identity-file <path> with --ssh-connection. No --use-agent flag exists.
Proxy / Client IP
sfp-server logs show wrong client IP (e.g. 127.0.0.1 or a proxy IP instead of the real client)
trustProxyHops is too low — sfp-server's Express layer reads X-Forwarded-For only up to the trusted hop count
Set TRUST_PROXY_HOPS=3 in .env for a Cloudflare → customer LB → Caddy → app chain (default 2 covers Cloudflare → Caddy → app). Range 0–10. Restart/recreate the stack.
Rate limiting applies to the wrong address
Same as above — Express-layer IP attribution and rate limits derive from the trusted X-Forwarded-For hop
Same fix: increase TRUST_PROXY_HOPS to match the actual proxy chain depth. All inbound paths must have the same hop count. Note: Caddy admin-port allowlists (ALLOWED_IPS) use remote_ip independently of this setting — see Security best practices.
Monitoring allowlists fail or audit logs attribute actions to a proxy IP
trustProxyHops too low, the proxy chain has inconsistent hop counts, or proxies are not forwarding X-Forwarded-For
Verify every proxy in the chain preserves/sets X-Forwarded-For, X-Forwarded-Proto, and X-Forwarded-Host; then set TRUST_PROXY_HOPS=3 for Cloudflare → customer LB → Caddy → app and restart/recreate the stack. All inbound paths must have the same hop count.
Webhooks
GitHub webhook delivery times out or reports connection refused
The webhook URL points to a private, VPN-only, or bastion-only host. GitHub.com cannot use SSH ProxyJump, an operator VPN, or a bastion path.
Expose a public HTTPS ingress that forwards privately to the server: public ALB / customer load balancer, managed reverse proxy / tunnel, or API Gateway with VPC Link / private integration. Verify the exact configured payload URL with GitHub Recent deliveries / Redeliver or the provider's equivalent delivery tooling. See Webhook ingress for private servers and Prerequisites → [2.e] Firewall / Security Group.
Firewall rules would require chasing many GitHub source IP ranges
GitHub.com webhook delivery IPs can change, and source IP is not the authenticity control
Do not make GitHub IP allowlists the primary design. Expose only the public HTTPS ingress, restrict the private backend to the ALB/proxy/tunnel/VPC Link path, and preserve sfp-server signature validation.
Public ingress exposes the whole sfp UI/API when only webhooks should be public
Load balancer, tunnel, or API Gateway route forwards all paths to Caddy
Add a path/method rule for only POST /sfp/api/repository/webhook; return 403 or 404 for other public paths unless the full sfp UI/API is intentionally public. Keep the backend host private.
Webhook reaches the load balancer but sfp-server sees HTTP or the wrong host
Load balancer / API Gateway is not preserving forwarded headers
Forward or set X-Forwarded-Proto: https, X-Forwarded-Host, and X-Forwarded-For; then set TRUST_PROXY_HOPS to the number of trusted proxy hops. See Proxy / Client IP.
Webhook through API Gateway returns 503 Service Unavailable
API Gateway VPC Link is still provisioning, cannot reach the load balancer listener, or targets an internet-facing load balancer instead of the intended private integration target
Wait until the VPC Link is AVAILABLE; use HTTP API private integration to an internal ALB/NLB listener; allow the VPC Link security group to reach the internal load balancer; verify the internal load balancer target group is healthy.
Webhook through API Gateway returns 404 or never reaches the expected Caddy route
API Gateway private integration is forwarding the stage or base path to the backend
Use a custom domain/base path mapping or request path override so Caddy receives the same path configured as the webhook payload URL.
Webhook signature validation fails behind a proxy, ALB, or API Gateway
Ingress changed the request body, dropped signature headers, or parsed and re-serialized the payload before sfp-server validation
Forward the raw request body unchanged and preserve provider signature/event headers. For GitHub, preserve X-Hub-Signature-256, X-GitHub-Event, and X-GitHub-Delivery. Remove API Gateway mapping templates or proxy body rewrites from webhook routes.
GitHub marks the delivery failed even though it reached the server
The ingress or backend waited for workflow execution instead of acknowledging receipt
The webhook request must return 2xx quickly. sfp-server dispatches to Hatchet and returns immediately; check ingress timeouts and logs if the request waits for workflow completion.
SELinux (Podman rootless on RHEL / Fedora)
cannot apply additional memory protection after relocation: Permission denied or RELRO protection failed
Podman graphroot has SELinux label user_home_t (non-default home mount)
Follow the manual fix procedure in Podman Support → 1.c
restorecon: unable to set context ... Permission denied on overlay diff paths during init
Rootless UID remapping prevents restorecon from accessing mapped-UID files
Expected and handled — see Podman Support → 1.c for details
Authentication
GitHub OAuth callback URL mismatch
Callback URL doesn't match domain
Must be exactly https://<your-domain>/auth/v1/callback — check GitHub OAuth App settings
GitHub login fails at github.com with "The redirect_uri is not associated with this application"
The redirect_uri GoTrue sent doesn't match the OAuth App's registered callback — usually because AUTH_SUPABASE_EXTERNAL_URL in the server .env is set to a host other than your public domain (see Wrong OAuth redirect_uri below)
Set AUTH_SUPABASE_EXTERNAL_URL=https://<your-domain> (or leave it unset), recreate the auth service, and confirm the OAuth App callback is https://<your-domain>/auth/v1/callback
Wrong OAuth redirect_uri domain
The redirect_uri GoTrue sends to GitHub/Azure is not built from DOMAIN or GOTRUE_SITE_URL — it is built from AUTH_SUPABASE_EXTERNAL_URL:
So if AUTH_SUPABASE_EXTERNAL_URL is set to a different host than the one users actually reach (for example a raw provisioning hostname instead of your public domain), GoTrue tells GitHub to redirect back to that other host. The OAuth App doesn't allow it, and GitHub shows "The redirect_uri is not associated with this application" — even though GOTRUE_SITE_URL and your browser are on the correct public domain.
Check the live value the auth container is using:
To fix, either leave AUTH_SUPABASE_EXTERNAL_URL unset (it then defaults to https://${DOMAIN}, which is always correct) or set it to your public domain — https://<your-domain> with no trailing /auth/v1 (compose appends /auth/v1 outside the default expression). Then recreate the auth service so it picks up the change, and confirm the GitHub OAuth App's Authorization callback URL is https://<your-domain>/auth/v1/callback. The same variable also drives API_EXTERNAL_URL and the SAML entity ID — see SAML entity ID has the wrong domain.
Cloud Supabase
init cannot reach Supabase Cloud
Project IP allowlist or rotated key
Verify allowlist covers server IP; curl -X GET "<url>/rest/v1/" -H "apikey: <anon>" to probe; check service key
network is unreachable for IPv6 ([2a05:d014:…])
Server has no IPv6 (common on Hetzner)
Switch SUPABASE_DB_URL to Session pooler (port 6543) from Supabase Dashboard → Connect
Same as above (alternatives)
Same
Buy Supabase IPv4 add-on; or enable host IPv6
Security best practices
Firewall (UFW)
sudo ufw allow 22,80,443/tcp && sudo ufw enable — SSH + HTTP redirect + HTTPS only
Host updates
Enable unattended security updates
Secret rotation
Rotate DOCKER_REGISTRY_TOKEN + Supabase service keys quarterly
Log shipping
Ship server logs to a central store (don't grep on the box)
Tenant backups
Snapshot <base-dir>/tenants/<tenant>/ — covers .env, generated compose, admin credentials.json
Database backups
Snapshot Supabase (cloud or self-hosted Postgres) on its own schedule — independent of tenant files
Admin dashboards
Restrict ports 8080, 3100, 4873 to admin IPs via ALLOWED_IPS in .env; restart Caddy after changes
Last updated
Was this helpful?