> For the complete documentation index, see [llms.txt](https://docs.flxbl.io/flxbl/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.flxbl.io/flxbl/sfp-server/setting-up/setting-up-sfp-server/troubleshooting.md). # Troubleshooting Common issues when setting up and running sfp server. *** ## General | Symptom | Cause | Fix | | ---------------------------------- | ---------------------------------------- | ------------------------------------------------------------------------------------------------ | | `denied: denied` from Docker login | Stale token, wrong registry | `docker logout && echo "" \| docker login -u --password-stdin` | | Domain returns NXDOMAIN | DNS not propagated or `A` record missing | `nslookup `; verify `A` record points at server; allow up to 48h propagation | | Need raw container logs | Compose project name matches tenant | `cd /opt/sfp-server/tenants/ && docker compose -p logs -f` | ## TLS / Certificates | Symptom | Cause | Fix | | --------------------------------- | ---------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | Caddy logs `cert file not found` | TLS cert not placed or not decoded | Re-run init with `ORIGIN_CERT`/`ORIGIN_KEY` secrets, or manually place `origin.pem` + `origin-key.pem` in `{tenantDir}/certs/` | | Let's Encrypt fails to issue cert | DNS not publicly resolvable or port 80 blocked | Verify `dig ` returns server IP from the public internet; open port 80 inbound for ACME challenge | | Certificate format error | Cert not PEM-encoded or not base64 | Certs must be PEM format; `.crt`/`.key` files are typically already PEM — rename and base64-encode: `base64 -w 0 origin.pem` | ## Services | Symptom | Cause | Fix | | ------------------------------------- | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------- | | `external volume "…" not found` | Volumes not created yet | Use `sfp server start` (not raw `docker compose up`); it creates external volumes automatically | | Server healthy but `curl` returns 502 | App server not ready yet | Wait 30–60s after start; check `sfp server logs --service app`; Caddy shows a maintenance page until backend is up | | Supabase containers not starting | Missing compose profile | `sfp server start` activates the `supabase` profile automatically; raw `docker compose up` does not | | `VaultBootstrapService` fetch failed | Supabase auth not ready at server boot | Restart the server container: `docker compose -p restart server`; auth containers need \~15s to initialize | ## SSH | Symptom | Cause | Fix | | -------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `sfp server init`: `SSH connection error: Timed out while waiting for handshake` then `Docker is not installed or not available` | Target port 22 unreachable (firewall, security group, bastion-only network). The "Docker" line is misleading — read the SSH line above it. | Open a local tunnel via the bastion: `ssh -fNT -L 2222::22 @`, then `sfp server init --ssh-connection @127.0.0.1:2222 --identity-file ~/.ssh/`; tear down with `pkill -f "ssh -fNT -L 2222"`. | | `sfp server init`: `getaddrinfo EAI_AGAIN ` | `--ssh-connection` was passed an `~/.ssh/config` Host alias. `sfp` uses Node `ssh2`, which does not read `~/.ssh/config` — `Host`, `ProxyJump`, `IdentityFile`, `User` are all ignored. | Pass a literal `user@host[:port]` plus `--identity-file `. For `ProxyJump`-style routing, use the local tunnel recipe in the row above. | | `sfp server init`: `Failed to read private key: ENOENT ` | `--identity-file` path missing or unreadable by the user running `sfp`. | Pass an absolute path: `--identity-file /home//.ssh/`. `~/` expands; `$HOME` does not. | | `sfp server init`: `No authentication method provided` | Neither `--identity-file` nor `--passphrase` passed. `sfp` does not fall back to `ssh-agent` or `~/.ssh/id_rsa`. | Always pass `--identity-file ` with `--ssh-connection`. No `--use-agent` flag exists. | ## Proxy / Client IP | Symptom | Cause | Fix | | ------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | sfp-server logs show wrong client IP (e.g. `127.0.0.1` or a proxy IP instead of the real client) | `trustProxyHops` is too low — sfp-server's Express layer reads `X-Forwarded-For` only up to the trusted hop count | Set `TRUST_PROXY_HOPS=3` in `.env` for a `Cloudflare → customer LB → Caddy → app` chain (default `2` covers `Cloudflare → Caddy → app`). Range `0`–`10`. Restart/recreate the stack. | | Rate limiting applies to the wrong address | Same as above — Express-layer IP attribution and rate limits derive from the trusted `X-Forwarded-For` hop | Same fix: increase `TRUST_PROXY_HOPS` to match the actual proxy chain depth. All inbound paths must have the same hop count. Note: Caddy admin-port allowlists (`ALLOWED_IPS`) use `remote_ip` independently of this setting — see [Security best practices](#security-best-practices). | | Monitoring allowlists fail or audit logs attribute actions to a proxy IP | `trustProxyHops` too low, the proxy chain has inconsistent hop counts, or proxies are not forwarding `X-Forwarded-For` | Verify every proxy in the chain preserves/sets `X-Forwarded-For`, `X-Forwarded-Proto`, and `X-Forwarded-Host`; then set `TRUST_PROXY_HOPS=3` for `Cloudflare → customer LB → Caddy → app` and restart/recreate the stack. All inbound paths must have the same hop count. | ## Webhooks | Symptom | Cause | Fix | | --------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | GitHub webhook delivery times out or reports connection refused | The webhook URL points to a private, VPN-only, or bastion-only host. GitHub.com cannot use SSH `ProxyJump`, an operator VPN, or a bastion path. | Expose a public HTTPS ingress that forwards privately to the server: public ALB / customer load balancer, managed reverse proxy / tunnel, or API Gateway with VPC Link / private integration. Verify the exact configured payload URL with GitHub **Recent deliveries** / **Redeliver** or the provider's equivalent delivery tooling. See [Webhook ingress for private servers](/flxbl/sfp-server/setting-up/webhook-ingress-for-private-servers.md) and [Prerequisites → \[2.e\] Firewall / Security Group](/flxbl/sfp-server/setting-up/prerequisites.md#2e-firewall--security-group). | | Firewall rules would require chasing many GitHub source IP ranges | GitHub.com webhook delivery IPs can change, and source IP is not the authenticity control | Do not make GitHub IP allowlists the primary design. Expose only the public HTTPS ingress, restrict the private backend to the ALB/proxy/tunnel/VPC Link path, and preserve sfp-server signature validation. | | Public ingress exposes the whole sfp UI/API when only webhooks should be public | Load balancer, tunnel, or API Gateway route forwards all paths to Caddy | Add a path/method rule for only `POST /sfp/api/repository/webhook`; return `403` or `404` for other public paths unless the full sfp UI/API is intentionally public. Keep the backend host private. | | Webhook reaches the load balancer but sfp-server sees HTTP or the wrong host | Load balancer / API Gateway is not preserving forwarded headers | Forward or set `X-Forwarded-Proto: https`, `X-Forwarded-Host`, and `X-Forwarded-For`; then set `TRUST_PROXY_HOPS` to the number of trusted proxy hops. See [Proxy / Client IP](#proxy--client-ip). | | Webhook through API Gateway returns `503 Service Unavailable` | API Gateway VPC Link is still provisioning, cannot reach the load balancer listener, or targets an internet-facing load balancer instead of the intended private integration target | Wait until the VPC Link is `AVAILABLE`; use HTTP API private integration to an internal ALB/NLB listener; allow the VPC Link security group to reach the internal load balancer; verify the internal load balancer target group is healthy. | | Webhook through API Gateway returns 404 or never reaches the expected Caddy route | API Gateway private integration is forwarding the stage or base path to the backend | Use a custom domain/base path mapping or request path override so Caddy receives the same path configured as the webhook payload URL. | | Webhook signature validation fails behind a proxy, ALB, or API Gateway | Ingress changed the request body, dropped signature headers, or parsed and re-serialized the payload before sfp-server validation | Forward the raw request body unchanged and preserve provider signature/event headers. For GitHub, preserve `X-Hub-Signature-256`, `X-GitHub-Event`, and `X-GitHub-Delivery`. Remove API Gateway mapping templates or proxy body rewrites from webhook routes. | | GitHub marks the delivery failed even though it reached the server | The ingress or backend waited for workflow execution instead of acknowledging receipt | The webhook request must return `2xx` quickly. sfp-server dispatches to Hatchet and returns immediately; check ingress timeouts and logs if the request waits for workflow completion. | ## SELinux (Podman rootless on RHEL / Fedora) | Symptom | Cause | Fix | | ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cannot apply additional memory protection after relocation: Permission denied` or `RELRO protection failed` | Podman graphroot has SELinux label `user_home_t` (non-default home mount) | Follow the manual fix procedure in [Podman Support → 1.c](/flxbl/sfp-server/setting-up/podman.md#1c-selinux--graphroot-labels-rootless-podman-on-rhel) | | `restorecon: unable to set context ... Permission denied` on overlay diff paths during init | Rootless UID remapping prevents `restorecon` from accessing mapped-UID files | Expected and handled — see [Podman Support → 1.c](/flxbl/sfp-server/setting-up/podman.md#1c-selinux--graphroot-labels-rootless-podman-on-rhel) for details | ## Authentication | Symptom | Cause | Fix | | ----------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | GitHub OAuth callback URL mismatch | Callback URL doesn't match domain | Must be exactly `https:///auth/v1/callback` — check GitHub OAuth App settings | | GitHub login fails at github.com with **"The redirect\_uri is not associated with this application"** | The `redirect_uri` GoTrue sent doesn't match the OAuth App's registered callback — usually because `AUTH_SUPABASE_EXTERNAL_URL` in the server `.env` is set to a host other than your public domain (see [Wrong OAuth redirect\_uri](#wrong-oauth-redirect_uri-domain) below) | Set `AUTH_SUPABASE_EXTERNAL_URL=https://` (or leave it unset), recreate the auth service, and confirm the OAuth App callback is `https:///auth/v1/callback` | ### Wrong OAuth redirect\_uri domain The `redirect_uri` GoTrue sends to GitHub/Azure is **not** built from `DOMAIN` or `GOTRUE_SITE_URL` — it is built from `AUTH_SUPABASE_EXTERNAL_URL`: ```yaml GOTRUE_EXTERNAL_GITHUB_REDIRECT_URI: ${AUTH_SUPABASE_EXTERNAL_URL:-https://${DOMAIN}}/auth/v1/callback GOTRUE_EXTERNAL_AZURE_REDIRECT_URI: ${AUTH_SUPABASE_EXTERNAL_URL:-https://${DOMAIN}}/auth/v1/callback ``` So if `AUTH_SUPABASE_EXTERNAL_URL` is set to a different host than the one users actually reach (for example a raw provisioning hostname instead of your public domain), GoTrue tells GitHub to redirect back to that other host. The OAuth App doesn't allow it, and GitHub shows **"The redirect\_uri is not associated with this application"** — even though `GOTRUE_SITE_URL` and your browser are on the correct public domain. Check the live value the auth container is using: ```bash docker exec -supabase-auth-1 printenv GOTRUE_EXTERNAL_GITHUB_REDIRECT_URI # must be: https:///auth/v1/callback ``` To fix, either **leave `AUTH_SUPABASE_EXTERNAL_URL` unset** (it then defaults to `https://${DOMAIN}`, which is always correct) or set it to your public domain — `https://` with **no trailing `/auth/v1`** (compose appends `/auth/v1` outside the default expression). Then recreate the auth service so it picks up the change, and confirm the GitHub OAuth App's *Authorization callback URL* is `https:///auth/v1/callback`. The same variable also drives `API_EXTERNAL_URL` and the SAML entity ID — see [SAML entity ID has the wrong domain](/flxbl/sfp-server/setting-up/saml-authentication/troubleshooting.md#saml-entity-id-has-the-wrong-domain-self-hosted). ## Cloud Supabase | Symptom | Cause | Fix | | --------------------------------------------------- | -------------------------------------- | ----------------------------------------------------------------------------------------------------------------- | | `init` cannot reach Supabase Cloud | Project IP allowlist or rotated key | Verify allowlist covers server IP; `curl -X GET "/rest/v1/" -H "apikey: "` to probe; check service key | | `network is unreachable` for IPv6 (`[2a05:d014:…]`) | Server has no IPv6 (common on Hetzner) | Switch `SUPABASE_DB_URL` to **Session pooler** (port 6543) from Supabase Dashboard → Connect | | Same as above (alternatives) | Same | Buy Supabase IPv4 add-on; or enable host IPv6 | *** ## Security best practices | Area | Action | | ---------------- | ----------------------------------------------------------------------------------------------------- | | Firewall (UFW) | `sudo ufw allow 22,80,443/tcp && sudo ufw enable` — SSH + HTTP redirect + HTTPS only | | Host updates | Enable unattended security updates | | Secret rotation | Rotate `DOCKER_REGISTRY_TOKEN` + Supabase service keys quarterly | | Log shipping | Ship server logs to a central store (don't grep on the box) | | Tenant backups | Snapshot `/tenants//` — covers `.env`, generated compose, admin `credentials.json` | | Database backups | Snapshot Supabase (cloud or self-hosted Postgres) on its own schedule — independent of tenant files | | Admin dashboards | Restrict ports 8080, 3100, 4873 to admin IPs via `ALLOWED_IPS` in `.env`; restart Caddy after changes | --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.flxbl.io/flxbl/sfp-server/setting-up/setting-up-sfp-server/troubleshooting.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.