Updating sfp server

Update a self-hosted sfp server with sfp server update — the drain, backup, collation reconcile, migration, and recovery steps the command runs.

sfp server update moves a tenant to a new server image. It runs as a maintenance-window update: the command backs up the tenant's configuration, waits for in-flight workflows to finish, swaps the running services for the new image, and re-applies database migrations. It does not do a zero-downtime rolling swap — the application services are stopped and recreated, and Caddy serves a maintenance page while they are down.

Run it on the server, or from your workstation over SSH with --ssh-connection and --identity-file. Every invocation takes --tenant.

sfp server update --tenant <tenant> --base-dir /opt/sfp-server

What the command does

The update runs a fixed sequence for the named tenant. Each step is described below in the order it happens.

1. Back up critical files. Unless --skip-backup is passed, the command copies .env, config/Caddyfile, and the secrets/ directory into tenants/<tenant>/backups/update-<timestamp>/ before anything changes. A backup failure is logged but does not stop the update.
2. Rewrite config and .env. The compose file, Caddyfile, and supporting config are regenerated from the current template, and the new image tag is written into .env as IMAGE_TAG (see Selecting the version). The API gateway configuration under config/envoy/ is reconciled to the installed release — the route map can change between releases — and legacy Kong files (kong.yml, kong-entrypoint.sh) plus the retired config.yaml/config.<env>.yaml secret stashes are removed.
3. Drain active workflows and pre-pull the image, in parallel. The command waits for in-flight Hatchet workflows to complete (see Workflow drain) while pulling the new image in the background, so the pull is already done when services stop. If the drain times out with workflows still active, the update aborts here, before any service is stopped.
4. Recreate Caddy. Caddy is brought up with the new config so it serves the maintenance page while the application is down.
5. Stop the application services. The application services — server, workers, file storage, and the Supabase API gateway — are stopped and removed. The remaining infrastructure (the Hatchet engine and databases, the Supabase database, the registry, metrics and logs stores) stays up; if the selective stop fails, the command falls back to stopping the whole stack.
6. Rebuild indexes when the postgres image changed the collation version. If the supabase/postgres base image in this release carries a new glibc, the command detects the resulting collation-version mismatch and reindexes each affected database before any migration runs. It extends the maintenance window once per affected database and is a no-op on every other update. See Rebuild indexes after a postgres image bump. A failure here is logged and does not stop the update; it is retried on the next update.
7. Start the new services. docker compose up -d recreates the stopped services on the new image and re-runs the migration init container.
8. Wait for migrations. The command polls the migration container until it exits, then verifies the exit code (see Database migrations).

Selecting the version

With no version flag, the update keeps the tenant's current IMAGE_TAG and re-runs the update sequence against it. Select a different version explicitly:

Flag	Effect
--cadence <cadence>	Follows a named release cadence
--docker-tag <tag>	Deploys a specific image tag, overriding the cadence

The resolved value is written into the tenant's .env as IMAGE_TAG, which the compose file interpolates into every service image. Check the sfp release notes for breaking changes before pinning a new version.

Workflow drain

Stopping the workers mid-run aborts the workflows they are executing, so the update drains before it stops them. The drain polls the server's /sfp/api health endpoint, which reports the active-workflow count as activeWorkflows: { running, queued, total }, every 10 seconds.

The update proceeds in these cases:

• A confirmed count of zero — no workflows are running or queued.
• The server is unreachable after the initial retries — the server is most likely already down, so there is nothing to protect. This is reported explicitly, not as "no active workflows".
• --force is passed, or no domain is configured for the tenant (the drain needs a reachable server URL). This stops the workers immediately and aborts any workflow still running.

If the drain timeout elapses with workflows still active — after --drain-timeout seconds (default 3600) — the update aborts. It stops before touching the running services, so nothing is interrupted; re-run with --force to update anyway (which interrupts the in-flight work), or retry once the workflows have finished.

An unknown count — the health probe is degraded but the server is up — or a transient error during the wait never short-circuits to "proceed". The drain keeps waiting for a confirmed count or the timeout instead.

On a tenant with continuous scheduled work — recurring jobs, long pool preparations that overlap across time zones — the active-workflow count may never reach zero. The drain then waits out the full timeout and the update aborts. Before updating such a tenant, pause scheduled jobs and let in-flight pool operations finish, or schedule the update in a quiet window.

--force stops the workers immediately. Any workflow that is running at that moment — a build, a release, a scratch-org provisioning task — is aborted. Use it only when you know nothing is in flight, or when the drain cannot reach the server.

Backup and recovery

The pre-update backup captures the files that carry the tenant's configuration and secrets:

Path	What it holds
.env	Image tag, worker counts, ports, and all runtime configuration
config/Caddyfile	The reverse-proxy and TLS configuration
secrets/	The tenant's secret material

Each backup lands in tenants/<tenant>/backups/update-<timestamp>/. It does not include the database or the Docker volumes — those are covered by your own volume-level backups (the critical data volumes are declared external in the compose file so they survive docker compose down).

To recover the configuration from a backup, copy the files back into the tenant directory and start the server:

cp -r tenants/<tenant>/backups/update-<timestamp>/. tenants/<tenant>/
sfp server start --tenant <tenant> --base-dir /opt/sfp-server

Database migrations

Migrations are applied by the supabase-migrations init container when the services start, not by the CLI directly. The update copies the new migration SQL into place, and when services restart the init container applies any migrations the new image adds.

Migrations are forward-only — the schema changes a newer version applies cannot be automatically reversed. After the services start, the command polls the migration container for up to two minutes:

• Exit code 0 — migrations applied. The command reloads the PostgREST schema cache and reports success.
• Non-zero exit — the command prints the migration container's logs and stops. Timeout — the command fails without printing them; read the migration container's logs directly. Either way the services are left on the new image with migrations unapplied — fix the cause before continuing.

Pass --continue-on-db-migration-failure to let the update finish even when a migration fails. The services are left restarted on the new image, and the failure is logged rather than raised. Use this only when you intend to resolve the migration state manually afterwards.

If a migration fails because of a manual schema change that conflicts with the migration (for example, a column the migration adds already exists), reconcile the schema against the migration's expectation, or restore the database from your volume backup, before retrying.

Rebuild indexes after a postgres image bump

A supabase/postgres base-image bump can carry a new glibc. A change from older builds to 15.14.x, for example, moved glibc 2.39 → 2.40. Postgres then warns that every database "has a collation version mismatch": btree indexes on text were built under the old ordering and may no longer agree with the collation the server now computes, which can drop rows from index scans and weaken unique constraints.

To correct this, sfp server update detects the mismatch and rebuilds the affected indexes before migrations run — a migration that depends on a unique index gets a sound one. This runs only for a self-hosted stack (a cloud Supabase project has no supabase-db container and is not reindexed).

For each affected database the command runs, as the supabase_admin superuser:

1. REINDEX DATABASE <db>
2. ALTER DATABASE <db> REFRESH COLLATION VERSION

The two statements run as separate invocations: REINDEX DATABASE (and REINDEX CONCURRENTLY) refuse to run inside a transaction block, and the Supabase CLI applies migrations in one. That is why this step is part of the update and not a migration.

Operator-visible effect:

• It extends the update's maintenance window once per affected database. The application is already down behind the Caddy maintenance page when this runs, and REINDEX holds the database — the added downtime is the reindex time.
• It runs at most once per database. After the indexes are rebuilt and the recorded version refreshed, subsequent updates detect no mismatch (one cheap detection query per update) and add no delay. Fresh installs record the current version at init and are unaffected.
• It is non-fatal. If the reconcile fails, the update logs a warning and continues; the mismatch keeps warning on later updates and is retried then.
• template0 is left untouched: it accepts no connections, holds no user data, and cannot be reindexed without altering pg_database.

Flags

Flag	Default	Purpose
--tenant, -t	(required)	Tenant to update
--base-dir	./sfp-server	Directory that holds the tenant tree
--cadence, -r	latest	Release cadence to follow
--docker-tag	(unset)	Specific image tag, overrides --cadence
--skip-backup	off	Skip the pre-update backup
--force	off	Skip the drain and update even if workflows are in flight
--drain-timeout	3600	Seconds to wait for the drain before the update aborts
--continue-on-db-migration-failure	off	Finish the update even if a migration fails
--ssh-connection	(unset)	Run on a remote host: user@host[:port]
--identity-file	(unset)	SSH private key for --ssh-connection
--config-file	(unset)	JSON file with server configuration values

Secrets during an update

The update reads the tenant's secrets the same way start does, through the configured secrets provider. Pass --secrets-provider to select it:

• infisical — a dedicated Infisical secrets manager.
• aws-secretsmanager — AWS Secrets Manager.
• custom — read from environment variables. Inject them before running the command using your secrets tooling; consult its documentation.

Running the update from a pipeline

The same command can run from a CI/CD pipeline instead of an operator shell: the pipeline installs the sfp CLI and runs sfp server update (over SSH with --ssh-connection when the runner is not the host), so updates leave an audit trail in the pipeline history. The requirements are the same as the manual path — network access to the host and the registry credentials — provided through your CI platform's runners and secrets; consult its documentation.

Related pages

OperationsScaling workersSelf Hosted Supabase Configuration