> For the complete documentation index, see [llms.txt](https://docs.flxbl.io/flxbl/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.flxbl.io/flxbl/sfp-server/setting-up/updating-sfp-server.md).

# Updating sfp server

`sfp server update` moves a tenant to a new server image. It runs as a maintenance-window update: the command backs up the tenant's configuration, waits for in-flight workflows to finish, swaps the running services for the new image, and re-applies database migrations. It does not do a zero-downtime rolling swap — the application services are stopped and recreated, and Caddy serves a maintenance page while they are down.

Run it on the server, or from your workstation over SSH with `--ssh-connection` and `--identity-file`. Every invocation takes `--tenant`.

```bash
sfp server update --tenant <tenant> --base-dir /opt/sfp-server
```

## What the command does

The update runs a fixed sequence for the named tenant. Each step is described below in the order it happens.

1. **Back up critical files.** Unless `--skip-backup` is passed, the command copies `.env`, `config/Caddyfile`, and the `secrets/` directory into `tenants/<tenant>/backups/update-<timestamp>/` before anything changes. A backup failure is logged but does not stop the update.
2. **Rewrite config and `.env`.** The compose file, `Caddyfile`, and supporting config are regenerated from the current template, and the new image tag is written into `.env` as `IMAGE_TAG` (see [Selecting the version](#selecting-the-version)).
3. **Drain active workflows and pre-pull the image, in parallel.** The command waits for in-flight Hatchet workflows to complete (see [Workflow drain](#workflow-drain)) while pulling the new image in the background, so the pull is already done when services stop. If the drain times out with workflows still active, the update aborts here, before any service is stopped.
4. **Recreate Caddy.** Caddy is brought up with the new config so it serves the maintenance page while the application is down.
5. **Stop the application services.** The application services — server, workers, and file storage — are stopped and removed. The remaining infrastructure (the Hatchet engine and databases, the registry, metrics and logs stores) stays up; if the selective stop fails, the command falls back to stopping the whole stack.
6. **Start the new services.** `docker compose up -d` recreates the stopped services on the new image and re-runs the migration init container.
7. **Wait for migrations.** The command polls the migration container until it exits, then verifies the exit code (see [Database migrations](#database-migrations)).

## Selecting the version

With no version flag, the update keeps the tenant's current `IMAGE_TAG` and re-runs the update sequence against it. Select a different version explicitly:

| Flag                  | Effect                                               |
| --------------------- | ---------------------------------------------------- |
| `--cadence <cadence>` | Follows a named release cadence                      |
| `--docker-tag <tag>`  | Deploys a specific image tag, overriding the cadence |

The resolved value is written into the tenant's `.env` as `IMAGE_TAG`, which the compose file interpolates into every service image. Check the [sfp release notes](https://github.com/flxbl-io/sfp/releases) for breaking changes before pinning a new version.

## Workflow drain

Stopping the workers mid-run aborts the workflows they are executing, so the update drains before it stops them. The drain polls the server's `/sfp/api` health endpoint, which reports the active-workflow count as `activeWorkflows: { running, queued, total }`, every 10 seconds.

The update proceeds in these cases:

* **A confirmed count of zero** — no workflows are running or queued.
* **The server is unreachable** after the initial retries — the server is most likely already down, so there is nothing to protect. This is reported explicitly, not as "no active workflows".
* **`--force` is passed**, or no domain is configured for the tenant (the drain needs a reachable server URL). This stops the workers immediately and aborts any workflow still running.

If the drain timeout elapses with workflows still active — after `--drain-timeout` seconds (default `3600`) — the update **aborts**. It stops before touching the running services, so nothing is interrupted; re-run with `--force` to update anyway (which interrupts the in-flight work), or retry once the workflows have finished.

An unknown count — the health probe is degraded but the server is up — or a transient error during the wait never short-circuits to "proceed". The drain keeps waiting for a confirmed count or the timeout instead.

{% hint style="warning" %}
On a tenant with continuous scheduled work — recurring jobs, long pool preparations that overlap across time zones — the active-workflow count may never reach zero. The drain then waits out the full timeout and the update aborts. Before updating such a tenant, pause scheduled jobs and let in-flight pool operations finish, or schedule the update in a quiet window.
{% endhint %}

{% hint style="warning" %}
`--force` stops the workers immediately. Any workflow that is running at that moment — a build, a release, a scratch-org provisioning task — is aborted. Use it only when you know nothing is in flight, or when the drain cannot reach the server.
{% endhint %}

## Backup and recovery

The pre-update backup captures the files that carry the tenant's configuration and secrets:

| Path               | What it holds                                                  |
| ------------------ | -------------------------------------------------------------- |
| `.env`             | Image tag, worker counts, ports, and all runtime configuration |
| `config/Caddyfile` | The reverse-proxy and TLS configuration                        |
| `secrets/`         | The tenant's secret material                                   |

Each backup lands in `tenants/<tenant>/backups/update-<timestamp>/`. It does not include the database or the Docker volumes — those are covered by your own volume-level backups (the critical data volumes are declared `external` in the compose file so they survive `docker compose down`).

To recover the configuration from a backup, copy the files back into the tenant directory and start the server:

```bash
cp -r tenants/<tenant>/backups/update-<timestamp>/. tenants/<tenant>/
sfp server start --tenant <tenant> --base-dir /opt/sfp-server
```

## Database migrations

Migrations are applied by the `supabase-migrations` init container when the services start, not by the CLI directly. The update copies the new migration SQL into place, and when services restart the init container applies any migrations the new image adds.

Migrations are **forward-only** — the schema changes a newer version applies cannot be automatically reversed. After the services start, the command polls the migration container for up to two minutes:

* **Exit code 0** — migrations applied. The command reloads the PostgREST schema cache and reports success.
* **Non-zero exit** — the command prints the migration container's logs and stops. **Timeout** — the command fails without printing them; read the migration container's logs directly. Either way the services are left on the new image with migrations unapplied — fix the cause before continuing.

Pass `--continue-on-db-migration-failure` to let the update finish even when a migration fails. The services are left restarted on the new image, and the failure is logged rather than raised. Use this only when you intend to resolve the migration state manually afterwards.

If a migration fails because of a manual schema change that conflicts with the migration (for example, a column the migration adds already exists), reconcile the schema against the migration's expectation, or restore the database from your volume backup, before retrying.

## Flags

| Flag                                 | Default        | Purpose                                                   |
| ------------------------------------ | -------------- | --------------------------------------------------------- |
| `--tenant`, `-t`                     | *(required)*   | Tenant to update                                          |
| `--base-dir`                         | `./sfp-server` | Directory that holds the tenant tree                      |
| `--cadence`, `-r`                    | `latest`       | Release cadence to follow                                 |
| `--docker-tag`                       | *(unset)*      | Specific image tag, overrides `--cadence`                 |
| `--skip-backup`                      | off            | Skip the pre-update backup                                |
| `--force`                            | off            | Skip the drain and update even if workflows are in flight |
| `--drain-timeout`                    | `3600`         | Seconds to wait for the drain before the update aborts    |
| `--continue-on-db-migration-failure` | off            | Finish the update even if a migration fails               |
| `--ssh-connection`                   | *(unset)*      | Run on a remote host: `user@host[:port]`                  |
| `--identity-file`                    | *(unset)*      | SSH private key for `--ssh-connection`                    |
| `--config-file`                      | *(unset)*      | JSON file with server configuration values                |

## Secrets during an update

The update reads the tenant's secrets the same way `start` does, through the configured secrets provider. Pass `--secrets-provider` to select it:

* `infisical` — a dedicated Infisical secrets manager.
* `aws-secretsmanager` — AWS Secrets Manager.
* `custom` — read from environment variables. Inject them before running the command using your secrets tooling; consult its documentation.

## Running the update from a pipeline

The same command can run from a CI/CD pipeline instead of an operator shell: the pipeline installs the sfp CLI and runs `sfp server update` (over SSH with `--ssh-connection` when the runner is not the host), so updates leave an audit trail in the pipeline history. The requirements are the same as the manual path — network access to the host and the registry credentials — provided through your CI platform's runners and secrets; consult its documentation.

## Related pages

{% content-ref url="/pages/kV7AsWP3tmvBIG40eyVP" %}
[Operations](/flxbl/sfp-server/setting-up/setting-up-sfp-server/operations.md)
{% endcontent-ref %}

{% content-ref url="/pages/Sc6jeeHH3MCOsPr5tB7W" %}
[Scaling workers](/flxbl/sfp-server/managing-your-sfp-server/scaling-hatchet-workers.md)
{% endcontent-ref %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.flxbl.io/flxbl/sfp-server/setting-up/updating-sfp-server.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.