Monitoring for sfp server

The server stack ships an optional OpenTelemetry Collector that gathers the metrics and logs of every container in the tenant's compose project and exports them over OTLP HTTP to an observability backend. It runs behind a compose profile, so it is off until you enable it.

Enabling the collector

1. Set the export target and, if the backend needs one, an auth value. These must be present in the environment that runs the start command — export them in the shell (or provide them through your secrets provider); a value that exists only in the tenant's .env does not enable the profile:

   export OTEL_EXPORTER_OTLP_ENDPOINT=<your backend's OTLP HTTP endpoint>
   export OTEL_AUTH_API_KEY=<auth value for the configured header>

2. Restart the tenant (sfp server stop then start). The start command enables the monitoring profile automatically when OTEL_EXPORTER_OTLP_ENDPOINT is set for the tenant.

3. If your backend authenticates with a header other than api-key, change the header name in the config file (see Changing the auth header).

The collector config is mounted from ./config/otel-collector-config.yaml and can be edited per deployment. sfp server update rewrites this file from the bundled template — reapply local edits after an update.

* Not required for backends that do not need authentication.

What gets collected

The collector runs two pipelines. Log collection discovers the containers of this tenant's compose project; the metrics pipeline filters out containers that carry another project's compose label.

Container metrics (docker_stats receiver, every 15 seconds):

• CPU usage per container, including per-CPU
• Memory usage and limits
• Network I/O — bytes and packets, in and out
• Block I/O — reads and writes

Container logs (filelog receivers, discovered per container): each container's stdout/stderr is tailed from its Docker json-file log, JSON-parsed, and forwarded with the container name and id attached.

Everything is tagged with:

• sfp.tenant — the tenant name (TENANT_NAME)
• deployment.environment — the environment (NODE_ENV)
• container.name and container.id
• Host and OS metadata on the metrics pipeline (auto-detected)

Host monitoring

The collector above reports container metrics and logs only; the stack does not collect host-level metrics (disk, filesystem, CPU, memory, load). Monitor the host with your observability provider's host agent, pointed at the same backend — consult their documentation. Host disk usage is the one to watch closest: the databases and container images grow over time, and a full disk stops the stack.

Backend configuration

The collector exports over standard OTLP HTTP. Any backend that accepts OTLP HTTP works — the product surface is two values and one configurable header:

The endpoint URL, the header name, and the auth-value format are defined by your observability provider — consult their OTLP ingest documentation. For a backend that needs no authentication, remove the headers section from the exporter config.

Both pipelines — metrics and logs — export through the single OTLP exporter to one endpoint. If your metrics and logs go to different targets, add a second exporter in otel-collector-config.yaml and wire each pipeline to its own exporter.

Changing the auth header

The default config sends auth via the api-key header. To use a different header name, edit the headers section in ./config/otel-collector-config.yaml:

exporters:
  otlp_http:
    endpoint: ${OTEL_EXPORTER_OTLP_ENDPOINT}
    headers:
      api-key: ${OTEL_AUTH_API_KEY}        # <-- change this key

What to alert on

The server exposes a health endpoint at GET /sfp/api that returns an overall status (healthy, degraded, or unhealthy) and a per-component breakdown. Alert on it directly, and back it with the metric and log signals from the collector.

curl -s https://<your-domain>/sfp/api | jq

The response reports:

• status — the overall health, driven only by the two critical components: the database and flows (the Hatchet engine). The other components do not affect it, so alert on status != healthy and on the per-component statuses below.
• components — the status and latency of api, database, metrics (the metrics store), logs (the logs store), flows (Hatchet), and registry (the package registry). Alert on any component reporting unhealthy, and watch the per-component latency for the ones trending up.
• activeWorkflows — { running, queued, total }. A queued count that stays high is the signal that workers are saturated; see Scaling workers.

From the collector's own signals, the alerts worth setting are:

• Container down or restarting — a container in the tenant's project that stops reporting metrics, or whose logs show a restart loop.
• Memory near the limit — container memory usage approaching the container's memory limit, which precedes an out-of-memory kill.
• Error-log rate — a rising rate of error-level log lines from the server or worker containers.

Troubleshooting

Collector not starting

• Check docker logs <project>-otel-collector-1 for config errors.
• An empty TENANT_NAME or NODE_ENV makes the resource processor fail; the compose file provides defaults (unknown / production).

401 / 403 errors

• Verify the auth value is correct.
• Verify the header name matches your backend (see the examples above).
• Check you are using the right regional endpoint.

No data in your backend

• Metrics are batched roughly every 10 seconds; allow half a minute after startup.
• Verify OTEL_EXPORTER_OTLP_ENDPOINT is set for the tenant and the stack was restarted after setting it.

Related pages