> For the complete documentation index, see [llms.txt](https://docs.flxbl.io/flxbl/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.flxbl.io/flxbl/sfp-server/managing-your-sfp-server/monitoring-for-sfp-server.md).

# Monitoring for sfp server

The server stack ships an optional OpenTelemetry Collector that gathers the metrics and logs of every container in the tenant's compose project and exports them over OTLP HTTP to an observability backend. It runs behind a compose profile, so it is off until you enable it.

## Enabling the collector

1. Set the export target and, if the backend needs one, an auth value. These must be present in the environment that runs the start command — export them in the shell (or provide them through your secrets provider); a value that exists only in the tenant's `.env` does not enable the profile:

   ```bash
   export OTEL_EXPORTER_OTLP_ENDPOINT=<your backend's OTLP HTTP endpoint>
   export OTEL_AUTH_API_KEY=<auth value for the configured header>
   ```
2. Restart the tenant (`sfp server stop` then `start`). The start command enables the monitoring profile automatically when `OTEL_EXPORTER_OTLP_ENDPOINT` is set for the tenant.
3. If your backend authenticates with a header other than `api-key`, change the header name in the config file (see [Changing the auth header](#changing-the-auth-header)).

The collector config is mounted from `./config/otel-collector-config.yaml` and can be edited per deployment. `sfp server update` rewrites this file from the bundled template — reapply local edits after an update.

| Variable                      | Required | Description                              |
| ----------------------------- | -------- | ---------------------------------------- |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | Yes      | OTLP HTTP endpoint of your backend       |
| `OTEL_AUTH_API_KEY`           | Yes\*    | Auth value sent in the configured header |

\* Not required for backends that do not need authentication.

## What gets collected

The collector runs two pipelines. Log collection discovers the containers of this tenant's compose project; the metrics pipeline filters out containers that carry another project's compose label.

**Container metrics** (`docker_stats` receiver, every 15 seconds):

* CPU usage per container, including per-CPU
* Memory usage and limits
* Network I/O — bytes and packets, in and out
* Block I/O — reads and writes

**Container logs** (`filelog` receivers, discovered per container): each container's stdout/stderr is tailed from its Docker json-file log, JSON-parsed, and forwarded with the container name and id attached.

Everything is tagged with:

* `sfp.tenant` — the tenant name (`TENANT_NAME`)
* `deployment.environment` — the environment (`NODE_ENV`)
* `container.name` and `container.id`
* Host and OS metadata on the metrics pipeline (auto-detected)

## Host monitoring

The collector above reports **container** metrics and logs only; the stack does not collect host-level metrics (disk, filesystem, CPU, memory, load). Monitor the host with your observability provider's host agent, pointed at the same backend — consult their documentation. Host disk usage is the one to watch closest: the databases and container images grow over time, and a full disk stops the stack.

## Backend configuration

The collector exports over standard OTLP HTTP. Any backend that accepts OTLP HTTP works — the product surface is two values and one configurable header:

| Variable                      | Description                                                                    |
| ----------------------------- | ------------------------------------------------------------------------------ |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | your backend's OTLP HTTP endpoint                                              |
| `OTEL_AUTH_API_KEY`           | the auth value, sent in a single configurable header (default name: `api-key`) |

The endpoint URL, the header name, and the auth-value format are defined by your observability provider — consult their OTLP ingest documentation. For a backend that needs no authentication, remove the `headers` section from the exporter config.

Both pipelines — metrics and logs — export through the single OTLP exporter to one endpoint. If your metrics and logs go to different targets, add a second exporter in `otel-collector-config.yaml` and wire each pipeline to its own exporter.

## Changing the auth header

The default config sends auth via the `api-key` header. To use a different header name, edit the `headers` section in `./config/otel-collector-config.yaml`:

```yaml
exporters:
  otlp_http:
    endpoint: ${OTEL_EXPORTER_OTLP_ENDPOINT}
    headers:
      api-key: ${OTEL_AUTH_API_KEY}        # <-- change this key
```

## What to alert on

The server exposes a health endpoint at `GET /sfp/api` that returns an overall status (`healthy`, `degraded`, or `unhealthy`) and a per-component breakdown. Alert on it directly, and back it with the metric and log signals from the collector.

```bash
curl -s https://<your-domain>/sfp/api | jq
```

The response reports:

* **`status`** — the overall health, driven only by the two critical components: the **database** and **flows** (the Hatchet engine). The other components do not affect it, so alert on `status != healthy` **and** on the per-component statuses below.
* **`components`** — the status and latency of `api`, `database`, `metrics` (the metrics store), `logs` (the logs store), `flows` (Hatchet), and `registry` (the package registry). Alert on any component reporting `unhealthy`, and watch the per-component latency for the ones trending up.
* **`activeWorkflows`** — `{ running, queued, total }`. A `queued` count that stays high is the signal that workers are saturated; see [Scaling workers](/flxbl/sfp-server/managing-your-sfp-server/scaling-hatchet-workers.md).

From the collector's own signals, the alerts worth setting are:

* **Container down or restarting** — a container in the tenant's project that stops reporting metrics, or whose logs show a restart loop.
* **Memory near the limit** — container memory usage approaching the container's memory limit, which precedes an out-of-memory kill.
* **Error-log rate** — a rising rate of error-level log lines from the server or worker containers.

## Troubleshooting

**Collector not starting**

* Check `docker logs <project>-otel-collector-1` for config errors.
* An empty `TENANT_NAME` or `NODE_ENV` makes the `resource` processor fail; the compose file provides defaults (`unknown` / `production`).

**401 / 403 errors**

* Verify the auth value is correct.
* Verify the header name matches your backend (see the examples above).
* Check you are using the right regional endpoint.

**No data in your backend**

* Metrics are batched roughly every 10 seconds; allow half a minute after startup.
* Verify `OTEL_EXPORTER_OTLP_ENDPOINT` is set for the tenant and the stack was restarted after setting it.

## Related pages

{% content-ref url="/pages/Sc6jeeHH3MCOsPr5tB7W" %}
[Scaling workers](/flxbl/sfp-server/managing-your-sfp-server/scaling-hatchet-workers.md)
{% endcontent-ref %}

{% content-ref url="/pages/lpYE9lvTzdDli7pWipwk" %}
[Updating sfp server](/flxbl/sfp-server/setting-up/updating-sfp-server.md)
{% endcontent-ref %}

{% content-ref url="/pages/t3uaVfN2OqoJPNeks5Xi" %}
[Troubleshooting common failures](/flxbl/sfp-server/managing-your-sfp-server/troubleshooting-common-failures.md)
{% endcontent-ref %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.flxbl.io/flxbl/sfp-server/managing-your-sfp-server/monitoring-for-sfp-server.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.