Skip to content

Observability

Health

  • GET /healthz: liveness. 200 {"status":"ok","version":"…"}. Reports the build version, which makes it a way to see what is actually running.
  • GET /readyz: readiness. 200 only when notify’s NATS connection to dagnats is up and the SQLite writer answers a ping; otherwise 503, with dagnats and db saying which. Both are checked live on each request; the ping has a 3-second timeout.

Use /healthz for a restart decision and /readyz to decide whether to send traffic.

Delivery health

Each integration carries a delivery health state (unverified, healthy, degraded, failing), worked out from its delivery results and from credential checks. A credential check runs right after an integration’s credentials change, and once a day for every active integration in every environment, so a key that was revoked at the provider shows up without waiting for a send to fail. The daily check is a dagnats cron trigger (notify.health-verify-daily) that starts the notify.health-verify-all workflow. It runs at 06:17 UTC by default. NOTIFY_HEALTH_VERIFY_CRON changes the schedule (a standard 5-field cron expression, read in UTC) or, set to off, turns the check off and removes the trigger. notify re-asserts the trigger on every start. It needs a dagnats with program triggers (dagnats#958, not in v0.0.54); on an older dagnats notify logs daily credential check not scheduled once and carries on, and the check starts working at the first start after dagnats is upgraded.

Read an integration’s health from GET /v1/integrations, and re-run its credential check on demand with POST /v1/integrations/{id}/verify. The console shows a health badge and the last error on each integration, and a banner on every page while any active integration is failing; the banner names the integration and links to it. Health is not exported to /metrics or sent as a webhook event yet.

Metrics

GET /metrics returns JSON of notify_-prefixed expvar variables only:

  • notify_delivery_total: delivery outcomes by channel.
  • notify_nats_status, notify_nats_disconnects, notify_nats_reconnects: the NATS link.

The process’s command line and memory statistics are not exposed, because the endpoint is typically public. It is unauthenticated: restrict it at your proxy if you need to.

Logs

notify logs to standard error. By default the format is the standard log package’s. Setting OTEL_EXPORTER_OTLP_ENDPOINT switches to structured slog text with trace_id and span_id on records logged inside a span.

Things worth alerting on:

  • Start-up failures naming the dagnats NATS connection, and /readyz reporting "dagnats": false.
  • emailloop: lines. Every inbound mail that is dropped is logged with the address and the reason (no_token, invalid_token, domain_unverified, no_binding, …). The loop is never silent.
  • retention: lines for purge failures.
  • trigger: digestKey resolved to nothing: a digest that is merging unrelated events.

Tracing

Set OTEL_EXPORTER_OTLP_ENDPOINT (and optionally OTEL_SERVICE_NAME, default notify) to export OpenTelemetry traces. With it unset, nothing is constructed and the OTel globals are untouched. A trigger is traced from the API call through the engine run to delivery.

Activity and run inspection

The console’s Activity view lists every notification and its messages with live tail. With NOTIFY_DAGNATS_CONSOLE_URL set, each run links to the run page in dagnats’ own console.

The analytics event stream

For every delivery event notify also publishes a small event to its NOTIFY_ANALYTICS stream, on notify.analytics.<environment id>. It carries the same Nats-Msg-Id as the delivery event, so a duplicate publish is stored once.

The event holds ids, enums and timestamps only: v (schema version, 1), messageId, orgId, envId, workflowId, notificationId, runId, stepId, transactionId, integrationId, subscriberId, channel, providerId, status, errorClass, createdAt and at. It holds no message content: no subject, body, recipient address or phone number, error text or skip reason. subscriberId is the opaque id your tenant chose. Replies to a conversation produce no analytics event.

The stream has no age limit, only a size cap (1% of the dagnats store budget, at least 64 MiB). The publish is best effort: if it fails, it is logged and the delivery is not affected.

Analytics backend

The analytics projector reads delivery events through a durable consumer, so it resumes where it stopped after downtime or a restart, and a redelivered event is counted once.

By default delivery analytics live in SQLite. A binary built with the duckdb tag stores them in an embedded DuckDB file instead (NOTIFY_DUCKDB, default notify-analytics.duckdb), while all transactional data stays in SQLite. It needs cgo, so the default image and release binaries do not include it:

CGO_ENABLED=1 go build -tags duckdb -o notify ./cmd/notify

Reading analytics from the lake

The metadata-only NOTIFY_ANALYTICS stream is drained by Lode into Parquet parts under <prefix>/notify/stream_notify_analytics/dt=YYYY-MM-DD/. A -tags duckdb binary can answer the analytics API from those parts instead: set NOTIFY_ANALYTICS_LAKE. It is off by default, and SQLite stays the default.

  • NOTIFY_ANALYTICS_LAKE=r2 reads the R2 lake in bucket NOTIFY_LAKE_BUCKET (default lode) under prefix NOTIFY_LAKE_PREFIX (default mesh). It uses NOTIFY_LAKE_R2_ENDPOINT, NOTIFY_LAKE_R2_ACCESS_KEY_ID and NOTIFY_LAKE_R2_SECRET_ACCESS_KEY. DuckDB loads its httpfs extension for this, and downloads it once if it is not installed.
  • Any other value is a local lake root directory.

Only parts with a commit record and no supersede record are read. Each row’s fields come from its JSON payload, and an event id that appears twice (in both the import and the live stream) is counted once. The projector keeps writing SQLite. A binary without the duckdb tag refuses to start with NOTIFY_ANALYTICS_LAKE set.

Importing SQLite analytics into the lake

notify analytics-import copies the analytics_deliveries rows created before the live publisher started into the lake once, as stream notify-import. Run it with -dry-run first: that counts rows per day and writes nothing. Running it again over the same rows writes nothing new.

notify analytics-import -db /path/notify.db -dry-run
notify analytics-import -db /path/notify.db            # R2, from NOTIFY_LAKE_R2_*
notify analytics-import -db /path/notify.db -lake /tmp/lake   # local lake root