feat(observability): uptime, Prometheus /metrics, and OTLP traces across all servers

CLAUDE.md

@@ -714,6 +714,59 @@ for the manual conformance workflow.
 
 ---
 
+## Observability (uptime, metrics, tracing)
+
+All four server binaries (`hfs`, `hts`, `sof-server`, `fhirpath-server`) share a
+uniform observability surface via the `helios-observability` crate
+(`crates/observability`). Each process installs a Prometheus recorder, tracks
+uptime, instruments every request, and can export OTLP traces.
+
+### Endpoints
+
+| Endpoint | Description |
+|----------|-------------|
+| `GET /metrics` | Prometheus exposition format (text). Always available; needs no auth (on HFS it is merged outside the auth layer). |
+| `GET /health` | Existing JSON liveness probe, now enriched with `uptime_seconds` and `started_at` (RFC 3339). |
+
+Exposed metrics (a global `service="<binary>"` label distinguishes processes
+when scraped into one backend):
+
+- `http_requests_total{service,method,route,status}` — request counter.
+- `http_request_duration_seconds{service,method,route,status}` — latency
+  histogram/summary.
+- `uptime_seconds{service}` — process uptime gauge (refreshed on each scrape).
+
+The `route` label uses the **templated** path (e.g. `/{resource_type}/{id}`),
+never raw URIs, to keep cardinality bounded. Tenant is recorded as a *trace span
+attribute* (read from `X-Tenant-ID`), never as a metric label.
+
+### OTLP traces (feature-gated)
+
+OTLP trace export is behind the `otel` cargo feature (kept out of the default
+build because the OTel SDK + tonic stack increases build time):
+
+```bash
+# Build any server with OTLP trace export available
+cargo build --bin hfs --features otel
+
+# Enable at runtime by pointing at a Collector / Jaeger (gRPC OTLP, :4317)
+OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317 \
+  cargo run --bin hfs --features otel
+```
+
+Export is inert unless built with `--features otel` **and**
+`OTEL_EXPORTER_OTLP_ENDPOINT` is set. Standard OTel env vars apply
+(`OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_SERVICE_NAME`). The `/metrics` endpoint
+and uptime need no configuration.
+
+**OTLP metrics** are produced out-of-process: point an OpenTelemetry Collector's
+`prometheus` receiver at `/metrics` and forward OTLP. We intentionally do not use
+`opentelemetry-prometheus` (unmaintained; carries a RUSTSEC advisory via
+`protobuf`) — `/metrics` uses the maintained `metrics` +
+`metrics-exporter-prometheus` stack instead.
+
+---
+
 ## Docker
 
 Generic Dockerfile supporting all server binaries via `BINARY_NAME` build arg:

Cargo.toml

@@ -22,6 +22,7 @@ default-members = [
 	"crates/auth",
 	"crates/audit",
 	"crates/subscriptions",
+	"crates/observability",
 ]
 resolver = "2"
 

crates/fhirpath/Cargo.toml

@@ -17,9 +17,13 @@ R4B = ["helios-fhir/R4B"]
 R5 = ["helios-fhir/R5"]
 R6 = ["helios-fhir/R6"]
 
+# OpenTelemetry OTLP trace export (heavy deps; opt-in for production images)
+otel = ["helios-observability/otel"]
+
 [dependencies]
 helios-fhir = { path = "../fhir", version = "0.1.47", default-features = false }
 helios-fhirpath-support = { path = "../fhirpath-support", version = "0.1.47" }
+helios-observability = { path = "../observability", version = "0.1.47" }
 chumsky = "0.10"
 roxmltree = "0.20"
 chrono = { workspace = true } # For date/time parsing and functions