Add request analytics — endpoint volume, latency, error rates, top consumers

[SyniRon]

The API currently has no analytics surface. We don't know which endpoints
are hit most, what the latency distribution looks like, what the error rate
is per endpoint, or which consumers drive the bulk of traffic.

Goals:

• Per-endpoint request volume
• Latency percentiles (p50 / p95 / p99)
• Error rates by code
• Top consumers (per API key)

Open: pick the stack. Should match whatever the wider 7Cav tool surface
adopts so we get one observability picture across the stack.

This is load-bearing for the two adjacent tasks "Replace deprecated gRPC
client APIs" and "Drop gRPC entirely" — the gRPC-vs-HTTP traffic split on
the two listener ports is the evidence that decides between patching the
deprecated client APIs and removing gRPC. Confidence is high that nothing
is calling the gRPC port externally, but a week of data should confirm
before acting on either.

Needs a scoping pass.

[SyniRon]

This was generated by AI during triage.

Triage Notes

What we've established so far:

• There is zero in-tree analytics code today. Closest current substitute is the per-request [CACHE] HIT/MISS INFO logging in the cache middleware.
• OpenTelemetry SDK + the net/http otelhttp instrumentation are already present as indirect dependencies via grpc-gateway's runtime. That makes OTel the lowest-friction choice if OTel is what gets picked — promote the deps direct and wire interceptors — but it does not by itself argue for OTel over alternatives.
• This ticket is load-bearing for Replace deprecated grpc.DialContext / WithBlock / WithInsecure in the gateway #94 (replace deprecated gRPC dial APIs) and Spitball: drop gRPC, rewrite as plain net/http + huma JSON service #95 (drop gRPC entirely): the gRPC-vs-HTTP traffic split on :10000 vs :11000 is the evidence that would decide between patching the deprecated client and removing the gRPC surface. Suggest treating Add request analytics — endpoint volume, latency, error rates, top consumers #92 → Replace deprecated grpc.DialContext / WithBlock / WithInsecure in the gateway #94/Spitball: drop gRPC, rewrite as plain net/http + huma JSON service #95 as a sequence.

What we still need from you (@SyniRon):

• Stack choice. OTel + a collector (Tempo / Jaeger / SigNoz / etc.)? Prometheus client + a scrape target? Hosted (Datadog, Grafana Cloud)? The issue body explicitly defers to "whatever the wider 7Cav tool surface adopts" — does that surface have a current direction, or is this the ticket where it gets picked?
• Scope of v1. All four goals at once (volume / latency percentiles / error rate / top consumers), or ship a smaller first slice that's enough to answer the gRPC-removal question for Replace deprecated grpc.DialContext / WithBlock / WithInsecure in the gateway #94/Spitball: drop gRPC, rewrite as plain net/http + huma JSON service #95?
• Per-key attribution. "Top consumers (per API key)" implies retaining the key identity in metric labels. Are we OK with key_id as the cardinality dimension (bounded, ACP-managed), or do we want a coarser grouping (user_id, scope set)?
• Where the data lives + retention. Self-hosted on the existing stack, an external SaaS, the existing forum host? Retention window?
• Auth-error visibility. Should the 401 path show up in the error-rate metric, or be excluded so legit unauthenticated probes don't muddy the signal?

Once those are settled this can move to ready-for-human (or get split — the gRPC-vs-HTTP traffic-split slice could land first to unblock #94/#95, with the richer per-consumer/percentile work as a follow-up).