Skip to content

feat(metrics): emit conserver.vcons.inflight UpDownCounter#177

Merged
pavanputhra merged 1 commit into
mainfrom
feat/vcons-inflight-updown-counter
May 20, 2026
Merged

feat(metrics): emit conserver.vcons.inflight UpDownCounter#177
pavanputhra merged 1 commit into
mainfrom
feat/vcons-inflight-updown-counter

Conversation

@pavanputhra
Copy link
Copy Markdown
Contributor

Summary

Adds an OpenTelemetry UpDownCounter (`conserver.vcons.inflight`) that tracks how many vCons are currently being processed — i.e. between BLPOP from the ingress list and the end of `_handle_vcon`.

`+1` on entry, `-1` in the `finally`, so every exit path symmetrically decrements:

  • normal chain completion
  • `before_processing` returning falsy (e.g. license check fail / early-return)
  • chain raising an exception (vCon routed to DLQ)

Attribute: `chain.name` only. Deliberately no `vcon.uuid` (cardinality).

Also adds an `add_updown_counter` helper in `common/lib/metrics.py` that mirrors the lazy-init pattern of `increment_counter` / `record_histogram`, so future UpDownCounter use sites can plug in without re-implementing the SDK plumbing.

Why an UpDownCounter (not a counter pair or observable gauge)

  • A counter pair (entry counter + exit counter, subtract) breaks across process restarts because each counter resets independently.
  • An observable gauge sampling local state needs the per-worker in-flight set hoisted to a module-level structure visible to the callback; fragile and racy.
  • `UpDownCounter` is the OTel primitive for exactly this — single instrument, increment/decrement, no fragile state.

Test plan

  • New `common/tests/test_inflight_counter.py`:
    • Helper creates the underlying instrument lazily and reuses it
    • No-op when endpoint unset
    • `_handle_vcon` increments once and decrements once on success
    • Same on `before_processing` early-return (e.g. license fail)
    • Same when the chain raises (DLQ path)
  • Existing tests still pass (39 of the dep-free suite)
  • After deploy: `conserver.vcons.inflight` shows the live count in the metrics backend, grouped by `chain.name`. Sum across pods gives total in-flight.

🤖 Generated with Claude Code

Adds an OpenTelemetry UpDownCounter that tracks how many vCons are
currently being processed — i.e. between BLPOP from the ingress list
and the end of ``_handle_vcon``. +1 on entry, -1 in finally, so every
exit path (success, early-return from ``before_processing``, chain
exception → DLQ) symmetrically decrements.

Attribute: ``chain.name`` only. ``vcon.uuid`` deliberately omitted to
avoid the per-event cardinality problem.

Also adds ``add_updown_counter`` helper to common/lib/metrics.py for
this and any future UpDownCounter use, matching the lazy-init pattern
of the existing ``increment_counter`` / ``record_histogram`` helpers.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@pavanputhra pavanputhra merged commit 70679da into main May 20, 2026
1 check passed
@pavanputhra pavanputhra deleted the feat/vcons-inflight-updown-counter branch May 20, 2026 17:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant