fix(relay): attribute storage metrics per user, not per community#1885
Draft
wpfleger96 wants to merge 2 commits into
Draft
fix(relay): attribute storage metrics per user, not per community#1885wpfleger96 wants to merge 2 commits into
wpfleger96 wants to merge 2 commits into
Conversation
Add logical-attribution storage observability: a partial index on audit_log for media_uploaded actions, per-community byte/upload gauges in the usage poller, and a real-time upload-bytes counter in the media handler. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
Group storage bytes/objects by (community_id, actor_pubkey), matching the PR's stated goal instead of only rolling up per community. Dedupe by (community_id, actor_pubkey, object_id) in a subquery so idempotent re-uploads of the same blob are charged once, not once per audit row. Extend the audit_log partial index to (community_id, actor_pubkey, object_id) — the new dedup key — and fail safe on malformed historical detail->>'size' values (non-numeric, negative, out-of-range) by attributing 0 bytes instead of aborting the whole poller query. Cast the aggregate SUM to bigint explicitly so Postgres returns an i64- compatible value instead of numeric. Rename polled *_uploads gauges/fields to *_objects to keep them distinct from the real-time buzz_media_upload_bytes_total event counter, which is intentionally unchanged. Add buzz_user_storage_bytes/buzz_user_storage_objects gauges labeled by community + pubkey; the previous community-only gauges remain as rollups of the new per-user rows. Co-authored-by: Will Pfleger <pfleger.will@gmail.com> Signed-off-by: Will Pfleger <pfleger.will@gmail.com>
wpfleger96
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add per-user logical-storage observability to the relay so Block can monitor how much media storage each uploader consumes per community, informing the upcoming free-plan storage limits.
These metrics are audit-derived logical upload attributions, not authoritative physical S3 usage and not a source for quota enforcement: audit-log delivery can fail independently of the upload, content may be physically deduplicated on disk, and deletion/GC of stored objects is not represented here.
0019_audit_log_media_index.sql): partial index onaudit_log(community_id, actor_pubkey, object_id) WHERE action = 'media_uploaded', builtCONCURRENTLYoutside a transaction so it doesn't lock writers on a populated relay.object_idis included because it's the dedup key the query relies on; the planner uses this index for anIndex Scan, followed by anIncremental Sort+Uniqueto resolveDISTINCT ON's max-size tiebreaker (not covered by the index).buzz-db/src/usage.rs):storage_byte_counts()returns per-user (community_id,actor_pubkey) totals. A subquery first dedupes to one row per(community_id, actor_pubkey, object_id)— a logical object requires a non-null actor and non-nullobject_id— so a user re-uploading the same blob is charged once, not once per audit-log row.detail->>'size'is historical, untyped JSONB; malformed, negative, or out-of-range values fall back to 0 bytes instead of aborting the whole poller query. The per-userSUMaccumulates innumeric(arbitrary precision) and is clamped to[0, i64::MAX]before the final cast tobigint, so an extreme aggregate degrades to a saturated reading instead of erroring the whole query.buzz-relay/src/main.rs):buzz_user_storage_bytes{community,pubkey}/buzz_user_storage_objects{community,pubkey}per uploader, rolled up intobuzz_community_storage_bytes{community}/buzz_community_storage_objects{community}andbuzz_total_storage_bytes/buzz_total_storage_objectsfleet totals via saturating addition (same overflow-safety rationale as the SQL aggregate). Named*_objectsrather than*_uploadsto stay distinct from the real-timebuzz_media_upload_bytes_totalevent counter, which is unchanged. Per-user cardinality (two custom series percommunity,pubkeypair) is intentional —BUZZ_USAGE_METRICS_PER_COMMUNITY=offremains the emergency kill switch.