feat(mongodb): alert when compaction is needed

[delthas]

Follow-up to ZENKO-5285 (PR #2431), which bundled two alerts in its description and only shipped the first (createIndexes-failed). This adds the second — a fragmentation / compaction-needed signal — that @DarkIsDude explicitly asked be filed as a follow-up.

Two commits, deliberately split

1. mongodb: enable dbstatsfreestorage collector in exporter — one-line values.yaml change adding --collector.dbstatsfreestorage to metrics.extraArgs. The chart's exporter no longer uses --collect-all (dropped in 8414833 for ZENKO-5281), so each sub-collector is opted into individually in extraArgs (already lists dbstats, diagnosticdata, replicasetstatus, topmetrics). Without dbstatsfreestorage, freeStorageSize / totalFreeStorageSize only appear as part of the clunky per-host mongodb_dbstats_raw_<host>_* expansion instead of as top-level mongodb_dbstats_* series.

2. mongodb: alert when compaction is needed — the alert proper.

The alert

mongodb_dbstats_totalFreeStorageSize > 0.3 * mongodb_dbstats_fsTotalSize
for: 1h

Per-(pod, database) granularity. Severity warning. Threshold exposed as the compactionFreeStorageRatioThreshold x-input (default 0.3).

Expressing the threshold as a fraction of filesystem capacity (not of the DB's own storage) lets it scale across cluster sizes:

• 100 GB FS → fires around 30 GB reclaimable
• 1 TB FS → fires around 300 GB reclaimable
• 10 TB FS → fires around 3 TB reclaimable

Heuristic notes

A previous draft combined three legs (FS-pressure + fragmentation ratio + absolute floor). @francoisferrand's review pointed out that the fragmentation-ratio leg added noise without much signal and the FS-pressure leg could mask real waste on under-used disks. The single fs-scaled threshold above folds both concerns into one condition.

Per-DB granularity means the alert tells you pod={{ "{{ $labels.pod }}" }} and database={{ "{{ $labels.database }}" }} but does not identify the specific collection. To find that, an operator runs collStats on the alerting DB. The exporter has a --collector.collstats flag that could expose per-collection visibility, but at Artesca-scale cardinality (thousands of buckets per DB) that's expensive — deferred.

Safety against missing / zero values

• Missing metric for a (pod, database) tuple → PromQL produces empty → no alert.
• totalFreeStorageSize = 0 (no fragmentation) → 0 > 0.3 * fsTotalSize is always false.
• fsTotalSize = 0 would degenerate (0.3 * 0 = 0, then any positive freeStorage would fire), but it's physically impossible for an attached PVC.

Follow-up (not in this PR)

Matching compactionFreeStorageRatioThreshold config option in ZKOP, to expose the knob to ops. Not filed as a ticket yet; will track when this lands.

Related

• ZENKO-5293 — this ticket
• ZENKO-5285 — parent alert ticket; createIndexes-failed half shipped in feat(mongodb): alert on createIndexes command failures #2431
• BB-784 — Backbeat-side LifecycleIndexCreationFailed (feat(lifecycle): alert when index creation fails backbeat#2753)
• BB-783 — putBucketIndexesFailed metric (feat(lifecycle): emit putBucketIndexesFailed metric on index creation errors backbeat#2751)
• Parent epic ARTESCA-16740 — Empty bucket feature

Issue: ZENKO-5293

[bert-e]

Hello delthas,

My role is to assist you with the merge of this
pull request. Please type @bert-e help to get information
on this process, or consult the user documentation.

Available options

name	description	privileged	authored
/after_pull_request	Wait for the given pull request id to be merged before continuing with the current one.
/bypass_author_approval	Bypass the pull request author's approval	⭐
/bypass_build_status	Bypass the build and test status	⭐
/bypass_commit_size	Bypass the check on the size of the changeset TBA	⭐
/bypass_incompatible_branch	Bypass the check on the source branch prefix	⭐
/bypass_jira_check	Bypass the Jira issue check	⭐
/bypass_peer_approval	Bypass the pull request peers' approval	⭐
/bypass_leader_approval	Bypass the pull request leaders' approval	⭐
/approve	Instruct Bert-E that the author has approved the pull request.		✍️
/create_pull_requests	Allow the creation of integration pull requests.
/create_integration_branches	Allow the creation of integration branches.
/no_octopus	Prevent Wall-E from doing any octopus merge and use multiple consecutive merge instead
/unanimity	Change review acceptance criteria from one reviewer at least to all reviewers
/wait	Instruct Bert-E not to run until further notice.

Available commands

name	description	privileged
/help	Print Bert-E's manual in the pull request.
/status	Print Bert-E's current status in the pull request TBA
/clear	Remove all comments from Bert-E from the history TBA
/retry	Re-start a fresh build TBA
/build	Re-start a fresh build TBA
/force_reset	Delete integration branches & pull requests, and restart merge process from the beginning.
/reset	Try to remove integration branches unless there are commits on them which do not appear on the source branch.

Status report is not available.

[bert-e]

Waiting for approval

The following approvals are needed before I can proceed with the merge:

• the author

• 2 peers

[delthas]

@francoisferrand Heuristics are up for discussion -- I chose a mix of:

• Disk is being significantly used (to avoid saving 10GB on a disk that has 10TB free space)
• There is a signficiant number of GBs to save (to avoid alarms on tiny collections)
• There is a significant ratio of data to save (to avoid running compact just to save 10GB on a 10TB DB)

[francoisferrand]

• not sure about thresholds/computation: are these relevant default?
• not sure we should merge in 2.15, or target 2.16 and get time to "preview" this and avoid shipping alerts which would lead to support calls...

[delthas]

Keeping on 2.15 for now as 2.16 doesnt exist.

[bert-e]

Waiting for approval

The following approvals are needed before I can proceed with the merge:

• the author

• 2 peers

The following reviewers are expecting changes from the author, or must review again:

• @francoisferrand

[delthas]

Intentionally set 30% by default (conservative), up for discussion

[bert-e]

Waiting for approval

The following approvals are needed before I can proceed with the merge:

• the author

• 2 peers