[Hubs] Fix ingestion OOM errors by eliminating strcat() in emptiness checks#2151
[Hubs] Fix ingestion OOM errors by eliminating strcat() in emptiness checks#2151Copilot wants to merge 7 commits into
Conversation
Agent-Logs-Url: https://github.com/microsoft/finops-toolkit/sessions/ab35dd4d-dd87-400b-8550-60c4a73c71e3 Co-authored-by: flanakin <399533+flanakin@users.noreply.github.com>
|
@copilot Were there any other places where this type of change is needed throughout the repo? |
There was a problem hiding this comment.
Pull request overview
This PR updates the FinOps hub ingestion KQL transforms to reduce memory pressure during Data Factory ingestion by removing a strcat()-based “emptiness guard” and replacing it with direct isnotempty() checks on the individual key fields used for price lookups.
Changes:
- Replaced
isnotempty(strcat(x_SkuMeterId, x_SkuOfferId))withisnotempty(x_SkuMeterId) and isnotempty(x_SkuOfferId)in the “populate missing prices” filter forCosts_transform_v1_0. - Replaced the same
strcat()-based guard with twoisnotempty()predicates inCosts_transform_v1_2. - Tightened the filter semantics to require both lookup-key components to be present (aligning with the lookup key construction).
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/scripts/IngestionSetup_v1_2.kql | Removes strcat() from an emptiness check in the missing-price population filter to reduce allocations and require both key fields. |
| src/templates/finops-hub/modules/Microsoft.FinOpsHubs/Analytics/scripts/IngestionSetup_v1_0.kql | Applies the same change as v1_2 to avoid unnecessary string allocation and enforce presence of both lookup fields. |
Yes — the same All other |
I re-checked the repo: the only remaining
Added in |
| - Fixed Data Factory ingestion memory pressure during emptiness filtering. | ||
| - Replaced `isnotempty(strcat(x_SkuMeterId, x_SkuOfferId))` with separate `isnotempty()` checks in FinOps hub ingestion scripts to avoid temporary string allocation. |
There was a problem hiding this comment.
Updated in 28d925d — the ingestion memory-pressure changelog item is now under the FinOps hubs v15 section and removed from v14.
Costs_transform_v1_0andCosts_transform_v1_2were hittingE_LOW_MEMORY_CONDITIONduring Data Factory pipeline ingestion due to unnecessary string allocation from usingstrcat()purely as an emptiness guard.Changes
IngestionSetup_v1_0.kql/IngestionSetup_v1_2.kql: Replace the singlestrcat()-based check with individualisnotempty()predicates in the "populate missing prices" filter:The old form is semantically looser (true if either field is non-empty), while the price lookup key
tmp_ReservationPriceLookupKeyrequires both — so the corrected form is also more precise. All otherstrcat()usages in these files are legitimate string construction calls and were left unchanged.docs-mslearn/toolkit/changelog.md: Added a v15 FinOps hubs changelog entry documenting the ingestion memory-pressure fix.