Add FOCUS 1.4 ingestion (IngestionSetup_v1_4.kql) by flanakin · Pull Request #2126 · microsoft/finops-toolkit

flanakin · 2026-05-06T15:07:27Z

Summary

Renames IngestionSetup_v1_3.kql → IngestionSetup_v1_4.kql, consolidating all FOCUS 1.3 and 1.4 column additions into a single GA release
Adds ContractCommitment_raw table definition to IngestionSetup_RawTables.kql
Updates app.bicep and .build.config to reference v1_4 (removes v1_3)
Removes all "preview" designations — FOCUS 1.4 ships as GA alongside the FOCUS 1.4 specification

Test plan

KQL functions compile without errors
IngestionSetup_v1_4.kql contains transform functions, final tables, and update policies for all datasets: Prices, Costs, CommitmentDiscountUsage, ContractCommitment, Recommendations, Transactions
.build.config Fabric KQL includes IngestionSetup_v1_4.kql (not v1_3)

🤖 Generated with Claude Code

Adds IngestionSetup_v1_3.kql with Costs/Prices/CommitmentDiscountUsage/ Recommendations/Transactions transforms and final tables renamed for FOCUS 1.3. Cost and Usage gains 8 new FOCUS 1.3 columns: AllocatedMethodId/Details/ResourceId/ResourceName/Tags (data-generator split cost allocation), ContractApplied (per-row contract commitment application), ServiceProviderName + HostProviderName (replacing the deprecated ProviderName/PublisherName, with empty-fallback for back compat). Costs_raw now carries the new columns so downstream v1_2 transforms keep working and v1_3 transforms can read them directly. The v1_3 file is wired into the Bicep deployment alongside v1_0 and v1_2. The unversioned Costs() function still aliases to v1_2 — phase 2 adds Costs_v1_3() and retargets Latest. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Consolidates FOCUS 1.3 and 1.4 column additions into a single v1_4 ingestion setup file. Removes the intermediate v1_3 file, adds ContractCommitment_raw table definition, and updates app.bicep and .build.config to reference v1_4. Co-Authored-By: Claude <noreply@anthropic.com>

flanakin

Changes here need to also be applied to the other 3 FOCUS 1.4 PRs

flanakin · 2026-05-20T19:58:13Z

+//======================================================================================================================
+
+// ContractCommitment_raw table -- Create the table if it doesn't exist
+.create-merge table ContractCommitment_raw ( ignore: string )


Shouldn't the base name be plural? This applies to all tables/functions/etc.

Suggested change

.create-merge table ContractCommitment_raw ( ignore: string )

.create-merge table ContractCommitments_raw ( ignore: string )

🤖 [AI][Claude Code] ✅ Implemented

Pluralized across the PR: ContractCommitment_raw → ContractCommitments_raw, ContractCommitment_transform_v1_4 → ContractCommitments_transform_v1_4, ContractCommitment_final_v1_4 → ContractCommitments_final_v1_4, the mapping name, retention/streaming policies, and the section header. FOCUS-spec column names (e.g., ContractCommitmentBenefitCategory) keep their singular prefix since they are spec-defined identifiers. Cascading the same rename to PR #2128 (hub functions) and PR #2135 (plugin docs).

flanakin · 2026-05-20T19:59:41Z

+    x_SourceName:                          string,    // Hubs v1_3+
+    x_SourceProvider:                      string,    // Hubs v1_3+
+    x_SourceType:                          string,    // Hubs v1_3+
+    x_SourceVersion:                       string     // Hubs v1_3+


Suggested change

x_SourceName: string, // Hubs v1_3+

x_SourceProvider: string, // Hubs v1_3+

x_SourceType: string, // Hubs v1_3+

x_SourceVersion: string // Hubs v1_3+

x_SourceName: string, // Hubs v1_4+

x_SourceProvider: string, // Hubs v1_4+

x_SourceType: string, // Hubs v1_4+

x_SourceVersion: string // Hubs v1_4+

🤖 [AI][Claude Code] ✅ Implemented

Updated // Hubs v1_3+ → // Hubs v1_4+ for all four x_Source* columns on ContractCommitments_raw.

flanakin · 2026-05-20T21:19:36Z

+.alter table ContractCommitment_raw ( ignore: string )
+
+// ContractCommitment_raw table -- Redefine all columns
+.alter table ContractCommitment_raw (


Missing new columns in 1.4 (also impacts ingestion function and other PRs for HubSetup):

ContractCommitmentBenefitCategory
ContractCommitmentCreated
ContractCommitmentDiscountPercentage
ContractCommitmentDurationType
ContractCommitmentFulfillmentInterval
ContractCommitmentLastUpdated
ContractCommitmentLifecycleStatus
ContractCommitmentModel
ContractCommitmentOfferCategory
ContractCommitmentPaymentInterval
ContractCommitmentPaymentModel
ContractCommitmentPaymentUpfrontPercentage
ServiceProviderName

🤖 [AI][Claude Code] ✅ Implemented

Added the 12 missing FOCUS 1.4 per-row contract-commitment columns to Costs_raw and Costs_final_v1_4, and projected them in Costs_transform_v1_4 using column_ifexists() for forward compat:

ContractCommitmentBenefitCategory, ContractCommitmentCreated, ContractCommitmentDiscountPercentage, ContractCommitmentDurationType, ContractCommitmentFulfillmentInterval, ContractCommitmentLastUpdated, ContractCommitmentLifecycleStatus, ContractCommitmentModel, ContractCommitmentOfferCategory, ContractCommitmentPaymentInterval, ContractCommitmentPaymentModel, ContractCommitmentPaymentUpfrontPercentage.

ServiceProviderName already exists at line 512, so it was left as-is. PR #2128 (hub functions) will receive the same column additions.

flanakin · 2026-05-20T21:26:00Z

@@ -359,6 +455,11 @@

 // Costs_raw table -- Redefine all columns
 .alter table Costs_raw (


FOCUS 1.4 has a requirement to include ALL native columns. This will significantly bloat the table size for cross-cloud data. Given we don't have a full set of columns for this yet, we need to have a way to ingest any additional columns either by auto-accepting new columns on ingestion or dumping them into a single x_ column. We need to research options and weigh pros/cons. This can happen in a separate PR, but don't resolve this without a link to that PR.

🤖 [AI][Claude Code] 🤔 Needs discussion

Filed a tracking issue to scope the research: #2159 — Research: Handle FOCUS 1.4 native-columns requirement to avoid raw table bloat. The issue captures the three candidate approaches (auto-accept new columns, single x_NativeColumns JSON, hybrid) and explicitly scopes implementation to a separate PR.

This PR ships the FOCUS 1.4 schema with the columns we know today; native-columns handling will follow #2159's resolution. Leaving this thread unresolved per your request until the follow-up PR is linked.

- Pluralize ContractCommitments_raw / _transform_v1_4 / _final_v1_4 / mapping / update policy - Add BillingPeriods_raw + IngestionSetup_v1_4 transforms/finals/policies (FOCUS 1.4 supplemental dataset) - Add InvoiceDetails_raw + IngestionSetup_v1_4 transforms/finals/policies (FOCUS 1.4 supplemental dataset) - Add 12 ContractCommitment* per-row columns + CommitmentProgramEligibilityDetails + InvoiceDetailId to Costs_raw, Costs_final_v1_4, and Costs_transform_v1_4 - Fix x_Source* annotations on ContractCommitments_raw to Hubs v1_4+ - Update Costs_raw ProviderName/PublisherName annotations to FOCUS 0.5-1.3 (removed in 1.4; kept for back compat) - Update Costs_raw Region annotation: deprecated -> removed - File #2159 for the FOCUS 1.4 native-columns architecture research 🤖 Generated with [Claude Code](https://claude.ai/claude-code) Co-Authored-By: Michael Flanakin <flanakin@users.noreply.github.com> Co-Authored-By: Claude <noreply@anthropic.com>

flanakin · 2026-05-26T07:22:37Z

🤖 [AI][Claude Code] PR Update Summary

Addressed: 7 thread(s)

✅ Implemented: 6
🤔 Needs discussion: 1 (Thread 7 — filed as issue Research: Handle FOCUS 1.4 native-columns requirement to avoid raw table bloat #2159)

Key changes:

Pluralized ContractCommitment_* -> ContractCommitments_* across tables, functions, mapping, and update policies (Thread 1)
Updated x_Source* annotations Hubs v1_3+ -> v1_4+ on ContractCommitments_raw (Thread 2)
Updated ProviderName/PublisherName/Region annotations to mark them as removed (Thread 3)
Added BillingPeriods_raw and InvoiceDetails_raw datasets with transforms/finals/policies (Thread 4)
Added 12 ContractCommitment* per-row columns + CommitmentProgramEligibilityDetails to Costs_raw (Thread 5)
Added InvoiceDetailId to Costs_raw (Thread 6)
Filed Research: Handle FOCUS 1.4 native-columns requirement to avoid raw table bloat #2159 for native-columns research (Thread 7)

Cascading next: Same pluralization + new BillingPeriods/InvoiceDetails datasets will be added to PRs #2128 (hub functions), #2135 (plugin docs), and #2136 (tests + changelog).

- Rename ContractCommitment_v1_4() -> ContractCommitments_v1_4() (and update folder label) - Add BillingPeriods_v1_4() and InvoiceDetails_v1_4() hub functions - Add BillingPeriods(), ContractCommitments() (renamed), and InvoiceDetails() unversioned aliases in HubSetup_Latest.kql - Add CommitmentProgramEligibilityDetails + 12 ContractCommitment* + InvoiceDetailId column defaults to Costs_final_v1_0 and Costs_final_v1_2 union arms (back-compat shims) and add them to the project list 🤖 Generated with [Claude Code](https://claude.ai/claude-code) Co-Authored-By: Michael Flanakin <flanakin@users.noreply.github.com> Co-Authored-By: Claude <noreply@anthropic.com>

- Rename ContractCommitment() -> ContractCommitments() in skill references and agent docs - Add BillingPeriods() and InvoiceDetails() to the function list and dedicated agent doc sections - Mention the new FOCUS 1.4 per-row columns (CommitmentProgramEligibilityDetails, 12 ContractCommitment* columns, InvoiceDetailId) - Update output-style guidance to reference all three supplemental datasets 🤖 Generated with [Claude Code](https://claude.ai/claude-code) Co-Authored-By: Michael Flanakin <flanakin@users.noreply.github.com> Co-Authored-By: Claude <noreply@anthropic.com>

Tests: - Pluralize ContractCommitments_raw / _final_v1_4 / _v1_4 assertions - Assert new BillingPeriods_raw + InvoiceDetails_raw datasets exist with their FOCUS 1.4 columns - Assert new BillingPeriods_v1_4(), InvoiceDetails_v1_4(), ContractCommitments_v1_4() hub functions - Assert HubSetup_Latest aliases for BillingPeriods(), ContractCommitments(), InvoiceDetails() - Assert Costs_raw and Costs_final_v1_4 include the 14 new FOCUS 1.4 columns - Assert NO singular ContractCommitment_raw / ContractCommitment_v1_4 / ContractCommitment() exists - Assert Costs_raw keeps removed ProviderName/PublisherName/Region for back compat Changelog: - Expand v15.0.0 entry with the 14 new FOCUS 1.4 columns and the three new supplemental datasets (ContractCommitments, BillingPeriods, InvoiceDetails) - Update plugin entry to list ContractCommitments() / BillingPeriods() / InvoiceDetails() 🤖 Generated with [Claude Code](https://claude.ai/claude-code) Co-Authored-By: Michael Flanakin <flanakin@users.noreply.github.com> Co-Authored-By: Claude <noreply@anthropic.com>

Copilot AI review requested due to automatic review settings May 6, 2026 15:07

flanakin requested review from MSBrett and RolandKrummenacher as code owners May 6, 2026 15:07

microsoft-github-policy-service Bot requested a review from helderpinto May 6, 2026 15:08

microsoft-github-policy-service Bot added the Needs: Review 👀 PR that is ready to be reviewed label May 6, 2026

microsoft-github-policy-service Bot assigned MSBrett May 6, 2026

microsoft-github-policy-service Bot added the Tool: FinOps hubs Data pipeline solution label May 6, 2026

microsoft-github-policy-service Bot assigned helderpinto and RolandKrummenacher May 6, 2026

Copilot started reviewing on behalf of flanakin May 6, 2026 15:08 View session

This was referenced May 6, 2026

Add FOCUS 1.4 hub functions (HubSetup_v1_4.kql) #2128

Draft

FOCUS 1.3 and 1.4 support across the toolkit #2120

Open

flanakin marked this pull request as draft May 6, 2026 23:16

flanakin added this to the v15 milestone May 15, 2026

flanakin changed the title ~~Add FOCUS 1.3 ingestion (IngestionSetup_v1_3.kql)~~ Add FOCUS 1.4 ingestion (IngestionSetup_v1_4.kql) May 17, 2026

This was referenced May 17, 2026

Add ContractCommitment dataset #2130

Closed

Add FOCUS 1.4-preview ingestion (IngestionSetup_v1_4.kql) #2132

Closed

flanakin commented May 26, 2026

View reviewed changes

flanakin mentioned this pull request May 26, 2026

Research: Handle FOCUS 1.4 native-columns requirement to avoid raw table bloat #2159

Open

flanakin mentioned this pull request May 26, 2026

Refresh Claude Code plugin skill files for FOCUS 1.4 #2135

Draft

2 tasks

flanakin mentioned this pull request May 27, 2026

Add FOCUS 1.4 schema tests and changelog entries #2136

Draft

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add FOCUS 1.4 ingestion (IngestionSetup_v1_4.kql)#2126

Add FOCUS 1.4 ingestion (IngestionSetup_v1_4.kql)#2126
flanakin wants to merge 3 commits into
flanakin/focus14from
flanakin/focus14-phase1-ingestion-1.3

flanakin commented May 6, 2026 •

edited

Loading

Uh oh!

flanakin left a comment

Uh oh!

flanakin May 20, 2026

Uh oh!

flanakin May 26, 2026

Uh oh!

flanakin May 20, 2026

Uh oh!

flanakin May 26, 2026

Uh oh!

Uh oh!

Uh oh!

flanakin May 20, 2026

Uh oh!

flanakin May 26, 2026

Uh oh!

Uh oh!

flanakin May 20, 2026

Uh oh!

flanakin May 26, 2026

Uh oh!

flanakin commented May 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

	.create-merge table ContractCommitment_raw ( ignore: string )
	.create-merge table ContractCommitments_raw ( ignore: string )

		@@ -359,6 +455,11 @@

		// Costs_raw table -- Redefine all columns
		.alter table Costs_raw (

Conversation

flanakin commented May 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Uh oh!

flanakin left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

flanakin commented May 26, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

flanakin commented May 6, 2026 •

edited

Loading