Skip to content

ClickHouseIO: Add Decimal(P, S) support for fixed-point columns - #39846

Open
tomnewton wants to merge 2 commits into
apache:masterfrom
tomnewton:clickhouse-decimal-support
Open

ClickHouseIO: Add Decimal(P, S) support for fixed-point columns#39846
tomnewton wants to merge 2 commits into
apache:masterfrom
tomnewton:clickhouse-decimal-support

Conversation

@tomnewton

@tomnewton tomnewton commented Aug 21, 2026

Copy link
Copy Markdown

ClickHouseIO's TableSchema and column-type parser had no notion of ClickHouse's fixed-point Decimal family, so ClickHouseIO.getTableSchema() failed on the DESCRIBE TABLE output of any table containing a Decimal column and pipelines carrying monetary/fixed-point values could not write to ClickHouse at all.

This change adds first-class Decimal(P, S) support to ClickHouseIO:

  • Schema model — new TypeName.DECIMAL; ColumnType carries scale alongside the existing precision (validated: P in [1, 76], S in [0, P]), with a ColumnType.decimal(precision, scale) factory. Bare Decimal is Decimal(10, 0) and Decimal(P) is Decimal(P, 0), matching ClickHouse's defaults.
  • Parser — JavaCC grammar rules for Decimal[(P[, S])] and the width aliases Decimal32(S) / Decimal64(S) / Decimal128(S) / Decimal256(S) (pinned to precisions 9/18/38/76 per ClickHouse's synonym rule; DESCRIBE TABLE canonicalizes the aliases to Decimal(P, S) anyway). Reachable through Nullable(...) and Array(...) via the existing primitive() rule. Lexical, syntactic and range-validation failures all surface as the parser's uniform failed to parse error.
  • Beam field-type mappingDecimal(P, S) maps to FixedPrecisionNumeric.of(P, S) (base type DECIMAL, values are BigDecimal), preserving the declared precision/scale through getEquivalentSchema and giving an early, loud failure at Row construction for values whose digits genuinely exceed the declared precision. This is the same logical type JdbcIO uses for NUMERIC.
  • Writer — serializes the unscaled value as a little-endian signed integer whose width (32/64/128/256 bits) is chosen from the declared precision, delegating to the ClickHouse client's BinaryStreamUtils.writeDecimal. Fractional digits beyond the column scale are truncated toward zero, consistent with ClickHouse's own "excessive digits in a fraction are discarded (not rounded)" semantics. The truncated value is then bounded by the declared precision and rejected with an IllegalArgumentException if out of range: writeDecimal alone only range-checks against the backing storage width, which is wider than the declared type, and ClickHouse's RowBinary reader does not re-check — so without this a value like 100000 would be stored in a Decimal(5, 0) column whose declared maximum is 99999.
  • DefaultsColumnType.parseDefaultExpression handles Decimal literals (DEFAULT 1.23), so getTableSchema no longer throws for tables with defaulted Decimal columns, and writeRow's existing null-substitution works for them.

Tests:

  • TableSchemaTest — parser cases for Decimal(10, 2), Decimal(5), bare Decimal, all four width aliases, Nullable(Decimal(10, 2)), Array(Decimal(38, 10)); uniform failed to parse errors for Decimal(77, 2), Decimal(0), Decimal(9, 10), Decimal(9, -1) and Decimal(abc); factory range validation; schema-mapping tests including nullable; a test pinning that Row construction under the mapped FixedPrecisionNumeric type rejects values exceeding the declared precision (the layer that guards the declared range, since the writer checks only the storage width); parseDefaultExpression for positive and negative literals.
  • ClickHouseWriterTest — byte-level tests locking the wire format for every width bucket: little-endian unscaled Int32, scaling below the column scale, 16 × 0xFF two's-complement sign extension for a negative Decimal128, a 32-byte Decimal256 encoding, truncation-toward-zero of excess fractional digits, storage-width overflow rejection, and null/non-null through writeNullableValue. Declared-precision enforcement is covered by rejecting 100000 in Decimal(5, 0) (which fits the 32-bit storage width) while accepting 99999, rejecting 1000.00 in Decimal(5, 2) while accepting 999.99 (the bound applies to the unscaled integer at the column scale), and confirming the check runs on the truncated value so 999.999 in Decimal(5, 2) still succeeds.
  • ClickHouseIOIT — round-trip integration tests against the ClickHouse test container for Decimal(9, 2), Decimal(18, 4), Decimal(38, 10) and Decimal(76, 20) (including negative values), Nullable(Decimal(10, 2)) with a null row, Array(Decimal(9, 2)), a DEFAULT 2.25 column exercised end-to-end by writing a null row, a getTableSchema round-trip over a table declared with the width aliases (verifying we parse the server's canonicalized DESCRIBE output), and the user-visible truncation of over-scaled values (-1.239-1.23, confirming ClickHouse truncates toward zero rather than rounding, in agreement with the client-side encoding).

fixes #39840


Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

  • Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
  • Update CHANGES.md with noteworthy changes.
  • If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

@tomnewton
tomnewton force-pushed the clickhouse-decimal-support branch from 17c461f to 8a9bece Compare August 21, 2026 20:40
@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @ahmedabu98 for label java.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

BinaryStreamUtils.writeDecimal only range-checks against the backing
storage width (32/64/128/256 bits, selected from the precision), which is
wider than the declared type, and ClickHouse's RowBinary reader does not
re-check. A value like 100000 was therefore written and stored in a
Decimal(5, 0) column whose declared maximum is 99999.

Truncate to the column scale toward zero, as before, then bound the
resulting unscaled integer by the declared precision and throw
IllegalArgumentException when it does not fit. The bound applies to the
unscaled integer at the column scale, so Decimal(5, 2) admits values up
to 999.99, and the check runs after truncation so 999.999 remains valid.

Removes the integration test that pinned the previous lenient behavior;
the rejection is now covered by unit tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature Request]: ClickHouseIO: Add Decimal(P, S) support for fixed-point columns

1 participant