repr: reject corrupt proto shapes and out-of-domain type modifiers when decoding - #37975
Open
def- wants to merge 4 commits into
Open
repr: reject corrupt proto shapes and out-of-domain type modifiers when decoding#37975def- wants to merge 4 commits into
def- wants to merge 4 commits into
Conversation
def-
marked this pull request as ready for review
July 31, 2026 09:58
`CharLength`, `VarCharMaxLength`, `NumericMaxScale` and `TimestampPrecision` each have a `TryFrom<i64>` that rejects out-of-domain values, but their `from_proto` copied the wire value straight through. `from_proto` is a trust boundary for durable catalog state and for the compute/storage protocol, so a corrupt or forged value became a type modifier no SQL statement could have produced. Consumers rely on the invariant being real: a `TimestampPrecision` above `MAX_PRECISION` panics `round_to_precision`. Route all four through `TryFrom<i64>` and surface the rejection as an `InvalidFieldError`. Narrow the `Arbitrary` impls to the same domains, which the validation above makes load-bearing. `CharLength` and `VarCharMaxLength` mapped an arbitrary `u32` through `% 300`, so they generated 0, and there is no `char(0)` or `varchar(0)`. `NumericMaxScale` and `TimestampPrecision` used `derive(Arbitrary)`, which knows nothing of `NUMERIC_DATUM_MAX_PRECISION` or `MAX_PRECISION`. Hand-written strategies keep the generated values inside the domain the type promises, so a round-trip property cannot fail on a value the constructor would have refused to build. `source-datas.txt` holds such a value, generated before the narrowing: 10 of its 40 SourceData stability entries carry a `numeric` max scale or a `timestamp` precision outside the domain, so `from_proto` now refuses to decode them. Those 10 are regenerated from the narrowed strategies. The other 30 stay byte-identical, and the encoding itself does not change, so `MINIMUM_CONSOLIDATED_VERSION` does not move.
`VersionedRelationDesc::validate` treats a names/column-types length mismatch and an out-of-bounds key column index as corruption, but `RelationDesc::from_proto` accepted both. Nothing downstream catches them: such a desc decodes and re-encodes cleanly and only panics at first use, `iter()` indexing `typ.columns()[typ_idx]` out of bounds or `into_iter()` tripping `zip_eq`. Both are reachable from untrusted proto bytes. Also record why only dense descs may be handed to `Codec::encode_schema`. `ProtoRelationDesc` has no field for the `ColumnIndex` keys, only the values in `ColumnIndex` order, so a sparse desc, which is what `VersionedRelationDesc::at_version` returns once a column has been dropped and what `RelationDesc::apply_demand` returns, comes back from `from_proto` renumbered to `0..n`. `ColumnIndex::to_stable_name` is the arrow field name that `RowColumnarEncoder` and `RowColumnarDecoder` agree on, so encoding a sparse desc as the schema of data written under the original indexes builds a decoder that looks up the wrong fields.
`VersionedRelationDesc::validate` derived each column's versions with `dropped.unwrap_or(added)`, which discards the add's version whenever a column was also dropped. Every version past the root is stamped exactly once, by the add or the drop that created it, so a column added after the root and later dropped accounts for two of them and both have to be counted. Collapsing them lost any desc where a non-root column was later dropped. Count both, filtering the root, which is the one version many columns can share.
def-
force-pushed
the
def/fuzz-05-repr-proto
branch
from
July 31, 2026 10:07
3385134 to
bbf4254
Compare
`decPackedToNumber` writes one BCD digit per input nibble into the coefficient of the `decNumber` it is handed, without checking that the coefficient can hold them. `Numeric` is a `Decimal<13>`, whose coefficient is a fixed `[u16; 13]` holding 39 digits, so a `ProtoNumeric` whose `bcd` exceeds 20 bytes overruns a struct on our stack. At 21 bytes that silently corrupts the value, yielding a `Datum::Numeric` holding `Infinity`, which the encoder treats as impossible for this datum type. A few KB of `bcd` walks off the frame and segfaults. `ProtoRow` decoding is reachable from untrusted bytes via `Codec for Row` and `Codec for SourceData`, so a corrupted or tampered persist blob turns what should be a decode error into memory corruption in environmentd or clusterd. The empty-bcd segfault was already rejected here, so bound the upper end next to it. `to_packed_bcd` never emits more than 20 bytes, so nothing we write is rejected. Adds `proto_row_oversized_numeric_bcd_is_error`, which covers the silent corruption and the segfault, and asserts a max-precision numeric still round-trips so the bound cannot drift below what we encode. Found by the row_proto_roundtrip cargo-fuzz target. That target cannot detect this on its own: the corrupted value re-encodes and re-decodes to itself, so its round-trip oracle stays silent and only the multi-KB segfault shows up.
def-
force-pushed
the
def/fuzz-05-repr-proto
branch
from
July 31, 2026 10:18
bbf4254 to
eba595f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four fixes to
mz-repr's proto decoding.from_protois a trust boundary fordurable catalog state and for the compute/storage protocol, so every one of
these is reachable from a corrupt or forged message, and each produced a value
that no SQL statement could have built.
Type-modifier domains were not enforced.
CharLength,VarCharMaxLength,NumericMaxScaleandTimestampPrecisioneach have aTryFrom<i64>that rejects out-of-domain values, but theirfrom_protocopied the wire value straight through. Consumers rely on the invariant being
real, since a
TimestampPrecisionaboveMAX_PRECISIONpanicsround_to_precision. All four now route throughTryFrom<i64>and surfacethe rejection as an
InvalidFieldError.The
Arbitraryimpls are narrowed to the same domains, which the validationmakes load-bearing.
CharLengthandVarCharMaxLengthmapped an arbitraryu32through% 300, so they generated 0, and there is nochar(0)orvarchar(0).NumericMaxScaleandTimestampPrecisionusedderive(Arbitrary), which knows nothing ofNUMERIC_DATUM_MAX_PRECISIONorMAX_PRECISION.Corrupt
ProtoRelationDescshapes were accepted.VersionedRelationDesc::validatetreats a names/column-types length mismatchand an out-of-bounds key column index as corruption, but
RelationDesc::from_protoaccepted both, and nothing downstream catches them:such a desc decodes and re-encodes cleanly and only panics at first use,
iter()indexingtyp.columns()[typ_idx]out of bounds orinto_iter()tripping
zip_eq.This commit also records why only dense descs may be handed to
Codec::encode_schema:ProtoRelationDeschas no field for theColumnIndexkeys, only the values inColumnIndexorder, so a sparse desccomes back from
from_protorenumbered to0..n. SinceColumnIndex::to_stable_nameis the arrow field name thatRowColumnarEncoderandRowColumnarDecoderagree on, encoding a sparse descas the schema of data written under the original indexes builds a decoder that
looks up the wrong fields.
validatemiscounted versions for a dropped non-root column. It derivedeach column's versions with
dropped.unwrap_or(added), discarding the add'sversion whenever a column was also dropped. Every version past the root is
stamped exactly once, so a column added after the root and later dropped
accounts for two and both must be counted. Collapsing them rejected any desc
where a non-root column was later dropped.
ProtoNumeric.bcdwas unbounded before the BCD FFI.decPackedToNumberwrites one BCD digit per input nibble into the coefficientof the
decNumberit is handed, without checking the coefficient can holdthem.
Tests
Each fix comes with a red test that fails without it. The
relation_desc_proto_roundtrip,scalar_type_proto_roundtripandcolumn_type_proto_roundtripcargo-fuzz targets cover this area and aresharpened in #37979, which should land after this.
Note for reviewers:
src/storage-types/src/snapshots/source-datas.txt, theSourceData serialization-stability corpus, was itself generated by the old wide
strategies. 10 of its 40 entries carry a
numericmax scale ortimestampprecision outside the domain, so
from_protonow refuses them. Those 10 areregenerated from the narrowed strategies, and the other 30 are byte-identical.
The encoding is untouched, so
MINIMUM_CONSOLIDATED_VERSIONdoes not move.