Add jsonExtractScalar scalar function for JSON_EXTRACT_SCALAR#18979
Open
Vamsi-klu wants to merge 1 commit into
Open
Add jsonExtractScalar scalar function for JSON_EXTRACT_SCALAR#18979Vamsi-klu wants to merge 1 commit into
Vamsi-klu wants to merge 1 commit into
Conversation
json_extract_scalar was only implemented as a transform function, so it could not be resolved in the multi-stage query engine or in ad-hoc scalar contexts. Register it as a scalar function with full result-type parity to the transform: INT/LONG/FLOAT/DOUBLE/BIG_DECIMAL/BOOLEAN/TIMESTAMP/STRING/JSON/BYTES single values plus INT/LONG/FLOAT/DOUBLE/BIG_DECIMAL/STRING arrays, reusing the transform's coercion rules (BooleanUtils numeric convention, TimestampUtils for ISO-8601 strings, a BigDecimal-preserving parser) and its throw-on-unresolved semantics. A BigDecimal-based long parser stands in for pinot-core's NumberUtils.parseJsonLong, which pinot-common cannot depend on. Supersedes apache#16910. Co-authored-by: Manik Somayaji <somayajimanik@gmail.com>
Author
|
Ready for review. This replaces #16910 and rebases it onto current master. I also went back through the points raised in that PR's review: it now handles all the same result types as the transform function, throws in the same situations when a path can't be resolved, and I added a lot more coverage in JsonFunctionsTest. @Jackie-Jiang you reviewed the original #16910, and @xiangfu0 you've been active in JsonFunctions lately, so a look from either of you would be really helpful whenever you get a chance. Happy to tweak anything. Drafted-by: Claude Code (Opus 4.8); reviewed by @Vamsi-klu before posting |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's the problem
json_extract_scalarexists only as a transform function(
JsonExtractScalarTransformFunction). It was never registered as ascalar function, so it cannot be resolved wherever the query planner needs
a scalar implementation — most visibly the multi-stage (v2) engine's
intermediate stages. A query such as
SELECT CAST(json_extract_scalar(a.col1, b.col2, 'INT') AS INT) FROM a JOIN b ...fails outright with
Unsupported function: JSONEXTRACTSCALAR.Why it matters
The multi-stage engine is Pinot's path for JOINs, sub-queries, and other
intermediate-stage computation. Without a scalar form, one of the most common
JSON-extraction operations is unusable across that whole class of queries, and
in ad-hoc scalar/constant-folding contexts. Issue #16486 tracks exactly this
gap.
What I did
Registered
jsonExtractScalaras a@ScalarFunctioninpinot-common'sJsonFunctions, with full parity to the transform:INT,LONG,FLOAT,DOUBLE,BIG_DECIMAL,BOOLEAN,TIMESTAMP,STRING,JSON,BYTES; arrays forINT/LONG/FLOAT/DOUBLE/BIG_DECIMAL/STRING.BOOLEANuses Pinot's numericconvention and returns stored
INT(0/1);TIMESTAMPtakes numeric epochmillis directly and ISO-8601 strings via
TimestampUtils;BIG_DECIMAL/STRING/JSONread through a BigDecimal-preserving parser.default throws; a multi-value path yields an empty array; a null element in a
resolved array throws unless a default is supplied; malformed JSON is treated
as unresolved.
was unsupported, and extended
JsonFunctionsTestacross every result type(happy / default / throw paths, arrays incl. null elements, malformed JSON,
BigDecimal precision, unsupported type).
Why I did it this way
transform forms are behaviorally identical — no second, subtly-different JSON
semantics for users to reason about.
JsonFunctions: the existingjsonPath*methods(including
master'scanExtractJsonPath()ingestion-hot-path fast-path) areuntouched, so this carries zero regression risk for current callers.
pinot-core'sNumberUtils.parseJsonLong, becausepinot-commoncannot depend onpinot-core; it reproduces the same truncate-toward-zero and exponenthandling for JSON numeric strings.
Impact
json_extract_scalarnow resolves in the multi-stage engine and scalarcontexts, with behavior identical to the transform. No existing function
changes behavior. Supersedes #16910 (revived, rebased on current
master, withthat PR's review gaps on parity, throw semantics, and coverage closed).
closes #16486
Co-authored-by: Manik Somayaji somayajimanik@gmail.com
Generated-by: Claude Code (Opus 4.8)