Skip to content

Add jsonExtractScalar scalar function for JSON_EXTRACT_SCALAR#18979

Open
Vamsi-klu wants to merge 1 commit into
apache:masterfrom
Vamsi-klu:feature/16486-json-extract-scalar-function
Open

Add jsonExtractScalar scalar function for JSON_EXTRACT_SCALAR#18979
Vamsi-klu wants to merge 1 commit into
apache:masterfrom
Vamsi-klu:feature/16486-json-extract-scalar-function

Conversation

@Vamsi-klu

@Vamsi-klu Vamsi-klu commented Jul 11, 2026

Copy link
Copy Markdown

What's the problem

json_extract_scalar exists only as a transform function
(JsonExtractScalarTransformFunction). It was never registered as a
scalar function, so it cannot be resolved wherever the query planner needs
a scalar implementation — most visibly the multi-stage (v2) engine's
intermediate stages. A query such as
SELECT CAST(json_extract_scalar(a.col1, b.col2, 'INT') AS INT) FROM a JOIN b ...
fails outright with Unsupported function: JSONEXTRACTSCALAR.

Why it matters

The multi-stage engine is Pinot's path for JOINs, sub-queries, and other
intermediate-stage computation. Without a scalar form, one of the most common
JSON-extraction operations is unusable across that whole class of queries, and
in ad-hoc scalar/constant-folding contexts. Issue #16486 tracks exactly this
gap.

What I did

Registered jsonExtractScalar as a @ScalarFunction in pinot-common's
JsonFunctions, with full parity to the transform:

  • Single values: INT, LONG, FLOAT, DOUBLE, BIG_DECIMAL, BOOLEAN,
    TIMESTAMP, STRING, JSON, BYTES; arrays for
    INT/LONG/FLOAT/DOUBLE/BIG_DECIMAL/STRING.
  • Same coercion rules as the transform: BOOLEAN uses Pinot's numeric
    convention and returns stored INT (0/1); TIMESTAMP takes numeric epoch
    millis directly and ISO-8601 strings via TimestampUtils;
    BIG_DECIMAL/STRING/JSON read through a BigDecimal-preserving parser.
  • Same throw-on-unresolved semantics: an unresolved single-value path without a
    default throws; a multi-value path yields an empty array; a null element in a
    resolved array throws unless a default is supplied; malformed JSON is treated
    as unresolved.
  • Removed the now-obsolete multi-stage negative test that asserted the function
    was unsupported, and extended JsonFunctionsTest across every result type
    (happy / default / throw paths, arrays incl. null elements, malformed JSON,
    BigDecimal precision, unsupported type).

Why I did it this way

  • Reused the transform's exact coercion and error rules so the scalar and
    transform forms are behaviorally identical — no second, subtly-different JSON
    semantics for users to reason about.
  • Add-only against JsonFunctions: the existing jsonPath* methods
    (including master's canExtractJsonPath() ingestion-hot-path fast-path) are
    untouched, so this carries zero regression risk for current callers.
  • A BigDecimal-based long parser stands in for pinot-core's
    NumberUtils.parseJsonLong, because pinot-common cannot depend on
    pinot-core; it reproduces the same truncate-toward-zero and exponent
    handling for JSON numeric strings.

Impact

json_extract_scalar now resolves in the multi-stage engine and scalar
contexts, with behavior identical to the transform. No existing function
changes behavior. Supersedes #16910 (revived, rebased on current master, with
that PR's review gaps on parity, throw semantics, and coverage closed).

closes #16486

Co-authored-by: Manik Somayaji somayajimanik@gmail.com

Generated-by: Claude Code (Opus 4.8)

json_extract_scalar was only implemented as a transform function, so it could
not be resolved in the multi-stage query engine or in ad-hoc scalar contexts.
Register it as a scalar function with full result-type parity to the transform:
INT/LONG/FLOAT/DOUBLE/BIG_DECIMAL/BOOLEAN/TIMESTAMP/STRING/JSON/BYTES single
values plus INT/LONG/FLOAT/DOUBLE/BIG_DECIMAL/STRING arrays, reusing the
transform's coercion rules (BooleanUtils numeric convention, TimestampUtils for
ISO-8601 strings, a BigDecimal-preserving parser) and its throw-on-unresolved
semantics. A BigDecimal-based long parser stands in for pinot-core's
NumberUtils.parseJsonLong, which pinot-common cannot depend on.

Supersedes apache#16910.

Co-authored-by: Manik Somayaji <somayajimanik@gmail.com>
@Vamsi-klu Vamsi-klu marked this pull request as ready for review July 12, 2026 00:01
@Vamsi-klu

Copy link
Copy Markdown
Author

Ready for review. This replaces #16910 and rebases it onto current master. I also went back through the points raised in that PR's review: it now handles all the same result types as the transform function, throws in the same situations when a path can't be resolved, and I added a lot more coverage in JsonFunctionsTest.

@Jackie-Jiang you reviewed the original #16910, and @xiangfu0 you've been active in JsonFunctions lately, so a look from either of you would be really helpful whenever you get a chance. Happy to tweak anything.


Drafted-by: Claude Code (Opus 4.8); reviewed by @Vamsi-klu before posting

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add ScalarFunction for JSON_EXTRACT_SCALAR

1 participant