[CALCITE-7448] Add support for `:` as a field/item access operator by tmater · Pull Request #4844 · apache/calcite

tmater · 2026-03-24T16:14:59Z

Changes Proposed

This PR adds opt-in support for : as a field/item access operator behind a new SqlConformance#isColonFieldAccessAllowed() hook. No built-in conformance enables it yet, so the default parser behavior is unchanged.

It also refactors postfix parsing so colon field access composes with existing dot and bracket access. That covers cases such as v:field, v:['field name'], arr[1]:field, obj['x']:nested['y'], and colon access followed by bracket or dot postfixes.

To avoid ambiguity when colon field access is enabled, JSON_OBJECT and JSON_OBJECTAGG must use VALUE syntax instead of : or comma-separated key/value shorthand.

Reproduction

select v:field, v:['field name'], arr[1]:field, obj['x']:nested from t;

select json_object(key v:field value arr[1]:field);

select (v::variant)[1] from t;

Testing

Added parser coverage in SqlParserTest and BabelParserTest for valid and invalid colon field access, JSON constructor disambiguation, and :: cast interactions.

Jira Link

CALCITE-7448

…cket access

caicancai · 2026-03-25T02:27:24Z

I might wait until the CI issue is resolved before reviewing this PR.

caicancai · 2026-03-25T13:09:30Z

I don't understand Calcite JavaCC code very well, I might need to learn it. If no one reviews it this week, I'll start working on it next week.

tmater · 2026-03-25T14:17:09Z

I don't understand Calcite JavaCC code very well, I might need to learn it. If no one reviews it this week, I'll start working on it next week.

Thank you for taking a look, @caicancai. I agree this is still a fairly complex PR, and I wouldn’t claim to fully understand all of its implications yet.

That’s also why I tried to include a wide range of test variations and keep the changes gated behind a conformance config.

I’ve organized it into five commits (prior to the CI failures) to make the review easier. If it would help further, I’m happy to split it into 4–5 smaller PRs.

mihaibudiu · 2026-03-25T17:21:42Z

Is the behavior with :: needing parenthesis a new feature, or was this already present previously?

…acket refactor The branch moved bracket ([...]) and dot (.) postfix handling out of Expression2()'s loop and into AddRegularPostfixes (called from AddExpression2b). Babel's InfixCast production was implicitly relying on Expression2()'s loop to consume postfixes after the DataType() call, so after the refactor, expressions like v::varchar[1] would fail to parse because no grammar rule picked up the trailing [1]. Fix: call AddRegularPostfixes(list) after DataType() in InfixCast, making the postfix consumption explicit rather than relying on the caller's loop. Also consolidate InfixCast test coverage: remove the now-redundant testInfixCastBracketAccessNeedsParentheses and extend testColonFieldAccessWithInfixCast with comprehensive cases covering bracket access, dot access, parenthesized forms, and array types—both with and without isColonFieldAccessAllowed.

tmater · 2026-04-01T12:26:30Z

Is the behavior with :: needing parenthesis a new feature, or was this already present previously?

@mihaibudiu , good catch! The parenthesis requirement itself is not new, on main, v::varchar[1] already means "cast to VARCHAR[1]" (bracket binds to the type), so parentheses were always needed to subscript the cast result.

However, you're right that something was off: the bracket refactor broke parsing v::varchar[1] entirely, because InfixCast was implicitly relying on Expression2()'s loop to consume postfixes. This commit fixes that by adding an explicit AddRegularPostfixes(list) call after DataType() in InfixCast.

caicancai · 2026-04-05T06:56:51Z

core/src/main/codegen/templates/Parser.jj

+    [
+        LOOKAHEAD(2, <COLON> SimpleIdentifier(),
+            { this.conformance.isColonFieldAccessAllowed() })
+        <COLON>


Why is chained colon access not supported?

I kept : intentionally non-chainable here. After one : we are already back in the existing postfix world, so the follow-up access can be expressed with the normal operators:

a:b.c instead of a:b:c

a:['x'].y instead of a:['x']:y

(a:b)['x'] / a:b['x'] instead of a:b:['x']

So a second : would mostly be extra surface syntax, not extra expressiveness. Keeping it to a single colon also keeps the grammar narrower and avoids widening the ambiguous space around other colon uses, especially :: in Babel and JSON constructor : handling.

If we decide later that repeated : is a real dialect requirement, we can extend the current [...] to a loop, but I wanted to start with the minimal syntax that covers the targeted cases.

caicancai · 2026-04-05T06:58:38Z

babel/src/main/codegen/includes/parserImpls.ftl

                s.pos()));
        list.add(dt);
    }
+    AddRegularPostfixes(list)


I'm not sure if it will affect existing semantics.

Before the refactor, InfixCast could stop after DataType() and rely on the outer Expression2() loop to notice any following . or [...]. After the refactor, that outer postfix branch was removed and the shared postfix handling moved earlier into AddExpression2b(). But Babel InfixCast is not part of that earlier path; it is an extraBinaryExpressions hook that runs later.

So if InfixCast does not invoke AddRegularPostfixes() itself, nobody else will. That is why it has to “own” it now: not because the semantics changed, but because the control flow changed. The shared postfix parser still exists, but this is now the only place on the Babel :: path where it can be called.

caicancai · 2026-04-05T06:59:17Z

babel/src/test/java/org/apache/calcite/test/BabelParserTest.java

+    // Double-colon followed by colon field access is not allowed
+    f.sql("select v::variant^:^field from t")
+        .fails("(?s).*Encountered \":.*\".*");
+  }


You can add a blank line.

caicancai · 2026-04-07T14:31:16Z

@tmater Thank you for following up. I left a question on Jira. Could you answer it when you have time?

sonarqubecloud · 2026-04-07T22:54:21Z

Quality Gate passed

Issues
2 New issues
0 Accepted issues

Measures
0 Security Hotspots
100.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

tmater added 5 commits March 24, 2026 08:52

[CALCITE-7448] Refactor expression postfix parsing

2e9384d

[CALCITE-7448] Add conformance hook for colon field access

23e023e

[CALCITE-7448] Add conformance-gated colon field access

9e79114

[CALCITE-7448] Disambiguate JSON constructors from colon field access

a158419

[CALCITE-7448] Refine parser coverage and parenthesize infix cast bra…

a0dec2c

…cket access

tmater added 2 commits March 25, 2026 09:45

[CALCITE-7448] Clarify infix cast bracket parser coverage

de84cfd

[CALCITE-7448] Remove redundant colon field conformance override

9562ed9

caicancai reviewed Apr 5, 2026

View reviewed changes

Add blank line between methods

bdf018c

caicancai self-assigned this Apr 7, 2026

Conversation

tmater commented Mar 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes Proposed

Reproduction

Testing

Jira Link

Uh oh!

caicancai commented Mar 25, 2026

Uh oh!

caicancai commented Mar 25, 2026

Uh oh!

tmater commented Mar 25, 2026

Uh oh!

mihaibudiu commented Mar 25, 2026

Uh oh!

tmater commented Apr 1, 2026

Uh oh!

caicancai Apr 5, 2026

Choose a reason for hiding this comment

Uh oh!

tmater Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

caicancai Apr 5, 2026

Choose a reason for hiding this comment

Uh oh!

tmater Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

caicancai Apr 5, 2026

Choose a reason for hiding this comment

Uh oh!

tmater Apr 7, 2026

Choose a reason for hiding this comment

Uh oh!

caicancai commented Apr 7, 2026

Uh oh!

sonarqubecloud bot commented Apr 7, 2026

Quality Gate passed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tmater commented Mar 24, 2026 •

edited

Loading