[CALCITE-6636] Support CNF condition of Arrow ArrowAdapter by caicancai · Pull Request #4848 · apache/calcite

caicancai · 2026-03-26T14:34:22Z

https://issues.apache.org/jira/browse/CALCITE-6636

mihaibudiu · 2026-03-26T22:16:07Z

core/src/main/java/org/apache/calcite/util/Bug.java

   * <a href="https://issues.apache.org/jira/browse/CALCITE/issues/CALCITE-6293">
   * [CALCITE-6293] Support OR condition in Arrow adapter</a> is fixed. */
-  public static final boolean CALCITE_6293_FIXED = false;
+  public static final boolean CALCITE_6293_FIXED = true;


Maybe these can be completely removed?

mihaibudiu · 2026-03-26T22:16:47Z

arrow/src/test/java/org/apache/calcite/adapter/arrow/ArrowAdapterTest.java

-    }
+    String plan = "PLAN=ArrowToEnumerableConverter\n"
+        + "  ArrowProject(intField=[$0], stringField=[$1])\n"
+        + "    ArrowFilter(condition=[SEARCH($0, Sarg[0, 1, 2])])\n"


Arrow supports search?

Arrow/Gandiva does not support the SEARCH operator; I have fully expanded the SEARCH operator.

mihaibudiu · 2026-03-26T22:18:09Z

arrow/src/main/java/org/apache/calcite/adapter/arrow/ArrowTranslator.java

+  List<List<String>> translateMatch(RexNode condition) {
+    // Expand SEARCH nodes and convert to CNF
+    final RexNode expanded = RexUtil.expandSearch(rexBuilder, null, condition);
+    final RexNode cnf = RexUtil.toCnf(rexBuilder, expanded);


This could be much larger than the original condition.
You should add some tests with deeper conditions (multiple nested levels of parens).

mihaibudiu · 2026-03-26T22:21:02Z

arrow/src/main/java/org/apache/calcite/adapter/arrow/ArrowTable.java

    return builder.build();
  }

+  private TreeNode parseSingleCondition(String condition) {


I don't really know the grammar for these conditions, so I cannot tell whether the spaces are where you expect them. Is this documented someplace?

What happens if you have a comparison with a string containing a space, for example?

I've added instructions, but it doesn't seem to solve the empty string problem yet. I need to think about other solutions. 🤔

I have now changed it to a structured token format instead of string parsing:

unary: [fieldName, operator]

binary: [fieldName, operator, value, type]

mihaibudiu · 2026-03-27T18:39:52Z

arrow/src/main/java/org/apache/calcite/adapter/arrow/ArrowRel.java

    /** Adds new predicates.
     *
-     * @param predicates Predicates
+     * @param predicates Predicates in CNF form (outer list is AND, inner list is OR)


I see 3 lists, can you explain all of them?

Thank you, I have improved the comments.

mihaibudiu · 2026-03-27T18:42:27Z

arrow/src/main/java/org/apache/calcite/adapter/arrow/ArrowTable.java

+   *       e.g. {@code ["intField", "isnull"]}</li>
+   * </ul>
+   *
+   * <p>Using structured tokens avoids string splitting and safely supports


There is no need to justify this choice.
In general, you should use higher-level representations as much as possible.

mihaibudiu · 2026-03-27T18:43:08Z

arrow/src/main/java/org/apache/calcite/adapter/arrow/ArrowTranslator.java

-    } else {
-      throw new UnsupportedOperationException("Unsupported disjunctive condition " + condition);
+  /** The maximum number of nodes allowed during CNF conversion.
+   * If exceeded, the original expression is returned unchanged. */


what happens to the translation in that case? It fails?

add testLiteralWithEmptyString

mihaibudiu · 2026-03-27T18:44:24Z

arrow/src/main/java/org/apache/calcite/adapter/arrow/ArrowTable.java

    return builder.build();
  }

+  /** Parses a single condition into a Gandiva {@link TreeNode}.


It looks to me that it would be better to define a new class for this data structure UnaryOrBinaryCondition perhaps, instead of using a List.

mihaibudiu · 2026-03-31T11:23:01Z

I am currently on vacation; I will be only able to review this later in April

caicancai · 2026-03-31T12:23:23Z

I am currently on vacation; I will be only able to review this later in April

Have a great holiday and have some fun!

mihaibudiu · 2026-04-01T22:57:44Z

arrow/src/main/java/org/apache/calcite/adapter/arrow/ArrowTable.java

  }

+  /** Parses a single {@link ConditionToken} into a Gandiva {@link TreeNode}. */
+  private TreeNode parseSingleCondition(ConditionToken token) {


"parse" is probably no longer appropriate, perhaps "convertConditionToGandiva?"

mihaibudiu · 2026-04-01T23:00:06Z

arrow/src/main/java/org/apache/calcite/adapter/arrow/ArrowTranslator.java

+   * the original expression unchanged, which may cause the subsequent
+   * translation to Gandiva predicates to fail with an
+   * {@link UnsupportedOperationException}. That exception is caught by
+   * {@link ArrowRules.ArrowFilterRule#onMatch}, which silently skips the


You are making here some assumptions about who is calling this code which may not hold; in general, when you write a library, you don't know how it will be used. You can be more precise by saying that this is a possible path: "when invoked by the module to convert to Arrow..."

mihaibudiu · 2026-04-01T23:00:54Z

arrow/src/main/java/org/apache/calcite/adapter/arrow/ArrowTranslator.java

+   * {@link UnsupportedOperationException}. That exception is caught by
+   * {@link ArrowRules.ArrowFilterRule#onMatch}, which silently skips the
+   * Arrow convention and falls back to an Enumerable plan. */
+  private static final int MAX_CNF_NODE_COUNT = 256;


Ideally this would not be a constant, but a parameter of translateMatch, similar to toCNF, but I don't know if that is possible.

I currently have no requirements for configurability; however, if users express a need for it in the future, I can certainly provide support. Let's let it run for a while first.

mihaibudiu · 2026-04-01T23:02:04Z

arrow/src/main/java/org/apache/calcite/adapter/arrow/ConditionToken.java

+ * and operator; binary conditions additionally have a literal value
+ * and its type.
+ *
+ * <p>This class replaces the raw {@code List<String>} representation


In general JavaDoc does not need to discuss things that do no longer exists. I think you can just remove this comment.

mihaibudiu · 2026-04-01T23:02:52Z

arrow/src/test/java/org/apache/calcite/adapter/arrow/ArrowAdapterTest.java

+  /** When a filter condition exceeds the CNF node limit, the Arrow adapter
+   * falls back to the Enumerable convention (EnumerableCalc) instead of
+   * using ArrowFilter. The query should still return correct results. */
+  @Test void testCnfExceedsLimitFallsBackToEnumerable() {


I am assuming you have validated these Arrow plans somehow.

sonarqubecloud · 2026-04-02T04:19:00Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
98.1% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

caicancai marked this pull request as draft March 26, 2026 14:35

caicancai force-pushed the 6636 branch from e95e250 to f1049a2 Compare March 26, 2026 14:40

caicancai marked this pull request as ready for review March 26, 2026 14:41

mihaibudiu reviewed Mar 26, 2026

View reviewed changes

caicancai force-pushed the 6636 branch 2 times, most recently from fcd59d0 to ed477d9 Compare March 27, 2026 14:10

caicancai requested a review from mihaibudiu March 27, 2026 14:43

mihaibudiu reviewed Mar 27, 2026

View reviewed changes

caicancai requested a review from mihaibudiu March 29, 2026 13:33

mihaibudiu approved these changes Apr 1, 2026

View reviewed changes

mihaibudiu added the LGTM-will-merge-soon Overall PR looks OK. Only minor things left. label Apr 1, 2026

[CALCITE-6636] Support CNF condition of Arrow ArrowAdapter

0366657

caicancai force-pushed the 6636 branch from eec5c88 to 0366657 Compare April 2, 2026 03:47

caicancai merged commit 1aacc43 into apache:main Apr 2, 2026
19 checks passed

Conversation

caicancai commented Mar 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

caicancai Mar 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mihaibudiu commented Mar 31, 2026

Uh oh!

caicancai commented Mar 31, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sonarqubecloud bot commented Apr 2, 2026

Quality Gate passed

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

caicancai commented Mar 26, 2026 •

edited

Loading

caicancai Mar 27, 2026 •

edited

Loading