Support PPL format command

[songkant-aws]

Description

Adds the PPL format command, which collapses tabular results into a single search-expression string. It supports configurable row, column, and multivalue delimiters; maxresults; emptystr; null handling; and quote/backslash escaping.

source=web_logs | fields status, host | format

The command also supports runtime search predicates produced by subsearches:

search source=web_logs status>=500 [ search source=rules | fields host ]

The subsearch result is formatted into one scalar search string and combined with the static parent predicate before the OpenSearch request is executed. Explicit format output remains a normal one-row result and can continue through later pipeline commands.

Design

• The grammar and AST represent format options explicitly, including the all-or-none positional delimiter group.
• FormatPlanner lowers each input row to a formatted expression, aggregates rows globally, applies the row wrapper and fallback, and projects the single search field.
• Search subqueries remain structured through parsing. When parent search requires an implicit formatted predicate, the planner builds an implicit Format scalar subquery.
• RuntimeSearchCorrelator rewrites only scalar subqueries registered as implicit Format inputs into the left side of a LogicalCorrelate. Other scalar, IN, and EXISTS subqueries remain untouched.
• The correlated right scan references the formatted string through its correlation variable. DynamicQueryStringSpec defers query-string compilation until the left input has produced its value.
• The runtime query_string filter is appended through the Calcite pushdown path, so it is combined with an existing pushed filter using AND semantics rather than replacing it.
• The legacy execution engine reports the command as Calcite-only.

Testing

• Lexer, parser, AST, anonymizer, and search predicate compiler unit tests.
• Full Calcite logical-plan tests for default options, custom delimiters, multivalue fields, escaping, empty input, explicit format, implicit format, multiple subsearches, and nested subsearches.
• Integration tests for explicit format output, continued pipeline processing, static and dynamic predicate composition, NOT subsearches, multiple subsearches, nested subsearches, explain output, and analytics-engine execution.
• Complete logical and physical explain-plan YAML golden files.
• Scan-level coverage proving that an existing pushed filter and a late-bound dynamic query_string filter are conjoined.

Related Issues

Related to #5233

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• New PPL command checklist all confirmed.
• API changes companion pull request is not applicable because this change does not alter the REST API shape.
• Commits are signed per the DCO using --signoff or -s.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 669bee1)

Here are some key observations to aid the review process:

🧪 PR contains tests

🔒 No security concerns identified

✅ No TODO sections

🔀 No multiple PR themes

⚡ Recommended focus areas for review Possible Issue The formatFieldName method uses a regex pattern that may not correctly handle all edge cases for field name validation. The pattern [A-Za-z_@][A-Za-z0-9_@-]*(\\.[A-Za-z_@][A-Za-z0-9_@-]*)* allows hyphens in the middle of identifiers but the escaping logic with backticks may not align with all PPL identifier rules. If a field name contains characters that match the regex but are not valid unquoted identifiers in the actual query execution context, the generated query string could be malformed. private String formatFieldName(String fieldName) { if (fieldName.matches("[A-Za-z_@][A-Za-z0-9_@-]*(\\.[A-Za-z_@][A-Za-z0-9_@-]*)*")) { return fieldName; } return "`" + fieldName.replace("`", "``") + "`"; } Possible Issue In the buildRuntimeQuery method, if queryParts contains a null element, calling part.toString() at line 172 will throw a NullPointerException. The code does not validate that array elements are non-null before concatenating them into the query string. This can occur if the correlation variable produces a null value at runtime. private String buildRuntimeQuery(String[] queryParts) { StringBuilder query = new StringBuilder(); for (int i = 0; i < queryParts.length; i++) { String part = queryParts[i]; if (pushDownContext.getDynamicQueryString().runtimePredicateParts().contains(i)) { part = pushDownContext.getDynamicQueryString().compiler().compile(part); } query.append(part); } return query.toString(); } Possible Issue The combineSubqueries method uses getFirst() on the subqueries list without checking if the list is empty. If findImplicitFormatSubqueries returns an empty list and this method is called, it will throw NoSuchElementException. Although the caller checks subqueries.isEmpty() before calling combineSubqueries, the method itself does not enforce this precondition, making it fragile to future refactoring. private static RelNode combineSubqueries(List<RexSubQuery> subqueries, RexBuilder rexBuilder) { RelNode result = subqueries.getFirst().rel; for (int i = 1; i < subqueries.size(); i++) { result = LogicalJoin.create( result, subqueries.get(i).rel, List.of(), rexBuilder.makeLiteral(true), Set.of(), JoinRelType.INNER); } return result; } Possible Issue The create method throws IllegalArgumentException if the number of matched predicate parts does not equal the number of runtime predicates. However, the identity comparison predicate == part at line 34 may fail if the same RexNode instance appears multiple times in the flattened parts list. This could lead to false negatives where a legitimate runtime predicate is not recognized, causing the method to throw an exception incorrectly. public static DynamicQueryStringSpec create( RexNode queryExpression, List<RexNode> runtimePredicates, SearchPredicateCompiler compiler) { List<RexNode> parts = new ArrayList<>(); flattenConcatenation(queryExpression, parts); Set<Integer> predicateParts = new LinkedHashSet<>(); for (int i = 0; i < parts.size(); i++) { RexNode part = parts.get(i); if (runtimePredicates.stream().anyMatch(predicate -> predicate == part)) { predicateParts.add(i); } } if (predicateParts.size() != runtimePredicates.size()) { throw new IllegalArgumentException( "Every runtime search predicate must be a query-string concatenation part"); } return new DynamicQueryStringSpec( queryExpression, List.copyOf(parts), Set.copyOf(predicateParts), compiler); }

[github-actions]

PR Code Suggestions ✨

Latest suggestions up to 669bee1
Explore these optional code suggestions:

Category	Suggestion	Impact
General	Validate before state modification The validation that checks getFieldCount() != 1 occurs after the subquery is already identified as an implicit format subquery. If this validation fails, it throws an exception but the seen set has already been modified. Consider performing the field count validation before adding to seen to maintain consistency. core/src/main/java/org/opensearch/sql/calcite/RuntimeSearchCorrelator.java [112-130] static List<RexSubQuery> findImplicitFormatSubqueries( RexNode condition, Predicate<RexSubQuery> isImplicitFormatSubquery) { List<RexSubQuery> subqueries = new ArrayList<>(); Set<RexSubQuery> seen = Collections.newSetFromMap(new IdentityHashMap<>()); condition.accept( new RexVisitorImpl<Void>(true) { @Override public Void visitSubQuery(RexSubQuery subquery) { - if (isImplicitFormatSubquery.test(subquery) && seen.add(subquery)) { + if (isImplicitFormatSubquery.test(subquery)) { if (subquery.rel.getRowType().getFieldCount() != 1) { throw new SemanticCheckException( "Implicit format subsearch must return exactly one column"); } - subqueries.add(subquery); + if (seen.add(subquery)) { + subqueries.add(subquery); + } } return null; } }); return subqueries; } Suggestion importance[1-10]: 7 __ Why: Performing validation before modifying the seen set ensures consistency in error handling. If validation fails, the state remains unchanged, which is a better practice for maintaining data integrity.	Medium
	Cache compiled regex pattern The regex pattern should be compiled once and reused to avoid repeated compilation overhead. Consider declaring a static Pattern field and using pattern.matcher(fieldName).matches() for better performance when this method is called frequently. core/src/main/java/org/opensearch/sql/calcite/FormatPlanner.java [270-275] +private static final Pattern FIELD_NAME_PATTERN = + Pattern.compile("[A-Za-z_@][A-Za-z0-9_@-]*(\\.[A-Za-z_@][A-Za-z0-9_@-]*)*"); + private String formatFieldName(String fieldName) { - if (fieldName.matches("[A-Za-z_@][A-Za-z0-9_@-]*(\\.[A-Za-z_@][A-Za-z0-9_@-]*)*")) { + if (FIELD_NAME_PATTERN.matcher(fieldName).matches()) { return fieldName; } return "`" + fieldName.replace("`", "``") + "`"; } Suggestion importance[1-10]: 6 __ Why: Compiling the regex pattern once as a static field improves performance when formatFieldName is called frequently. This is a good optimization for a method that processes field names repeatedly.	Low
	Extract repeated method calls The contains(i) check on a Set is performed inside the loop for every iteration. Consider extracting runtimePredicateParts() and compiler() to local variables before the loop to avoid repeated method calls and improve readability. opensearch/src/main/java/org/opensearch/sql/opensearch/storage/scan/CalciteEnumerableIndexScan.java [163-173] private String buildRuntimeQuery(String[] queryParts) { StringBuilder query = new StringBuilder(); + Set<Integer> runtimeParts = pushDownContext.getDynamicQueryString().runtimePredicateParts(); + SearchPredicateCompiler compiler = pushDownContext.getDynamicQueryString().compiler(); for (int i = 0; i < queryParts.length; i++) { String part = queryParts[i]; - if (pushDownContext.getDynamicQueryString().runtimePredicateParts().contains(i)) { - part = pushDownContext.getDynamicQueryString().compiler().compile(part); + if (runtimeParts.contains(i)) { + part = compiler.compile(part); } query.append(part); } return query.toString(); } Suggestion importance[1-10]: 5 __ Why: Extracting runtimePredicateParts() and compiler() to local variables before the loop reduces repeated method calls and improves code readability, though the performance gain is modest.	Low
	Use fully qualified class name The error message should include the actual class name for better debugging. Consider using filterNode.getClass().getName() instead of getSimpleName() to provide the fully qualified class name, which is more helpful when diagnosing issues across different packages. core/src/main/java/org/opensearch/sql/calcite/RuntimeSearchCorrelator.java [50-54] if (!(filterNode instanceof Filter filter)) { throw new IllegalStateException( "Runtime search query must produce a filter, but got " - + filterNode.getClass().getSimpleName()); + + filterNode.getClass().getName()); } Suggestion importance[1-10]: 4 __ Why: Using getName() instead of getSimpleName() provides more context in error messages, but this is a minor improvement that doesn't significantly impact functionality or debugging in most cases.	Low

---

Previous suggestions

Suggestions up to commit da0ec85

Category	Suggestion	Impact
Possible issue	Prevent potential NoSuchElementException The method uses getFirst() which throws NoSuchElementException if the list is empty. Add a guard to verify the list is non-empty before accessing elements, or handle the empty case explicitly. core/src/main/java/org/opensearch/sql/calcite/RuntimeSearchCorrelator.java [133-146] private static RelNode combineSubqueries(List<RexSubQuery> subqueries, RexBuilder rexBuilder) { - RelNode result = subqueries.getFirst().rel; + if (subqueries.isEmpty()) { + throw new IllegalArgumentException("Cannot combine empty subquery list"); + } + RelNode result = subqueries.get(0).rel; for (int i = 1; i < subqueries.size(); i++) { result = LogicalJoin.create( result, subqueries.get(i).rel, List.of(), rexBuilder.makeLiteral(true), Set.of(), JoinRelType.INNER); } return result; } Suggestion importance[1-10]: 7 __ Why: The method uses getFirst() which can throw NoSuchElementException if the list is empty. However, the caller correlate() already validates that subqueries is non-empty before calling this method (line 58), so this is a defensive improvement rather than a critical bug fix.	Medium
	Add null safety for query parts The method doesn't validate that queryParts is non-null or that its length matches expectations. If runtimeQueryParts is null when passed to scan(), this will cause a NullPointerException. Add null checks before processing. opensearch/src/main/java/org/opensearch/sql/opensearch/storage/scan/CalciteEnumerableIndexScan.java [163-173] private String buildRuntimeQuery(String[] queryParts) { + if (queryParts == null || queryParts.length == 0) { + throw new IllegalArgumentException("Query parts cannot be null or empty"); + } StringBuilder query = new StringBuilder(); for (int i = 0; i < queryParts.length; i++) { String part = queryParts[i]; if (pushDownContext.getDynamicQueryString().runtimePredicateParts().contains(i)) { part = pushDownContext.getDynamicQueryString().compiler().compile(part); } query.append(part); } return query.toString(); } Suggestion importance[1-10]: 6 __ Why: The method buildRuntimeQuery() is only called from scan() at line 148, where runtimeQueryParts is already checked for null before calling this method. However, adding explicit validation improves defensive programming and makes the method more robust if called from other contexts in the future.	Low
General	Validate implicit format field requirements When no fields are present, the method returns early without checking if node.isImplicit() is true. For implicit format subsearches, this could bypass necessary correlation setup. Verify that early return is safe for implicit format scenarios. core/src/main/java/org/opensearch/sql/calcite/FormatPlanner.java [53-56] if (fields.isEmpty()) { + if (node.isImplicit()) { + throw new IllegalStateException("Implicit format subsearch requires at least one field"); + } builder.values(new String[] {SEARCH_FIELD}, node.getEmptyString()); return builder.peek(); } Suggestion importance[1-10]: 5 __ Why: The suggestion raises a valid concern about early return for implicit format, but the current implementation appears intentional. The emptyString fallback is a valid result even for implicit subsearches. The suggestion may be overly restrictive without clear evidence of a bug.	Low

[github-actions]

PR Code Analyzer ❗

AI-powered 'Code-Diff-Analyzer' found issues on commit 669bee1.

Path	Line	Severity	Description
ppl/src/main/java/org/opensearch/sql/ppl/parser/PPLSearchPredicateCompiler.java	30	medium	Empty or null predicate compiles to '*:*' (match-all). If a subsearch returns an empty or null 'search' field, the implicit format will produce a predicate that matches every document in the parent index, potentially exposing data the caller did not intend to retrieve. This is an unguarded default that could be exploited by crafting a subsearch that always returns an empty search value.
core/src/main/java/org/opensearch/sql/calcite/FormatPlanner.java	89	medium	A scalar field named 'search' in a subsearch result is injected verbatim as an OpenSearch query_string predicate after only PPL parse validation. A principal who can write arbitrary values into an indexed 'search' field (or control eval expressions in a subsearch) can craft search predicates that influence which parent-index documents are returned — for example using OpenSearch query_string wildcards or field-level queries against fields they should not be able to filter on directly.

The table above displays the top 10 most important findings.

Total: 2 | Critical: 0 | High: 0 | Medium: 2 | Low: 0

---

Pull Requests Author(s): Please update your Pull Request according to the report above.

Repository Maintainer(s): You can bypass diff analyzer by adding label skip-diff-analyzer after reviewing the changes carefully, then re-run failed actions. To re-enable the analyzer, remove the label, then re-run all actions.

---

⚠️ Note: The Code-Diff-Analyzer helps protect against potentially harmful code patterns. Please ensure you have thoroughly reviewed the changes beforehand.

Thanks.

[github-actions]

Persistent review updated to latest commit 669bee1

[penghuo]

enable doc-test for format command.

[penghuo]

Test query not releated to format command.

[penghuo]

Add query results.

[penghuo]

Add query results.

[penghuo]

Why mention calcite engine specific? It is our default execution engine.

[penghuo]

This is not a real limitation.

[penghuo]

Polish limitation section. make it user friendly.

[penghuo]

What is upper-bound?

[penghuo]

this is enhancement of search command, please update search command doc.

[penghuo]

Does this feature works with append + search command?

[penghuo]

Does this feature works with append + search command?

[penghuo]

avoid using record. hard to backport.

[penghuo]

buildRuntimeQuery is a function of DynamicQueryStringSpec

[penghuo]

add UT for new class.

[penghuo]

what if the results can not be convert to string? what is customer facing error message?

[penghuo]

queryString and subsearch can not co-exist?

[penghuo]

source = outer a in [ source = inner | fields a ] is as same as source = outer [ source = inner | fields a ]. the dynamic string pushdown works for both cases?