feat: unify filter sidebar and query input into single where clause

[Official-Krish]

Summary

The filter sidebar and the query input box previously held independent state — selecting a value in the sidebar applied it to the search but never appeared in the query input, so the two could silently drift out of sync. This PR makes the where clause the single source of truth for both.

How it works:

• Selecting a filter in the sidebar rewrites the matching facet clause directly in the query input (e.g. clicking error in the level facet writes level:"error" into the box)
• The sidebar reads its checked state back from the where text, so editing the query also updates the sidebar
• Free-text and complex query content is preserved — only the specific facet clauses are rewritten
• Works for both query dialects (Lucene and SQL)
• The separate filters URL param is removed; a one-time migration moves any legacy persisted filters into the where clause on first load

New internals in common-utils/filters.ts:

• parseWhereClauseToFilterState — parse a where string back into FilterState
• filterStateToWhereClause — render FilterState to a where string
• replaceFilterClauses — surgically replace only the facet clauses in a where string, preserving everything else
• dateTimeValueExpr extracted to common-utils/core/dateTimeValue.ts to avoid duplication between filters.ts and queryParser.ts

Before	After
Clicking a sidebar filter applies it to the search, but the query input stays empty.	Clicking a sidebar filter writes the condition into the query input (e.g. level:"error").

How to test on Vercel preview

Preview routes: /search

Steps:

1. Open the Search page and select any log/trace source
2. Click a value in the filter sidebar (e.g. a service name or log level)
3. Verify the corresponding condition appears in the query input box (e.g. ServiceName:"my-service" or level:"error")
4. Manually type a condition into the query input (e.g. level:"warn") and press enter
5. Verify the matching value is highlighted/checked in the filter sidebar
6. With existing query text like my error message level:"error", click a different level in the sidebar
7. Verify only the level: clause is rewritten — the free text my error message is preserved

References

Fixes #2751

[changeset-bot]

🦋 Changeset detected

Latest commit: c50da64

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages

Name	Type
@hyperdx/common-utils	Minor
@hyperdx/api	Patch
@hyperdx/app	Patch
@hyperdx/otel-collector	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[vercel]

@Official-Krish is attempting to deploy a commit to the HyperDX Team on Vercel.

A member of the Team first needs to authorize it.

[greptile-apps]

Greptile Summary

The PR makes the search where clause the canonical source for query text and sidebar facet state while preserving legacy filters and supporting Lucene/SQL transitions.

• Adds shared parsing, rendering, migration, replacement, and query-diagnostic utilities.
• Connects sidebar mutations and language changes to the canonical where form.
• Handles repeated SQL facet predicates by replacing all predicates owned by the sidebar.
• Adds focused round-trip and integration regression coverage.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the previously reported Lucene and SQL replacement issues are addressed by the current implementation.

Important Files Changed

Filename	Overview
packages/common-utils/src/filters.ts	Implements bidirectional facet parsing and replacement, including fixes for boolean semantics, unmanaged Lucene predicates, case-insensitive SQL facets, and repeated SQL predicates.
packages/app/src/searchFilters.tsx	Adapts canonical where clauses to the existing sidebar FilterState and supports replacement, migration, and language translation.
packages/app/src/DBSearchPage.tsx	Makes where text canonical, migrates legacy filters, translates facets during language changes, and surfaces incomplete or partially representable queries.
packages/common-utils/src/tests/filterRoundTrip.test.ts	Adds comprehensive regression coverage for parsing, rendering, migration, replacement, negation, boolean grouping, and SQL predicate handling.

_{Reviews (5): Last reviewed commit: "fix(search): replace all duplicate SQL p..." | Re-trigger Greptile}

[pulpdrew]

Hey @Official-Krish, thanks for the PR, this is definitely a feature we'd love to see!

However, there are some things that I think will need to be addressed.

Bugs in WHERE string generation (Lucene)

1. NOT term AND ServiceName:"api" → click a filter → term AND (ServiceName:"api" OR ServiceName:"accounting"). The NOT is stripped, so the query is inverted.
2. ServiceName:"api" OR SeverityText:"error" → click a filter → ServiceName:"api" AND SeverityText:"error". OR silently becomes AND.
3. ServiceName:("api" OR "web") AND term → click a filter → ("api" OR "web") AND term AND (ServiceName:"api" OR ServiceName:"web"). The field name is dropped, so a field-scoped search becomes a full-text search, and the clause is duplicated.
4. Duration:[* TO 100] AND ServiceName:"api" → click a filter → ServiceName:"web". The range clause is deleted because it starts at character offset 0 (falsy-offset check).
5. Duration:{10 TO 20} → click a filter → Duration:[10 TO 20]. Exclusive bounds silently become inclusive.
6. msg:"hello"~2 AND ServiceName:"api" → click a filter → the ~2 proximity modifier is dropped (same for ^3 boost).
7. level:error (unquoted, the idiomatic Lucene form) → click warn in the sidebar → level:error AND level:"warn". Always returns zero rows.
8. Click a sidebar value for an attribute key containing (, ", { or [ (e.g. LogAttributes['a(b)']) → the emitted text fails to parse, so the query breaks and cannot be fixed from the sidebar. A key containing a space (LogAttributes['my key']) parses but becomes free text plus a nonexistent field.
9. "timeout)" OR "error" → click a filter → "timeout)" OR "error" AND level:"x". The paren inside the quoted value defeats the top-level-OR check, so the parens that f89fc39c9 added are omitted and AND binds tighter.
10. term1 OR term2 → click a filter four times → (term1 OR term2) AND .... One space is added per interaction, growing the query text and URL without bound.

Bugs in WHERE string generation (SQL)

11. ServiceName = 'a' OR ServiceName = 'b' → click a filter → ServiceName = 'a' OR ServiceName = 'b' AND SeverityText IN ('error'). Same precedence bug f89fc39c9 fixed for Lucene; the filter applies only to the 'b' branch.
12. ServiceName = 'a' -- temp note → click a filter → ServiceName = 'a' -- temp note AND SeverityText IN ('error'). The new predicate lands inside the comment: the checkbox looks applied but nothing is filtered.
13. msg = 'x AND y IN (' (unbalanced paren in a string) or `it's` = 1 (quote in a quoted identifier) → click the same filter twice → ... AND ServiceName IN ('a') AND ServiceName IN ('b'). Paren counting runs before the in-string guard, so conjunct splitting stops and clauses pile up. Unchecking stops working; new selections have no effect.
14. Same duplication for any column needing backticks: click a value on a service-name column twice → `service-name` IN ('a') AND `service-name` IN ('a', 'b'). The matcher compares quoted keys against unquoted ones.
15. ServiceName IN (SELECT name FROM t) AND foo = 1 → the sidebar renders a checkbox labelled SELECT name FROM t; click any filter → foo = 1 AND ServiceName IN ('b'). The subquery is destroyed.

Backwards incompatibility (existing URLs and saved searches)

16. Open a bookmarked URL / saved search with whereLanguage=lucene, where=ServiceName:"api" and filters=[SeverityText IN ('error')] → migration produces SeverityText:"error". The where clause's own filter is destroyed and written back to the URL.

UX regressions

17. It's common to start querying in lucene, potentially adding filters, then switch to SQL when a more complex condition is needed. Previously, all filters persisted when switching languages. Now all filters are lost until the user re-selects them or re-writes the WHERE input, which is stuck in the previous language. Ideally we should not lose filters when switching languages, the filters should transfer over to the new language.
18. Type service:" (any incomplete query) → all checkboxes clear, and clicking a filter does nothing at all while still triggering a re-query. No error or explanation is shown.
19. NOT ServiceName:"api" or term AND NOT ServiceName:"api" → the sidebar shows api as checked, i.e. the opposite of what the query does.
20. ServiceName:"api" OR SeverityText:"error" → the sidebar shows both as checked, implying an AND.

Performance regressions

21. Type in the search input → the whole filter sidebar re-renders per keystroke. handleSetFilters now depends on the watched where value, so all eight mutators get new identities and defeat memo(DBSearchPageFiltersComponent). The search page is noticeably laggy when typing, with these changes.

[Official-Krish]

Thanks for the review @pulpdrew! I'll work through these issues, push a revised implementation that addresses them, and update the PR shortly.

[Official-Krish]

Hey @pulpdrew! I believe I've addressed all the issues you pointed out in the review. I've pushed the latest changes, would appreciate it if you could take another look when you have a chance. Thanks!

[pulpdrew]

Thanks for addressing those @Official-Krish. I think there are still some gaps though:

1. Unquoted / wildcard / comparison Lucene terms are deleted by an unrelated sidebar click

collectFromAst (filters.ts:341) marks a field managed for unquoted terms but contributes no value, so renderNode (filters.ts:1011) prunes the clause and nothing re-emits it.

Repro — search page, Lucene:

1. Type ServiceName:api* AND SeverityText:"error", submit.
2. Click warn in the SeverityText facet.
3. Query box now reads SeverityText:"warn" — the ServiceName:api* term is gone.

Same for SeverityText:error AND ServiceName:"api" and Duration:>100 AND ServiceName:"api". Tests at filterRoundTrip.test.ts:655-676 only cover the case where newState does contain the unquoted field.

2. Multi-line SQL where is duplicated by a sidebar click, compounding each time

splitSqlConjuncts (filters.ts:1104) only splits on the literal ' AND ', so a newline before AND isn't a separator and the whole text is treated as one facet.

Repro — search page, SQL (Shift-Enter for the newline):

1. Type Body LIKE '%x%' ⏎ AND ServiceName IN ('api'), submit.
2. Click web in the ServiceName facet.
3. Query box now reads Body LIKE '%x%'\nAND ServiceName IN ('api') AND Body LIKE '%x%'\nAND ServiceName IN ('api') AND ServiceName IN ('web'). Each further click doubles it again.

The sidebar also shows a garbage facet named Body LIKE '%x%'\nAND ServiceName. Multi-line SQL is first-class here (allowMultiline = true, SearchWhereInput.tsx:157; Shift-Enter keybinding, SQLInlineEditor.tsx:283-292).

3. Newline-adjacent top-level OR in SQL escapes the paren guard

hasTopLevelOrSql (filters.ts:958) matches only the literal ' OR ', so a = 1\nOR b = 2 isn't detected and the residual isn't parenthesized.

Repro:

replaceFilterClauses("a = 1\nOR b = 2", 'sql', {svc: {included: {'x'}}})
→ "a = 1\nOR b = 2 AND svc IN ('x')"   // parses as a = 1 OR (b = 2 AND svc IN ('x'))

Exactly the mis-parse the function's docstring says it prevents. Lucene is unaffected.

4. Lucene field-name escaping isn't reversible; backslashes multiply on every click

escapeLuceneFieldName (filters.ts:178) escapes 9 characters, but decodeSpecialTokens only reverses \: and \".

Repro — attribute key containing a space, e.g. LogAttributes['my key']:

1. Click that value in the sidebar → LogAttributes.my\ key:"a".
2. Click again → LogAttributes.my\\\\\ key:"a", emitting SQL key LogAttributes['my\\\\\\ key'].
3. Repeat → grows without bound.

After the first click the predicate targets a nonexistent map key (zero rows) and the checkbox never reads back as checked. filterRoundTrip.test.ts:678 only asserts parse() doesn't throw.

5. Language switch emits syntactically mixed clauses and destroys the original text

translateWhereClauseInQuery joins the source-language residual to target-language facets with AND.

Repro:

1. In SQL, type ServiceName = 'api' AND level IN ('error').
2. Switch the language toggle to Lucene.
3. Query box reads ServiceName = 'api' AND level:"error" — the residual is now Lucene free text, a silently different query. Reverse direction gives invalid SQL.

Previously the text was preserved verbatim on a switch, so switching back recovered it; now it's unrecoverable.

6. coerceBooleanValue converts the string values "true"/"false" to booleans

filters.ts:478.

Repro:

1. In Lucene, type msg:"true".
2. Switch to SQL → msg IN (true), a type error against a String column.

There's no way to filter a string column on the literal value true.

7. NOT (col IN (...)) produces a self-contradicting clause

The negation isn't recognized as a facet (so isn't pruned) but the field is still re-emitted.

Repro:

replaceFilterClauses("NOT (svc IN ('a'))", 'sql', {svc: {included: {'b'}}})
→ "NOT (svc IN ('a')) AND svc IN ('b')"

The sidebar shows only svc = b while the query also excludes a.

Related: the new subquery guard (filters.ts:1658) matches ' SELECT '/' FROM ' with surrounding spaces, so col IN (SELECT max(x)) slips through and is treated as a facet.

8. Dead code / cleanups

• filters.ts:400-405: leftManaged / rightManaged are built and never read.
• filters.ts:1010: parse(whereText) called a second time when ast is already available.
• filters.ts:1026: hasTopLevelOr(residual) gets the untrimmed residual while the parenthesized value uses trimmedResidual.
• filters.ts is now ~1980 lines; the round-trip machinery (collect / render / span / split) would read better as its own module.

9. Missing changeset for @hyperdx/app

.changeset/quiet-lions-repeat.md bumps only @hyperdx/common-utils, and its body describes the follow-up fixes rather than the feature. The user-visible change lives in @hyperdx/app (sidebar⇄query unification, removal of the filters URL param, new sidebar alerts). @hyperdx/app changesets are an established convention here (03f5bee87, 90f9343f3).

[pulpdrew]

A few comments

[elizabetdev]

Hi @Official-Krish! I’m a designer on the HyperDX team. The PR looks good! I’m reviewing it from a design perspective and noticed two small issues. Both are non-blockers!

Thanks for working on this! 🙌

[elizabetdev]

As documented in https://github.com/hyperdxio/hyperdx/blob/main/agent_docs/code_style.md#semantic-component-variants-alert--text--danger-controls you should use a semantic variant for the alert.

Suggested change

	<Alert
	variant="light"
	color="orange"
	radius="sm"
	p="xs"
	title="Query is incomplete"
	icon={<IconAlertCircle size={16} />}
	>
	<Text size="xs" c="dimmed">
	Finish the query text above, or click a value below to replace
	the incomplete text.
	</Text>
	</Alert>
	<Alert
	variant="warning"
	radius="sm"
	p="xs"
	title="Query is incomplete"
	icon={<IconAlertTriangle size={16} />}
	>
	<Text size="xs">
	Finish the query text above, or click a value below to replace
	the incomplete text.
	</Text>
	</Alert>

[elizabetdev]

Same here.