feat(lapis): support scalar functions in aggregation fields via dot notation by fhennig · Pull Request #1780 · GenSpectrum/LAPIS

fhennig · 2026-07-23T14:38:50Z

resolves #1767

Summary

Users can now specify computed fields like date.isoWeek in the fields parameter of the /aggregated endpoint. LAPIS validates the function against a whitelist (currently isoWeek for date fields), generates a map() step before groupBy() in the SaneQL query, and renames the internal alias columns back to the user-facing field.function names in the response.

PR Checklist

All necessary documentation has been adapted.
All necessary changes are explained in the llms.txt.
The implemented feature is covered by an appropriate test.

vercel · 2026-07-23T14:38:55Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
lapis	Ready	Preview, Comment	Jul 29, 2026 2:05pm

fhennig · 2026-07-27T08:20:23Z

Code review (high effort)

Reviewed git diff main...HEAD for this branch. One confirmed correctness bug, plus a few lower-severity cleanup findings.

🐞 Correctness bug (CONFIRMED)

orderBy references to computed fields are not case-normalized the way fields entries are — lapis/src/main/kotlin/org/genspectrum/lapis/model/SiloQueryModel.kt:74

OrderByFieldsCleaner.clean (request/OrderByField.kt:148) only delegates to CaseInsensitiveFieldsCleaner.clean, which has no awareness of the <field>.<function> dot syntax and no dot-splitting logic. For a dotted orderBy value it returns null and falls back (?: fieldName) to the raw, un-normalized string the client sent — unlike fields, which always goes through CaseInsensitiveFieldConverter.convert and produces a canonical "$sourceField.${function.saneQlMethodName}" string.

rewriteOrderByForComputedFields's userFacingToAlias[field.field] lookup is an exact, case-sensitive match against that canonical form.

Repro: fields=["date.isoWeek"], orderBy=[{field:"DATE.ISOWEEK"}] (or any casing variant differing from the canonical form). The alias lookup misses, the substitution silently doesn't happen, and the raw string "DATE.ISOWEEK" is emitted verbatim via id() as a literal SaneQL column identifier — a column that doesn't exist in the post-groupBy result (only date/__scalar_isoWeek_date exist), causing an unexpected SILO-side error instead of the case-insensitive behavior the rest of the API guarantees for plain fields.

None of the added tests exercise this path — SiloQueryModelTest constructs the orderBy field directly from isoWeekField.fieldName in memory, bypassing the real cleaner/converter entirely, so the mismatch is untested.

🧹 Cleanup findings (lower severity)

Repeated scans of the same list — SiloQueryModel.kt:43-55. getAggregated calls sequenceFilters.fields.filterIsInstance<Field.Plain>() / filterIsInstance<Field.Computed>() four separate times over the same list instead of partitioning once. Minor per-request cost, and any future change to how computed fields are derived needs updating in three call sites instead of one.
Duplicated lowercase lookup map — request/Field.kt:38-39. fieldTypesByLowercaseName rebuilds the same metadata.associateBy(name.lowercase()) that CaseInsensitiveFieldsCleaner.fieldsMap already builds from the identical source list. Currently safe since both derive from the same source, but the non-null assertion at Field.kt:69 relies on that invariant holding — if the two maps ever drift, a BadRequestException turns into an unhandled NPE.
Computed-field support is bolted onto AggregatedAction only — silo/SiloQuery.kt:229 onward. The computedFields/map-step/alias/rename/orderBy-rewrite logic is wired specifically into AggregatedAction and SiloQueryModel, with other endpoints rejecting computed fields via an ad hoc throw in the controller (LapisController.kt:1494-1499) rather than a shared capability. If scalar functions are later extended to another endpoint, this whole mechanism will likely be copy-pasted rather than reused.

Note: I also checked whether the dot-notation parsing regresses metadata fields whose names literally contain a . (since convert() treats any dotted string as computed-field syntax) — this is not an issue, DatabaseConfigValidator already rejects . in field names at config-load time, predating this branch.

🤖 Generated with Claude Code

Copilot

Pull request overview

Adds support for scalar/computed fields (e.g. date.isoWeek) in the fields parameter for the /aggregated endpoint by introducing a typed Field model, validating scalar functions against a whitelist, generating a pre-groupBy() map() step in SaneQL, and translating internal alias columns back to user-facing keys in the response.

Changes:

Introduces Field.Plain vs Field.Computed plus ScalarFunction whitelist and validation (type-checked).
Extends aggregated SaneQL generation with a map() step for computed fields and adjusts ordering/response key rewriting.
Adds unit + e2e tests and documentation for computed fields; blocks computed fields in /details.

Reviewed changes

Copilot reviewed 22 out of 22 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
lapis/src/main/kotlin/org/genspectrum/lapis/request/Field.kt	Introduces `Field` sealed type and parsing for `<field>.<function>` computed syntax with type validation.
lapis/src/main/kotlin/org/genspectrum/lapis/request/ScalarFunction.kt	Adds scalar function whitelist (currently `isoWeek` for `DATE`).
lapis/src/main/kotlin/org/genspectrum/lapis/request/OrderByField.kt	Canonicalizes `orderBy` fields using the same converter as `fields` (supports computed fields casing).
lapis/src/main/kotlin/org/genspectrum/lapis/silo/SiloQuery.kt	Adds computed-field `map()` step before `groupBy()` and groups by computed aliases.
lapis/src/main/kotlin/org/genspectrum/lapis/model/SiloQueryModel.kt	Splits plain vs computed fields, rewrites computed `orderBy` to aliases, and renames aliases back in responses.
lapis/src/main/kotlin/org/genspectrum/lapis/model/mutationsOverTime/QueriesOverTimeModel.kt	Updates aggregated call site to named parameters after signature expansion.
lapis/src/main/kotlin/org/genspectrum/lapis/controller/LapisController.kt	Rejects computed fields for `/details` requests with a 400.
lapis/src/main/kotlin/org/genspectrum/lapis/controller/ControllerDescriptions.kt	Documents computed-field dot notation in aggregated `fields` OpenAPI description.
lapis/src/test/kotlin/org/genspectrum/lapis/silo/SiloQueryToSaneQlTest.kt	Adds SaneQL rendering test for computed fields (map + groupBy alias).
lapis/src/test/kotlin/org/genspectrum/lapis/silo/SiloQueryTest.kt	Updates aggregated action tests to named parameters.
lapis/src/test/kotlin/org/genspectrum/lapis/request/SequenceFiltersRequestWithFieldsTest.kt	Updates tests to use `Field.Plain`.
lapis/src/test/kotlin/org/genspectrum/lapis/request/ScalarFunctionFieldTest.kt	New tests for computed-field parsing/validation.
lapis/src/test/kotlin/org/genspectrum/lapis/request/OrderByFieldConverterTest.kt	Adds coverage for computed fields in `orderBy` conversion/canonicalization.
lapis/src/test/kotlin/org/genspectrum/lapis/model/SiloQueryModelTest.kt	Adds tests for map-step emission, alias renaming, and orderBy alias rewriting.
lapis/src/test/kotlin/org/genspectrum/lapis/model/mutationsOverTime/Helpers.kt	Updates aggregated action helper call site to named parameters.
lapis/src/test/kotlin/org/genspectrum/lapis/controller/LapisControllerCommonFieldsTest.kt	Updates controller tests to use `Field.Plain`.
lapis/src/test/kotlin/org/genspectrum/lapis/controller/Helpers.kt	Updates request helper to construct `Field.Plain`.
lapis-e2e/test/details.spec.ts	Adds e2e assertion that computed fields are rejected on `/details`.
lapis-e2e/test/aggregated.spec.ts	Adds e2e coverage for computed fields in `/aggregated` including ordering and error cases.
lapis-docs/src/content/docs/references/introduction.mdx	Links to computed fields concept doc.
lapis-docs/src/content/docs/concepts/computed-fields.mdx	New documentation page describing computed fields and available functions.
lapis-docs/astro.config.mjs	Adds “Computed fields” to docs navigation.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

…otation Users can now specify computed fields like `date.isoWeek` in the `fields` parameter of the `/aggregated` endpoint. LAPIS validates the function against a whitelist (currently `isoWeek` for date fields), generates a `map()` step before `groupBy()` in the SaneQL query, and renames the internal alias columns back to the user-facing `field.function` names in the response. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add a doc comment to ScalarFunction explaining new functions must be whitelisted there, and describe the <field>.<function> syntax in the OpenAPI description for the aggregated endpoint's fields parameter. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Cover stratifying and ordering by a computed field (date.isoWeek), bad requests for unknown functions and wrong field types, and that computed fields are rejected on /details. Verified against a real SILO instance. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

Add a concepts page explaining the <field>.<function> dot-notation syntax for the fields parameter (e.g. date.isoWeek), and link it from the references introduction. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

orderBy previously used a plain-field-only cleaner, so a computed field (e.g. "date.isoWeek") with different casing than in `fields` would not be recognized as the same field, silently breaking the alias rewrite and sending an invalid column reference to SILO. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…eries SILO accepts a quoted identifier containing a literal "." as a column name (verified against a live SILO instance), so the computed field's canonical name (e.g. "date.isoWeek") can be used directly as the SaneQL map/groupBy/orderBy column instead of routing through an internal __scalar_<function>_<field> alias. This removes the response-column rename and orderBy-rewrite steps entirely, along with the class of bugs that comes from keeping two names in sync. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

- Change Field from a sealed class with inner Plain/Computed subclasses to a sealed interface with top-level PlainField and ComputedField - ComputedField exposes outputColumnName (e.g. "date.isoWeek") matching the SequencePositionField pattern from main, eliminating the need for a separate internal alias and column-renaming step in SiloQueryModel - Extract dot-notation parsing into ScalarFunctionFieldConverter in request/converter/, mirroring main's converter package structure - Revert SaneQlAssignment to auto-quoting ("name":=value), undoing the temporary workaround; SaneQlAssignment(field.outputColumnName, ...) now produces the correct quoted alias directly - Update all call sites and tests accordingly Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

- PlainFieldConverter: reject dot-notation fields with a clear error ("Scalar functions are not supported in fields for this endpoint") so the /details e2e test gets the expected message - AggregatedFieldConverterTest: pass scalarFunctionFieldConverter in constructor - ScalarFunctionFieldTest: use JUnit assertTrue instead of Kotlin assert (Kotlin assert is a no-op without -ea JVM flag) - ControllerDescriptions: remove backslash-escapes from raw strings (\" in triple-quoted strings renders as literal \", not ") - computed-fields.mdx: apply prettier formatting Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

The page was added to the nav but not to the ordered page list used by the next-button and navigation link tests. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

AggregatedAction already has @JsonInclude(NON_EMPTY), so empty lists are suppressed anyway. The JSON serialization is never sent to SILO (it uses SaneQL), so the annotation served no purpose. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 26 out of 26 changed files in this pull request and generated no new comments.

fengelniederhammer

A couple of minor things, looks good otherwise 👍

fengelniederhammer · 2026-07-30T06:21:38Z

+
+:::note
+Computed fields are only supported in the `fields` parameter of `/sample/aggregated`
+(and the analogous endpoints for other sequence types).


which other endpoint?

fengelniederhammer · 2026-08-06T07:27:43Z

+## Available functions
+
+| Function  | Applicable field types | Description                          |
+| --------- | ---------------------- | ------------------------------------ |
+| `isoWeek` | `date`                 | The ISO 8601 week of the date field. |


Something messed up the table formatting...

fengelniederhammer · 2026-08-06T07:38:02Z

+```
+[URL to LAPIS instance]/sample/aggregated?fields=date.isoWeek
+```
+
+```json
+{
+    "fields": ["date.isoWeek"]
+}
+```


Suggested change

```

[URL to LAPIS instance]/sample/aggregated?fields=date.isoWeek

```

```json

{

"fields": ["date.isoWeek"]

}

```

```

POST /sample/aggregated?fields=date.isoWeek

{

"fields": ["date.isoWeek"]

}

```

I think other examples also already use a format like this?

fengelniederhammer · 2026-08-06T07:39:06Z

+  it('should order by a computed field using dot notation', async () => {
+    const result = await lapisClient.postAggregated({
+      aggregatedPostRequest: {
+        date: '2021-06-05',


really filter by date, too? I think the test would be more meaningful without the filter, but still deterministic due to the orderBy.

fengelniederhammer · 2026-08-06T07:42:09Z

+@Component
+class ScalarFunctionFieldConverter(
+    private val caseInsensitiveFieldsCleaner: CaseInsensitiveFieldsCleaner,
+    private val databaseConfig: DatabaseConfig,


Suggested change

private val databaseConfig: DatabaseConfig,

private databaseConfig: DatabaseConfig,

fengelniederhammer · 2026-08-06T07:49:34Z

+    @Test
+    fun `isoWeek on non-date field throws BadRequestException`() {
+        val ex = assertThrows<BadRequestException> { underTest.tryConvert("country.isoWeek") }
+        assertTrue(ex.message.orEmpty().contains("STRING"), "Expected error to mention type, got: ${ex.message}")


Suggested change

assertTrue(ex.message.orEmpty().contains("STRING"), "Expected error to mention type, got: ${ex.message}")

assertTrue(ex.message.contains("STRING"), "Expected error to mention type, got: ${ex.message}")

According to my IDE, this is redundant.

fengelniederhammer · 2026-08-06T07:49:59Z

+    ),
+)
+
+private val underTest = ScalarFunctionFieldConverter(


Nit: Usually, in Kotlin and Java, those underTests are declared inside the test class.

fhennig self-assigned this Jul 23, 2026

vercel Bot deployed to Preview July 23, 2026 14:39 View deployment

vercel Bot deployed to Preview July 27, 2026 08:10 View deployment

vercel Bot deployed to Preview July 27, 2026 08:33 View deployment

vercel Bot deployed to Preview July 27, 2026 08:55 View deployment

fengelniederhammer requested a review from Copilot July 27, 2026 08:57

Copilot started reviewing on behalf of fengelniederhammer July 27, 2026 08:57 View session

Copilot AI reviewed Jul 27, 2026

View reviewed changes

Comment thread lapis/src/test/kotlin/org/genspectrum/lapis/request/ScalarFunctionFieldTest.kt

Comment thread lapis/src/main/kotlin/org/genspectrum/lapis/controller/ControllerDescriptions.kt Outdated

vercel Bot deployed to Preview July 27, 2026 09:19 View deployment

Felix Hennig and others added 7 commits July 29, 2026 11:05

docs(lapis-docs): document computed fields syntax

9dc7cdd

Add a concepts page explaining the <field>.<function> dot-notation syntax for the fields parameter (e.g. date.isoWeek), and link it from the references introduction. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

fhennig force-pushed the feature/scalar-functions-in-aggregation-fields branch from 50e05da to 2092281 Compare July 29, 2026 09:32

vercel Bot deployed to Preview July 29, 2026 09:32 View deployment

vercel Bot deployed to Preview July 29, 2026 12:42 View deployment

test(lapis-docs): add computed-fields page to docs navigation test

b7745e0

The page was added to the nav but not to the ordered page list used by the next-button and navigation link tests. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

vercel Bot deployed to Preview July 29, 2026 12:58 View deployment

chore(lapis-e2e): apply prettier formatting to details.spec.ts

fb28838

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

vercel Bot deployed to Preview July 29, 2026 13:05 View deployment

fhennig marked this pull request as ready for review July 29, 2026 13:15

vercel Bot deployed to Preview July 29, 2026 13:59 View deployment

docs(lapis): document computed fields in llms.txt

421d832

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

fhennig requested a review from fengelniederhammer July 29, 2026 14:04

vercel Bot deployed to Preview July 29, 2026 14:05 View deployment

fengelniederhammer requested a review from Copilot July 30, 2026 06:20

Copilot started reviewing on behalf of fengelniederhammer July 30, 2026 06:21 View session

Copilot AI reviewed Jul 30, 2026

View reviewed changes

fengelniederhammer approved these changes Aug 6, 2026

View reviewed changes

Navigation Menu

Uh oh!

feat(lapis): support scalar functions in aggregation fields via dot notation - #1780

feat(lapis): support scalar functions in aggregation fields via dot notation#1780
fhennig wants to merge 12 commits into
mainfrom
feature/scalar-functions-in-aggregation-fields

fhennig commented Jul 23, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Jul 23, 2026 •

edited

Loading

Uh oh!

fhennig commented Jul 27, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

fengelniederhammer left a comment

Uh oh!

fengelniederhammer Jul 30, 2026

Uh oh!

fengelniederhammer Aug 6, 2026

Uh oh!

fengelniederhammer Aug 6, 2026

Uh oh!

fengelniederhammer Aug 6, 2026

Uh oh!

fengelniederhammer Aug 6, 2026

Uh oh!

fengelniederhammer Aug 6, 2026

Uh oh!

fengelniederhammer Aug 6, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	private val databaseConfig: DatabaseConfig,
	private databaseConfig: DatabaseConfig,

	assertTrue(ex.message.orEmpty().contains("STRING"), "Expected error to mention type, got: ${ex.message}")
	assertTrue(ex.message.contains("STRING"), "Expected error to mention type, got: ${ex.message}")

Uh oh!

Conversation

fhennig commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

PR Checklist

Uh oh!

vercel Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fhennig commented Jul 27, 2026

Code review (high effort)

🐞 Correctness bug (CONFIRMED)

🧹 Cleanup findings (lower severity)

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

fengelniederhammer left a comment

Choose a reason for hiding this comment

Uh oh!

fengelniederhammer Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

fengelniederhammer Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

fengelniederhammer Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

fengelniederhammer Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

fengelniederhammer Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

fengelniederhammer Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

fengelniederhammer Aug 6, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

fhennig commented Jul 23, 2026 •

edited

Loading

vercel Bot commented Jul 23, 2026 •

edited

Loading