feat(fhirpath): storage-backed resolve() for server-stored resources (#167)#182
Open
mauripunzueta wants to merge 2 commits into
Open
feat(fhirpath): storage-backed resolve() for server-stored resources (#167)#182mauripunzueta wants to merge 2 commits into
mauripunzueta wants to merge 2 commits into
Conversation
Follow-up to #166 (#167). resolve() previously dereferenced only contained resources and resources already in the evaluation context; a relative Type/id reference to a resource living in a storage backend fell back to a typed stub. This adds eager, tenant-scoped pre-hydration so server-side view evaluation can dereference stored references — e.g. Observation.subject.resolve().name. Approach A (eager pre-hydration), mirroring the existing trusted-server remote-resolve() prefetch but sourcing from ResourceStorage instead of HTTP. The evaluator stays synchronous: all I/O happens in async code before the engine runs. - helios-persistence: new sof::reference_resolver module — a pure relative-ref scanner (collect_missing_references) plus a StorageReferenceResolver trait and a StorageBackedResolver over any ResourceStorage. Tenant-scoped (read_batch), FHIR-version-matched, fan-out capped, deduped, one level deep, no network (relative Type/id only — absolute URLs / urn: / fragments excluded). - InProcessSofRunner gains an optional resolver (with_reference_resolver): per run it collects the scanned resources' missing references, prefetches them, and folds them into the resolution pool via process_chunk_with_external. Absent resolver → unchanged stub/empty fallback. - Wired into the S3 backend's in-process runner (the FHIRPath-engine path). The SQL runners (SQLite/PG) don't compile resolve() — that remains Uncompilable/422 and is out of scope here. - helios-sof: expose parse_json_to_fhir_resource_pub (was pub(crate)) so the persistence prefetch can build the typed external pool. Tests: pure-scanner unit tests; StorageBackedResolver integration tests vs SQLite (stored hit, cross-tenant miss, not-found fallback, version match); end-to-end runner tests proving resolve() dereferences a stored Patient during a view run and that the no-resolver path is unchanged. Full persistence lib suite (672) green; clippy clean.
|
Helios FHIR ServerThe Helios FHIR Server is an implementation of the HL7® FHIR® standard, built in Rust for high performance and optimized for clinical analytics workloads. It provides modular components that can be run as standalone command-line tools, integrated as microservices, or embedded directly into your data analytics pipeline. Why Helios FHIR Server?
What People Build with the Helios FHIR Server
Quick StartThe Helios FHIR Server includes several components:
See Core Components for details on each. The server supports SQLite, PostgreSQL, Elasticsearch, and S3 in various configurations — see Storage Backends for setup options. Using Release BinariesPre-built binaries are available on the GitHub Releases page. Download the appropriate archive for your platform and extract it.
The following are independent examples showing how to run each executable — pick whichever ones apply to your use case: # FHIR server (access at http://localhost:8080/metadata)
./hfs
# FHIR Terminology Server (access at http://localhost:8090/metadata)
./hts
# Evaluate a FHIRPath expression
echo '{"resourceType": "Patient", "id": "123"}' | ./fhirpath-cli 'Patient.id'
# Transform FHIR Bundle to CSV using SQL-on-FHIR
./sof-cli --view examples/patient-view.json --bundle examples/patients.json
# Transform NDJSON file to CSV using SQL-on-FHIR
./sof-cli --view examples/patient-view.json --bundle examples/patients.ndjson
# SQL-on-FHIR HTTP server (POST to http://localhost:8080/ViewDefinition/$viewdefinition-run)
./sof-server
# FHIRPath HTTP server (POST expressions to http://localhost:3000/fhirpath)
./fhirpath-serverUsing Docker ImagesPre-built multi-arch Docker images (amd64/arm64) are available on GitHub Container Registry. # FHIR Server (default: R4, in-memory SQLite, port 8080)
docker run -p 8080:8080 ghcr.io/heliossoftware/hfs:latest
# With persistent SQLite storage
docker run -p 8080:8080 -v hfs-data:/data -e HFS_DATABASE_URL=/data/fhir.db ghcr.io/heliossoftware/hfs:latest
# With PostgreSQL
docker run -p 8080:8080 \
-e HFS_STORAGE_BACKEND=postgres \
-e HFS_DATABASE_URL="postgresql://user:pass@host:5432/fhir" \
ghcr.io/heliossoftware/hfs:latest
# FHIRPath Server (port 3000)
docker run -p 3000:3000 ghcr.io/heliossoftware/fhirpath-server:latest
# SQL-on-FHIR Server (port 8080)
docker run -p 8080:8080 ghcr.io/heliossoftware/sof-server:latest
# FHIR Terminology Server (default: SQLite, port 8090)
docker run -p 8090:8090 ghcr.io/heliossoftware/hts:latest
# HTS with persistent SQLite storage
docker run -p 8090:8090 -v hts-data:/data -e HTS_DATABASE_URL=/data/hts.db ghcr.io/heliossoftware/hts:latestSee Environment Variables for all available configuration options. Building From SourcePrerequisites
💡 Tip: If you run out of memory during compilation on Linux, especially on high CPU core count machines, limit parallel jobs to 4 (or less): Build and Install# Clone the repository
git clone https://github.com/HeliosSoftware/hfs.git
cd hfs
# Build (R4 only by default). Uses workspace default-members and skips the Python bindings crate (pysof).
cargo build --release
# Or build with all FHIR versions
cargo build --release --features R4,R4B,R5,R6
# Build all workspace members (including pysof)
cargo build --workspace --release
Storage BackendsThe Helios FHIR Server supports multiple storage backend configurations. Choose a configuration based on your search requirements and deployment scale. Available Configurations
Running the Server# SQLite (default) — no external dependencies
./hfs
# SQLite + Elasticsearch
HFS_STORAGE_BACKEND=sqlite-elasticsearch \
HFS_ELASTICSEARCH_NODES=http://localhost:9200 \
./hfs
# PostgreSQL
HFS_STORAGE_BACKEND=postgres \
HFS_DATABASE_URL="postgresql://user:pass@localhost:5432/fhir" \
./hfs
# PostgreSQL + Elasticsearch
HFS_STORAGE_BACKEND=postgres-elasticsearch \
HFS_DATABASE_URL="postgresql://user:pass@localhost:5432/fhir" \
HFS_ELASTICSEARCH_NODES=http://localhost:9200 \
./hfs
# S3 (requires AWS credentials via standard provider chain:
# https://docs.aws.amazon.com/sdkref/latest/guide/standardized-credentials.html)
HFS_STORAGE_BACKEND=s3 \
HFS_S3_BUCKET=my-fhir-bucket \
AWS_PROFILE=your-aws-profile \
AWS_REGION=us-east-1 \
./hfs
# S3 + Elasticsearch
HFS_STORAGE_BACKEND=s3-elasticsearch \
HFS_S3_BUCKET=my-fhir-bucket \
HFS_ELASTICSEARCH_NODES=http://localhost:9200 \
AWS_PROFILE=your-aws-profile \
AWS_REGION=us-east-1 \
./hfsEnvironment VariablesServer
Limits & behavior
Request bodies may be sent compressed ( CORS
Multi-tenancy
Terminology
Elasticsearch
PostgreSQL (used to assemble a connection when
MongoDB
S3
Authentication & SMART-on-FHIR JWT/bearer auth ( Audit, subscriptions & bulk export
The SMTP delivery channel ( For detailed backend setup instructions (building from source, Docker commands, and search offloading architecture), see the persistence crate documentation. Architecture OverviewThe Helios FHIR Server is organized as a Rust workspace with modular components that can be used independently or together. Each component is designed for high performance and can be embedded directly into your data analytics pipeline. Core Components1.
|
Code review1 issue found. Bug: fan-out cap drops whole resource types based on alphabetical sort orderFile: hfs/crates/persistence/src/sof/reference_resolver.rs Lines 99 to 106 in 5db2bc0 The cap is applied as a plain prefix slice of Concrete example: with Suggested fix: cap per-type after grouping, not on the flat sorted list. For example, distribute the budget evenly across types so no single type crowds out others due to alphabetical sort order. |
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
CLAUDE.mdThis file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. Architecture OverviewWorkspace StructureThe project is a Rust workspace with 17 crates (16 default-members;
Binaries
Key Design PatternsVersion-Agnostic AbstractionThe codebase uses enum wrappers and traits to handle multiple FHIR versions: // Example from sof crate
pub enum SofViewDefinition {
R4(fhir::r4::ViewDefinition),
R4B(fhir::r4b::ViewDefinition),
R5(fhir::r5::ViewDefinition),
R6(fhir::r6::ViewDefinition),
}Trait-Based ProcessingCore functionality is defined through traits, allowing version-independent logic:
Persistence Trait HierarchyStorage backends implement a progressive trait hierarchy: Tenant-First DesignAll persistence operations take a Composite StorageThe Project SkillsDetailed operational guidance lives in project skills under
Environment SetupLLD Linker ConfigurationAdd to [target.x86_64-unknown-linux-gnu]
linker = "clang"
rustflags = ["-C", "link-arg=-fuse-ld=lld"]Memory-Constrained Buildsexport CARGO_BUILD_JOBS=4Debugging Tips
Important Notes
|
AGENTS.mdThis file provides guidance to Codex when working with code in this repository. Architecture OverviewWorkspace StructureThe project is a Rust workspace with 17 crates (16 default-members;
Binaries
Key Design PatternsVersion-Agnostic AbstractionThe codebase uses enum wrappers and traits to handle multiple FHIR versions: // Example from sof crate
pub enum SofViewDefinition {
R4(fhir::r4::ViewDefinition),
R4B(fhir::r4b::ViewDefinition),
R5(fhir::r5::ViewDefinition),
R6(fhir::r6::ViewDefinition),
}Trait-Based ProcessingCore functionality is defined through traits, allowing version-independent logic:
Persistence Trait HierarchyStorage backends implement a progressive trait hierarchy: Tenant-First DesignAll persistence operations take a Composite StorageThe Project SkillsDetailed operational guidance lives in Codex project skills under
Environment SetupLLD Linker ConfigurationAdd to [target.x86_64-unknown-linux-gnu]
linker = "clang"
rustflags = ["-C", "link-arg=-fuse-ld=lld"]Memory-Constrained Buildsexport CARGO_BUILD_JOBS=4Debugging Tips
Important Notes
|
…2026-0185
Two ambient CI breakages on origin/main, unrelated to this PR's changes:
- clippy 1.91 newly flags collapsible_else_if in sof/emit.rs; collapse the
nested else { if .. } into else if.
- cargo audit fails on RUSTSEC-2026-0185 (quinn-proto remote memory
exhaustion), a transitive reqwest QUIC dep. We never accept inbound QUIC,
so the reassembly path is unreachable; ignore it with justification.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #167 (follow-up to #166).
resolve()previously dereferenced onlycontainedresources and resources already in the evaluation context; a relativeType/idreference to a resource living in a storage backend fell back to a typed stub{resourceType: Type}. This adds eager, tenant-scoped pre-hydration so server-side view evaluation can dereference stored references — e.g.Observation.subject.resolve().name.Approach
A — eager pre-hydration (the issue's recommended starting point), mirroring the existing trusted-server remote-
resolve()prefetch (helios_sof::remote_fetch) but sourcing resources from a tenant-scopedResourceStorageinstead of HTTP. The evaluator stays synchronous: all storage I/O happens in async code before the engine runs — noblock_onon async worker threads, no evaluator changes.What changed
helios-persistence— newsof::reference_resolvermodule:collect_missing_references— a pure, order-stable scanner that extracts the distinct relativeType/idreferences in the resources under evaluation, excluding those already in scope.StorageReferenceResolvertrait +StorageBackedResolverover anyResourceStorage: tenant-scoped (read_batch), FHIR-version-matched, fan-out capped (DEFAULT_MAX_FANOUT), deduped, one level deep, no network (relativeType/idonly — absolute URLs /urn:/#fragmentexcluded; absolute-URL resolution remains the separate allowlisted concern ofremote_resolver).InProcessSofRunnergains an optional resolver (with_reference_resolver): per run it collects the scanned resources' missing references, prefetches them, and folds them into the FHIRPath resolution pool viaprocess_chunk_with_external. With no resolver, behavior is unchanged (stub/empty fallback).S3Backendserves as its own tenant-scoped resolver.helios-sof: exposeparse_json_to_fhir_resource_pub(waspub(crate)) so the persistence prefetch can build the typedexternalpool.Scope boundary
This targets the in-process FHIRPath engine path (S3 and S3-primary composite backends), which is where views are evaluated by the engine. The SQLite/PostgreSQL in-DB runners compile views to SQL and do not compile
resolve()— such views are alreadyUncompilable→422there; loweringresolve()to SQL joins is a separate, much larger feature out of scope for #167.Acceptance criteria
Type/idreference to a stored resource (tenant-scoped) —resolves_stored_reference_during_view_run,resolves_stored_resource_for_owning_tenant.without_resolver_reference_is_not_dereferenced; the runner defaults to no resolver.block_onon async worker threads — prefetch is async, beforespawn_blocking.does_not_resolve_across_tenants.Testing
Pure-scanner unit tests (5),
StorageBackedResolverintegration tests vs SQLite (4), end-to-end runner tests (2). Fullhelios-persistencelib suite (672) green;cargo fmtclean;cargo clippy --all-targets -D warningsclean onhelios-persistenceandhelios-sof.🤖 Generated with Claude Code