Add a metastore read replica role for read-only routing by shuheiktgw · Pull Request #6548 · quickwit-oss/quickwit

shuheiktgw · 2026-06-25T06:05:20Z

Description

This PR adds support for deploying metastore read replicas as a dedicated metastore_read_replica service and lets searchers opt into using them through searcher.use_metastore_read_replica.

How it works

Users deploy standalone metastore_read_replica nodes with metastore_read_replica_uri pointing to a PostgreSQL read replica. If searcher.use_metastore_read_replica is true, searchers use metastore_read_replica instead of the primary metastore.
Searchers use a dedicated read-only metastore client, MetastoreReadServiceClient, which exposes only the read-only subset of metastore RPCs to prevent write methods from being used.

Limitations / Future Improvements

metastore_read_replica needs to run as a standalone service to keep the implementation simple. It shares the same gRPC service as metastore, so it cannot run alongside it, and running it separately also simplifies the metadata service connection handling.
Currently, searchers configured to use metastore read replicas still need access to the primary metastore. Quickwit treats every node as an API server, and some API requests need to write to the metastore. However, the readiness check for the read-only metastore only verifies whether the read replicas are accessible. This means that if the primary database is down, the read-only metastore may be unable to access it and could drop write requests. We decided that this is the right trade-off: it separates the search path from the write path and keeps searchers alive during primary DB degradation.
There are many more places where we could use the read replica metastore, but the primary focus of this PR is the searcher. We can expand support further later.

shuheiktgw · 2026-06-26T07:19:18Z

+#   # DataFusion when enabled, to nodes running the `metastore_read_replica`
+#   # service. Searchers require at least one `metastore_read_replica` node at
+#   # startup and do not fall back to the primary metastore.
+#   use_metastore_read_replica: false


The reason we need this flag in addition to metastore_read_replica_uri is that users can set metastore_read_replica_uri via the QW_METASTORE_READ_REPLICA_URI environment variable. Without this flag, searchers would need access to that same environment variable even though they don’t need the secret itself.

Introduce the `metastore_read_replica` service role and the `metastore_read_replica_uri` node config option: the foundation for routing read-only metastore traffic to a PostgreSQL read replica. - Add `QuickwitService::MetastoreReadReplica`, parsed from `metastore_read_replica` / `metastore-read-replica`. - Split `QuickwitService::default_services()` from `supported_services()` so the new role is opt-in and never enabled implicitly on all-in-one nodes. - Add the optional `metastore_read_replica_uri` field (env `QW_METASTORE_READ_REPLICA_URI`), redacted alongside `metastore_uri`. - Validate that the role requires a PostgreSQL `metastore_read_replica_uri` and cannot be co-located with the `metastore` role.

Add the resolution plumbing for connecting to a PostgreSQL read replica over a read-only connection. - Add `MetastoreFactoryOptions { read_only }` and thread it through `MetastoreFactory::resolve`. - Add `MetastoreResolver::resolve_read_only`, which rejects any non- PostgreSQL backend. - Key the PostgreSQL factory cache on `(uri, options)` so the read-write and read-only clients get distinct connection pools. - Add `PostgresqlMetastore::new_read_only`: a read-only connection pool with migrations skipped (the replica is migrated by the primary).

Resolve and expose a metastore gRPC server when the `metastore_read_replica` role is enabled, backed by a read-only connection to `metastore_read_replica_uri`. - The read replica server reuses the metrics + load-shed layers but omits the control-plane event layers, which only wrap write RPCs. - A read-replica node is exempted from the control-plane connectivity wait, like a primary metastore node, so dedicated replica pods start independently. - Extract `metastore_max_in_flight_requests` shared by both roles.

Add `ReadReplicaRoutingMetastore`, a `MetastoreService` wrapper that routes the stale-tolerant reads issued by the search and analytics paths (`index_metadata`, `indexes_metadata`, `list_indexes_metadata`, `list_splits`, `list_metrics_splits`, `list_sketch_splits`) to read replica nodes when any are connected, and everything else (writes and non-hot-path reads) to the primary. - Routing is decided per request from the read replica balance channel's live connection set, so it degrades to the primary when no replica is deployed. The check is a synchronous `watch` read with no borrow held across an await. - Wire it into the searcher service and the DataFusion session builder, so all search (REST, Elasticsearch, gRPC) and metrics analytics benefit, while REST admin handlers keep read-your-writes against the primary.

- Document `metastore_read_replica_uri` in the node config reference and the example `quickwit.yaml`. - Add a PostgreSQL-gated test that resolves a read-only metastore against a real database and verifies it serves read RPCs.

Replace the full `MetastoreService` implementation on `ReadReplicaRoutingMetastore` (which forced ~45 delegating methods, most of them writes) with a narrow, read-only `MetastoreReadService` trait — the read-only subset of the metastore RPCs (`index_metadata`, `list_indexes_metadata`, `list_splits`, `list_metrics_splits`, `list_sketch_splits`). - `MetastoreServiceClient` implements `MetastoreReadService`; `MetastoreReadServiceClient = Arc<dyn MetastoreReadService>`. - `ReadReplicaRoutingMetastore` now implements only the 5-method trait, so writes are excluded at the type level rather than delegated. - The search and DataFusion read paths take `MetastoreReadServiceClient` / `&dyn MetastoreReadService`; `list_parquet_splits_*` take `&dyn MetastoreReadService`. `single_node_search` keeps its concrete `MetastoreServiceClient` parameter and adapts internally. Addresses review feedback that the wrapper should expose a Go-style read-only interface instead of reimplementing the whole service.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 09c32ada6a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c319f00dfd

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

shuheiktgw · 2026-07-10T16:38:16Z

I refactored the metastore initialization logic by introducing a LocalMetastoreServer enum and start_metastore_service_if_needed, with the goal of keeping all the related logic in one place.

1ebe477

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 1ebe477999

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

shuheiktgw · 2026-07-10T16:44:09Z

We discussed yesterday that the startup of the read-only metastore is unnecessarily blocked by the primary metastore. And yes, the read-only metastore does wait up to 300s for a metastore node before mounting the read-only metastore service.

But I took a closer look and realized this is actually fine, because a metastore read-replica node still exposes the API server, and some API endpoints need access to the primary metastore for writes. Readiness itself is still based on the read-replica metastore once the node has started.

WDYT?

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e8d51a982a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

guilload

I think this works but unfortunately, the code was not written in a way that optimizes for "proof readability". The logic in quickwit/quickwit-serve/src/lib.rs needs more logging.

guilload · 2026-07-14T15:38:44Z

-            sql.to_string(PostgresQueryBuilder),
-            r#"SELECT * FROM "splits" WHERE split_id <> ALL(ARRAY ['s1','s2']::text[])"#
+        let sql = sql.to_string(PostgresQueryBuilder);
+        assert!(


The "rendering" of the query should be made deterministic.

guilload · 2026-07-14T15:41:17Z

        };
+        if options.read_only && backend != MetastoreBackend::PostgreSQL {
+            return Err(MetastoreResolverError::UnsupportedBackend(
+                "read-only metastore connections are only supported for PostgreSQL".to_string(),


Suggested change

"read-only metastore connections are only supported for PostgreSQL".to_string(),

"read-only option is only supported for the PostgreSQL backend".to_string(),

guilload · 2026-07-14T15:44:10Z

                    }
                }
+                (Some(pg_error_codes::READ_ONLY_SQL_TRANSACTION), _) => MetastoreError::Forbidden {
+                    message: format!("postgresql metastore is read-only: {boxed_db_error}"),


Suggested change

message: format!("postgresql metastore is read-only: {boxed_db_error}"),

message: format!(""cannot issue write request on a read-only replica": {boxed_db_error}"),

guilload · 2026-07-14T15:48:34Z

    }
 }

+/// The metastore gRPC server this node serves locally — or `NotServed` if it serves none.


We need to stop the unbounded growth of this file. Move the new metastore-related logic to a new file.

guilload · 2026-07-14T15:56:46Z

    }

+    #[tokio::test]
+    async fn test_readiness_uses_read_replica_without_requiring_primary() {


This doesn't test anything new, is it?

guilload · 2026-07-14T16:02:20Z

+        .resolve_read_only_client(&cluster, &node_config)
+        .await?;
+
+    let metastore_read_client = read_only_metastore_client_opt


The naming is bad. This is not always a read-only client and as result later when we pass it to the readiness tasks, it's hard to know against which metastore the readiness task is running.

shuheiktgw force-pushed the metastore-read-replica-role branch 2 times, most recently from c0ae4dd to d14a422 Compare June 25, 2026 16:13

shuheiktgw commented Jun 26, 2026

View reviewed changes

shuheiktgw added 16 commits June 26, 2026 17:15

Avoid primary fallback for read replica routing

e50cd29

Fix clippy warning in read replica config validation

f405a5f

Add searcher metastore read replica setting

902134b

Remove config URI validation for read replica

2b0308b

Fix metastore gRPC mount comment

4832972

Update metastore readiness checks

ae8f916

Require metastore read replica standalone role

1828871

Remove unnecessary unit tests

150174e

Refine metastore read replica wiring

2002587

Fix default service startup in tests

09c32ad

shuheiktgw force-pushed the metastore-read-replica-role branch from 1bf29fc to 09c32ad Compare June 26, 2026 15:16

shuheiktgw marked this pull request as ready for review June 26, 2026 15:28

shuheiktgw requested review from a team as code owners June 26, 2026 15:28

chatgpt-codex-connector Bot reviewed Jun 26, 2026

View reviewed changes

Comment thread quickwit/quickwit-serve/src/lib.rs Outdated

Comment thread quickwit/quickwit-serve/src/lib.rs

guilload reviewed Jun 30, 2026

View reviewed changes

shuheiktgw added 5 commits July 6, 2026 12:22

Simplify metastore read replica validation

6c0c2d6

Trim metastore read replica config tests

651617f

Classify read-only Postgres errors

a513bbb

Merge origin/main into metastore-read-replica-role

c86f3d8

Fix search test clippy warnings

5d8b9e3

shuheiktgw added 3 commits July 6, 2026 17:55

Simplify readiness test mock setup

2d7e71e

Remove metastore read service

3e6e756

Merge origin/main into metastore-read-replica-role

43f6074

guilload reviewed Jul 8, 2026

View reviewed changes

shuheiktgw added 9 commits July 10, 2026 09:08

Merge origin/main into metastore-read-replica-role

d1b69dd

Default metastore read replica routing to false

72f623b

Update Quickwit env var index test

1b98c50

Exclude read replica role from TLS test nodes

66a0652

Use ensure for metastore checks

d895d39

Simplify read replica URI assertion

2f7bc69

Test read-only Postgres writes

305759a

Use service name in metastore connection log

56dcfaa

Use read replica metastore directly

ab0720f

shuheiktgw force-pushed the metastore-read-replica-role branch from af5a9dc to c319f00 Compare July 10, 2026 12:13

chatgpt-codex-connector Bot reviewed Jul 10, 2026

View reviewed changes

Comment thread quickwit/quickwit-serve/src/lib.rs Outdated

Comment thread quickwit/quickwit-serve/src/lib.rs Outdated

Fix read replica metastore readiness

74c9efe

shuheiktgw force-pushed the metastore-read-replica-role branch from c319f00 to 74c9efe Compare July 10, 2026 12:37

Refactor local metastore server handling

1ebe477

shuheiktgw force-pushed the metastore-read-replica-role branch from 5746393 to 1ebe477 Compare July 10, 2026 16:29

chatgpt-codex-connector Bot reviewed Jul 10, 2026

View reviewed changes

Comment thread quickwit/quickwit-serve/src/lib.rs

Validate metastore read replica searcher config

e8d51a9

chatgpt-codex-connector Bot reviewed Jul 10, 2026

View reviewed changes

Comment thread quickwit/quickwit-config/src/node_config/serialize.rs

guilload reviewed Jul 14, 2026

View reviewed changes

	"read-only metastore connections are only supported for PostgreSQL".to_string(),
	"read-only option is only supported for the PostgreSQL backend".to_string(),

	message: format!("postgresql metastore is read-only: {boxed_db_error}"),
	message: format!(""cannot issue write request on a read-only replica": {boxed_db_error}"),

Uh oh!

Conversation

shuheiktgw commented Jun 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

How it works

Limitations / Future Improvements

Uh oh!

shuheiktgw Jun 26, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

Uh oh!

shuheiktgw commented Jul 10, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

shuheiktgw commented Jul 10, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

guilload left a comment

Choose a reason for hiding this comment

Uh oh!

guilload Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

guilload Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

guilload Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

guilload Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

guilload Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

guilload Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

shuheiktgw commented Jun 25, 2026 •

edited

Loading