diff --git a/docs/configuration/metastore-config.md b/docs/configuration/metastore-config.md index 05ed0a6b2fd..0b03a7f1e0b 100644 --- a/docs/configuration/metastore-config.md +++ b/docs/configuration/metastore-config.md @@ -43,6 +43,12 @@ On its first execution, Quickwit will transparently create the necessary tables. Likewise, if you upgrade Quickwit to a version that includes some changes in the PostgreSQL schema, Quickwit will transparently operate the migration startup. +## PostgreSQL connection pool + +Each Quickwit node running the `metastore` service maintains its own PostgreSQL connection pool. Database-backed metastore nodes admit at most `2 * metastore.postgres.max_connections` in-flight requests, so the default is `20`. + +When sizing the pool, keep `metastore_nodes * metastore.postgres.max_connections` below the PostgreSQL connection limit with operational headroom. If Quickwit logs `pool timed out while waiting for an open connection`, check `quickwit_metastore_active_connections`, `quickwit_metastore_idle_connections`, and `quickwit_metastore_acquire_connections`. + # File-backed metastore For convenience, Quickwit also makes it possible to store its metadata in files using a file-backed metastore. In that case, Quickwit will write one file per index. diff --git a/docs/configuration/node-config.md b/docs/configuration/node-config.md index 66da698665b..fdcd26054d3 100644 --- a/docs/configuration/node-config.md +++ b/docs/configuration/node-config.md @@ -216,6 +216,8 @@ File-backed metastore doesn't have any node level configuration. You can configu | `idle_connection_timeout` | Maximum idle duration before closing individual connections. | `10min` | | `max_connection_lifetime` | Maximum lifetime of individual connections. | `30min` | +`max_connections` applies per Quickwit node running the `metastore` service. Database-backed metastore nodes admit at most `2 * max_connections` in-flight requests. + Example of a metastore configuration for PostgreSQL in YAML format: ```yaml diff --git a/docs/reference/metrics.md b/docs/reference/metrics.md index 7d6b69fa7e7..03993b1c9d4 100644 --- a/docs/reference/metrics.md +++ b/docs/reference/metrics.md @@ -60,6 +60,14 @@ All metastore methods are monitored by the 3 metrics: Examples of operation names: `create_index`, `index_metadata`, `delete_index`, `stage_splits`, `publish_splits`, `list_splits`, `add_source`, ... +PostgreSQL-backed metastores also expose connection pool gauges: + +| Namespace | Metric Name | Description | Type | +| --------- | ----------- | ----------- | ---- | +| `quickwit_metastore` | `active_connections` | Number of active PostgreSQL pool connections, including used and idle connections | `gauge` | +| `quickwit_metastore` | `idle_connections` | Number of idle PostgreSQL pool connections | `gauge` | +| `quickwit_metastore` | `acquire_connections` | Number of requests currently waiting to acquire a PostgreSQL pool connection | `gauge` | + ## Rest API Metrics | Namespace | Metric Name | Description | Type | diff --git a/quickwit/quickwit-serve/src/lib.rs b/quickwit/quickwit-serve/src/lib.rs index e502363aaa3..a93033acde3 100644 --- a/quickwit/quickwit-serve/src/lib.rs +++ b/quickwit/quickwit-serve/src/lib.rs @@ -80,7 +80,7 @@ use quickwit_compaction::{ wait_for_compactor_decommission, }; use quickwit_config::service::QuickwitService; -use quickwit_config::{ClusterConfig, IngestApiConfig, NodeConfig}; +use quickwit_config::{ClusterConfig, IngestApiConfig, NodeConfig, PostgresMetastoreConfig}; use quickwit_control_plane::control_plane::{ControlPlane, ControlPlaneEventSubscriber}; use quickwit_control_plane::{IndexerNodeInfo, IndexerPool}; use quickwit_index_management::{IndexService as IndexManager, IndexServiceError}; @@ -587,12 +587,12 @@ pub async fn serve_quickwit( ) })?; let max_in_flight_requests = if node_config.metastore_uri.protocol().is_database() { - node_config + let max_connections = node_config .metastore_configs .find_postgres() - .map(|config| config.max_connections.get() * 2) - .unwrap_or_default() - .max(100) + .map(|config| config.max_connections) + .unwrap_or_else(PostgresMetastoreConfig::default_max_connections); + max_connections.get().saturating_mul(2) } else { 100 };