Metastore inflight default

docs/configuration/metastore-config.md

@@ -43,6 +43,12 @@ On its first execution, Quickwit will transparently create the necessary tables.
 
 Likewise, if you upgrade Quickwit to a version that includes some changes in the PostgreSQL schema, Quickwit will transparently operate the migration startup.
 
+## PostgreSQL connection pool
+
+Each Quickwit node running the `metastore` service maintains its own PostgreSQL connection pool. Database-backed metastore nodes admit at most `2 * metastore.postgres.max_connections` in-flight requests, so the default is `20`.
+
+When sizing the pool, keep `metastore_nodes * metastore.postgres.max_connections` below the PostgreSQL connection limit with operational headroom. If Quickwit logs `pool timed out while waiting for an open connection`, check `quickwit_metastore_active_connections`, `quickwit_metastore_idle_connections`, and `quickwit_metastore_acquire_connections`.
+
 # File-backed metastore
 
 For convenience, Quickwit also makes it possible to store its metadata in files using a file-backed metastore. In that case, Quickwit will write one file per index.

docs/configuration/node-config.md

@@ -216,6 +216,8 @@ File-backed metastore doesn't have any node level configuration. You can configu
 | `idle_connection_timeout` | Maximum idle duration before closing individual connections. | `10min` |
 | `max_connection_lifetime` | Maximum lifetime of individual connections. | `30min` |
 
+`max_connections` applies per Quickwit node running the `metastore` service. Database-backed metastore nodes admit at most `2 * max_connections` in-flight requests.
+
 Example of a metastore configuration for PostgreSQL in YAML format:
 
 ```yaml

docs/reference/metrics.md

@@ -60,6 +60,14 @@ All metastore methods are monitored by the 3 metrics:
 
 Examples of operation names: `create_index`, `index_metadata`, `delete_index`, `stage_splits`, `publish_splits`, `list_splits`, `add_source`, ...
 
+PostgreSQL-backed metastores also expose connection pool gauges:
+
+| Namespace | Metric Name | Description | Type |
+| --------- | ----------- | ----------- | ---- |
+| `quickwit_metastore` | `active_connections` | Number of active PostgreSQL pool connections, including used and idle connections | `gauge` |
+| `quickwit_metastore` | `idle_connections` | Number of idle PostgreSQL pool connections | `gauge` |
+| `quickwit_metastore` | `acquire_connections` | Number of requests currently waiting to acquire a PostgreSQL pool connection | `gauge` |
+
 ## Rest API Metrics
 
 | Namespace | Metric Name | Description | Type |

quickwit/quickwit-serve/src/lib.rs

@@ -80,7 +80,7 @@ use quickwit_compaction::{
     wait_for_compactor_decommission,
 };
 use quickwit_config::service::QuickwitService;
-use quickwit_config::{ClusterConfig, IngestApiConfig, NodeConfig};
+use quickwit_config::{ClusterConfig, IngestApiConfig, NodeConfig, PostgresMetastoreConfig};
 use quickwit_control_plane::control_plane::{ControlPlane, ControlPlaneEventSubscriber};
 use quickwit_control_plane::{IndexerNodeInfo, IndexerPool};
 use quickwit_index_management::{IndexService as IndexManager, IndexServiceError};
@@ -587,12 +587,12 @@ pub async fn serve_quickwit(
                     )
                 })?;
             let max_in_flight_requests = if node_config.metastore_uri.protocol().is_database() {
-                node_config
+                let max_connections = node_config
                     .metastore_configs
                     .find_postgres()
-                    .map(|config| config.max_connections.get() * 2)
-                    .unwrap_or_default()
-                    .max(100)
+                    .map(|config| config.max_connections)
+                    .unwrap_or_else(PostgresMetastoreConfig::default_max_connections);
+                max_connections.get().saturating_mul(2)
             } else {
                 100
             };