Release 0.9.0

CHANGELOG.md

@@ -24,19 +24,59 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 # [0.9.0]
 
+### Breaking / Migration
+- **Ingest V2 is now the default ingest path.** The `/api/v1/{index}/ingest` endpoint transparently routes to V2.
+- The `rest_listen_port` top-level config field is deprecated; use `rest.listen_port` under the new `rest` block. The old field still works but emits a warning.
+- Quickwit metrics now use `metrics-rs`: gRPC metric names use a `service` label instead of embedding the service in the metric name, and janitor GC metric names no longer carry the duplicated `quickwit_` prefix (#6374).
+- Stemming is now restricted to the `multilang` cargo feature (#6085); the previously bundled generic stemmer is no longer available by default.
+- The unused multilang tokenizer feature was removed (#6154).
+- Metastore format: new `maturity` field and compaction columns on splits; PostgreSQL metastore migrates automatically on first start.
+- Building from source now requires Rust 1.92 (#6432, #6545).
+
 ### Added
-- Add Ingest V2 (#5600, #5566, #5463, #5375, #5350, #5252 #5202)
+- Add Ingest V2 — now the default (#5600, #5566, #5463, #5375, #5350, #5252, #5202, #6078, #6185, #6203, #6207, #6217, #6249)
+- Offload leaf-search work to AWS Lambda functions — searchers can farm out part of their workload to Lambda (#6157, #5c1e60f)
+- Add experimental DataFusion query layer for Parquet metrics with SQL/Substrait execution, Arrow IPC streaming, and distributed workers (#6276)
+- Extend the Parquet metrics pipeline with DDSketch, sorted-series and row-key metadata, zone-map pruning, partitioning, sketch split pagination, and Substrait execution metadata (#6248, #6257, #6290, #6292, #6295, #6340, #6347, #6348, #6349, #6363, #6364, #6368)
+- Add column-major and streaming Parquet merge primitives, merge scheduler wiring, and a rollout flag for routing regular merges through the streaming engine (#6335, #6362, #6367, #6377, #6384, #6386, #6406, #6407, #6408, #6409, #6423, #6424, #6425, #6426, #6428, #6441)
+- Add optional mTLS validation to the REST API and an optional dedicated health-check HTTP server for deployments that put the REST API behind mTLS (#6467, #6528)
 - Add SQS source (#5374, #5335, #5148)
+- Add Jaeger v2 support (#6023)
+- Extract and propagate W3C `traceparent` header on incoming HTTP requests (#6224)
+- Add distributed tracing to the gRPC stack (#6403)
+- Support configurable OTLP exporter protocol for traces and logs (#6254)
+- Export internal logs via OTLP exporter (#6142)
+- Add `QW_LOG_FORMAT=DDG` JSON log formatter (#6215)
+- Add ES-compatible endpoints for Trino connector support (#6168)
+- Elasticsearch DSL: prefix and wildcard queries (#6000), `case_insensitive` parameter on supported queries (#6005), regexp shorthand, concatenate-fields exposure, and `text` → `keyword` mapping in `_mapping` (#6208), `index_filter` on field capabilities API (#6102), `ignore_unavailable` query parameter (#5971)
+- Make Elasticsearch `TermsQuery` use `TermSetQuery` internally for better performance (#6151)
+- Add composite aggregation (#6214) and aggregations alias (#6314)
+- Add `skip_aggregation_finalization` to `SearchRequest` (#6145)
+- Add `list_index_stats` endpoint (#6035)
+- Add `validate_docs` ingest setting (#5984); validate doc-mapping updates (#5988)
+- Support updating the doc mapper through the API (#5253)
+- Add CORS debug mode (#5955)
+- Add config to fail search when it targets too many splits (#6009)
+- Differentiate leaf- and root-level search timeouts (#6255)
+- Predicate cache in leaf search (#6024); skip CPU work when it cannot improve the result (#6001); propagate cancellation within leaf search (#6002)
+- Expose search resource stats, CPU thread count, root phase wall times, and warmup memory currently in flight (#6416, #6514, #6533)
+- Set `searcher.warmup_single_split_initial_allocation` default to 300 MB (#c34966c6)
+- Rebalance shards when ingester status changes (#6185) and improved rebalance algorithm (#6018)
+- Add object storage metrics for GCS (#5889)
+- Add IO metrics and track bytes written to the WAL (#6429)
+- Add configurable system prefix and separator for metrics (#6445)
+- Expose per-shard load configuration and disable the Tokio LIFO slot by default (#5898, #5899)
+- Redact sensitive information in developer API debug output (#6191)
 - Disable control plane check for searcher (#5599, #5360)
 - Partially implement `_elastic/_cluster/health` (#5595)
 - Make Jaeger span attribute-to-tag conversion exhaustive (#5574)
 - Use `content_length_limit` for ES bulk limit (#5573)
 - Limit and monitor warmup memory usage (#5568)
 - Add eviction metrics to caches (#5523)
 - Record object storage request latencies (#5521)
-- Add some kind of throttling on the janitor to prevent it from overloading (#5510)
+- Throttle the janitor to prevent overloading the metastore (#5510)
 - Prevent single split searches from different `leaf_search` from interleaving (#5509)
-- Retry on S3 internal error (#5504)
+- Retry on S3 internal error (#5504); make more S3 errors retryable (#5384)
 - Allow specifying OTEL index ID in header (#5503)
 - Add a metric to count storage errors and their error code (#5497)
 - Add support for concatenated fields (#4773, #5369, #5331)
@@ -47,14 +87,16 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Add some additional search metrics (#5447)
 - Improve GC resilience and add metrics (#5420)
 - Enable force shutdown with 2nd Ctrl+C (#5414)
-- Add request_timeout_secs config to searcher config (#5402)
+- Add `request_timeout_secs` to searcher config (#5402)
 - Memoize S3 client (#5377)
 - Add more env var config for Postgres (#5365)
 - Enable str fast field range queries (#5324)
 - Allow querying non-existing fields (#5308)
-- Support updating doc mapper through api (#5253)
 - Add optional special handling for hex in code tokenizer (#5200)
+- Add `UnicodeSegmenter` tokenizer and case-insensitive regex support for lowercasing tokenizers (#6277, #6454)
 - Added a circuit breaker layer (#5134)
+- Follow AWS hints for Lambda retries (#6195)
+- Improve cluster sizing documentation for control plane, metastore and janitor (#6202)
 - Various performance optimizations in Tantivy (https://github.com/quickwit-oss/tantivy/blob/main/CHANGELOG.md)
 
 ### Changed
@@ -68,12 +110,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Improve merge pipeline finalization (#5475)
 - Allow failed splits in root search (#5440)
 - Batch delete from GC (#5404, #5380)
-- Make some S3 errors retryable (#5384)
 - Change default timestamps in OTEL logs (#5366)
 - Only return root spans for Jaeger HTTP API (#5358)
 - Share aggregation limit on node (#5357)
+- Make regex lenient to start and end anchors (#6089) — *later reverted, see Fixed*
+- Use correct precision level for fastfield-based term queries on datetime (#6027)
+- Disable the Tokio LIFO slot and tune per-shard default load (#5898)
+- Improve control-plane logging (#6003)
+- Migrate Quickwit metrics to `metrics-rs` and move exporter setup into `quickwit-telemetry-exporters` (#6374)
+- Move OTLP metrics export to `metrics-opentelemetry`, remove the temporality override, and add default retry policy to OTLP exporters (#6499, #6510, #6522, #6535)
+- Use CRC32C instead of MD5 for object-storage checksums, and document the `disable_checksums` storage config parameter (#6325, #6442)
+- Make `Storage::get_slice` zero-copy for single-segment bodies (#6336)
+- Add Protobuf WAL mrecord serialization support for forward/backward-compatible schema evolution; legacy WAL writes remain supported (#6521)
+- Improve search scheduling by using current node load, prioritizing result merging over split searches, and aborting split warmup when a required term is missing (#6390, #6511, #6512)
+- Update Lambda release artifacts and automate Lambda binary builds in the release workflow (#6330, #6456, #6457, #6458)
+- Upgraded Tantivy to 1e859fd with multiple search, aggregation, and tokenizer improvements (see Tantivy CHANGELOG) (#6327, #6360, #6365, #6380, #6444, #6495, #6516, #6532)
 
 ### Fixed
+- Fix timestamp field bound precision and add range checks to floating-point timestamp parsing (#6201, #6525)
 - Fix existence queries for nested fields (#5581)
 - Fix lenient option with wildcard queries (#5575)
 - Fix incompatible ES Java date format (#5462)
@@ -86,12 +140,43 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - Fix configuration interpolation (#5403)
 - Fix jaeger duration parse error (#5518)
 - Fix unit conversion in jaeger http search endpoint (#5519)
+- Fix Kinesis source panic on resharding (#5912)
+- Fix Azure multipart upload data corruption (#5919)
+- Fix leaf list fields merging logic (#5908)
+- Fix empty intermediate aggregation results merge (#5930)
+- Fix error when running a scoring query with cache (#6025)
+- Fix `f64` not working properly with concatenated fields (#6074)
+- Fix wrong range result when datetime was inferred in JSON type (#6048)
+- Fix infinite-loop OOM bug caused by rebalancing shards from unavailable ingesters (#6078)
+- Fix index reincarnation routing bug (#6217)
+- Fix bug when deploying a new routing table (#6249)
+- Fix retiring/decommissioned indexer handling, shard ownership deletion, empty-shard indexing checks, and material indexer pool updates (#6427, #6504, #6509, #6518, #6553)
+- Fix control-plane scheduling idempotency and unavailable-node reporting (#6503, #6524, #6551)
+- Fix descending sorted-series encoding, null ordering in sorted-series keys, and maturity assignment on Parquet splits (#6338, #6343, #6399)
+- Fix Parquet merge adapter and consumer edge cases around sorted input, sub-regions, and stronger verifiers (#6426, #6428)
+- Fix OpenDAL GCS TLS regression, GCS put memory growth, single-part MD5 headers, S3 stalled-stream retryability, S3 dispatch retries, and localstack checksum compatibility (#6478, #6482, #6485, #6492, #6501, #6502, #6539, #6541)
+- Fix Azure delete metrics for the Azure blob backend (#6469)
+- Fix Lambda retry log storms and empty `LAMBDA_ZIP_PATH` handling (#6318, #6462)
+- Fix gRPC telemetry exporters (#6558)
+- Truncate split lists in error logs to keep large routing errors readable (#6315)
+- Fix Chitchat gossip bug triggering excess gRPC traffic and bump chitchat (#6082, #6323)
+- Revert over-eager anchor lenience in regex queries (#6089)
+- `skip_aggregation_finalization` fixes for composite aggregations
+- Jaeger: query resource attributes when the Jaeger request carries tags
 
 ### Removed
-- Remove support for 2-digit years in java datetime parser (#5596)
-- Remove DocMapper trait (#5508)
-- Remove support for AWS Lambda (#5884)
+- Remove support for 2-digit years in Java datetime parser (#5596)
+- Remove `DocMapper` trait (#5508)
+- Remove standalone AWS Lambda deployment mode — the previous dedicated search/indexing Lambda binaries are gone (#5884). *AWS Lambda is still supported for search offloading; see the Added section.*
 - Remove search stream endpoint (#5886)
+- Remove mentions of the stream API from the docs (#5958)
+- Remove the legacy telemetry crate and native-tls from transitive dependencies (#6431, #6537)
+- Remove the legacy `/api/v1/{index}/ingest-v2` REST endpoint (V2 is now served from `/ingest` — see Breaking / Migration)
+- Remove the unused multilang tokenizer feature (#6154)
+- Restrict stemming to the `multilang` feature (#6085)
+
+### Security
+- Upgrade dependencies including `rustls-webpki` for RUSTSEC-2026-0104, OpenSSL, Python requests, and UI packages with known vulnerabilities (#6219, #6307, #6341, #6342, #6387)
 
 # [0.8.1]
 

docs/get-started/quickstart.md

@@ -6,7 +6,7 @@ sidebar_position: 1
 import Tabs from '@theme/Tabs';
 import TabItem from '@theme/TabItem';
 
-In this quick start guide, we will install Quickwit, create an index, add documents and finally execute search queries. All the Quickwit commands used in this guide are documented [in the CLI reference documentation](/docs/reference/cli.md).
+In this quick start guide, we will install Quickwit, create an index, add documents and finally execute search queries. All the Quickwit commands used in this guide are documented [in the CLI reference documentation](../reference/cli.md).
 
 ## Install Quickwit using Quickwit installer
 
@@ -31,13 +31,7 @@ You can also pull and run the Quickwit binary in an isolated Docker container.
 ```bash
 # Create first the data directory.
 mkdir qwdata
-docker run --rm quickwit/quickwit --version
-```
-
-If you are using Apple silicon based macOS system you might need to specify the platform. You can also safely ignore jemalloc warnings.
-
-```bash
-docker run --rm --platform linux/amd64 quickwit/quickwit --version
+docker run --rm quickwit/quickwit:0.9.0 --version
 ```
 
 ## Start Quickwit server
@@ -55,7 +49,7 @@ docker run --rm --platform linux/amd64 quickwit/quickwit --version
 <TabItem value="docker" label="Docker">
 
 ```bash
-docker run --rm -v $(pwd)/qwdata:/quickwit/qwdata -p 127.0.0.1:7280:7280 quickwit/quickwit run
+docker run --rm -v $(pwd)/qwdata:/quickwit/qwdata -p 127.0.0.1:7280:7280 quickwit/quickwit:0.9.0 run
 ```
 
 </TabItem>
@@ -72,24 +66,24 @@ curl http://localhost:7280/api/v1/version
 
 ## Create your first index
 
-Before adding documents to Quickwit, you need to create an index configured with a YAML config file. This config file notably lets you define how to map your input documents to your index fields and whether these fields should be stored and indexed. See the [index config documentation](/docs/configuration/index-config.md).
+Before adding documents to Quickwit, you need to create an index configured with a YAML config file. This config file notably lets you define how to map your input documents to your index fields and whether these fields should be stored and indexed. See the [index config documentation](../configuration/index-config.md).
 
 Let's create an index configured to receive Stackoverflow posts (questions and answers).
 
 ```bash
 # First, download the stackoverflow dataset config from Quickwit repository.
-curl -o stackoverflow-index-config.yaml https://raw.githubusercontent.com/quickwit-oss/quickwit/main/config/tutorials/stackoverflow/index-config.yaml
+curl -o stackoverflow-index-config.yaml https://raw.githubusercontent.com/quickwit-oss/quickwit/v0.9.0/config/tutorials/stackoverflow/index-config.yaml
 ```
 
-The index config defines three fields: `title`, `body` and `creationDate`. `title` and `body` are [indexed and tokenized](../configuration/index-config.md#text-type), and they are also used as default search fields, which means they will be used for search if you do not target a specific field in your query. `creationDate` serves as the timestamp for each record. There are no more explicit field definitions as we can use the default dynamic [mode](/docs/configuration/index-config.md#mode): the undeclared fields will still be indexed, by default fast fields are enabled to enable aggregation queries. and the `raw` tokenizer is used for text. 
+The index config defines three fields: `title`, `body` and `creationDate`. `title` and `body` are [indexed and tokenized](../configuration/index-config.md#text-type), and they are also used as default search fields, which means they will be used for search if you do not target a specific field in your query. `creationDate` serves as the timestamp for each record. There are no more explicit field definitions as we can use the default dynamic [mode](../configuration/index-config.md#mode): the undeclared fields will still be indexed, by default fast fields are enabled to enable aggregation queries. and the `raw` tokenizer is used for text.
 
 And here is the complete config:
 
 ```yaml title="stackoverflow-index-config.yaml"
 #
 # Index config file for stackoverflow dataset.
 #
-version: 0.7
+version: 0.8
 
 index_id: stackoverflow
 
@@ -117,7 +111,7 @@ search_settings:
   default_search_fields: [title, body]
 
 indexing_settings:
-  commit_timeout_secs: 30
+  commit_timeout_secs: 10
 ```
 
 Now we can create the index with the command:
@@ -148,7 +142,7 @@ You're now ready to fill the index.
 
 ## Let's add some documents
 
-Quickwit can index data from many [sources](/docs/configuration/source-config.md). We will use a new line delimited json [ndjson](http://ndjson.org/) datasets as our data source.
+Quickwit can index data from many [sources](../configuration/source-config.md). We will use a new line delimited json [ndjson](http://ndjson.org/) datasets as our data source.
 Let's download [a bunch of stackoverflow posts (10 000)](https://quickwit-datasets-public.s3.amazonaws.com/stackoverflow.posts.transformed-10000.json) in [ndjson](http://ndjson.org/) format and index it.
 
 ```bash
@@ -276,7 +270,7 @@ Congrats! You can level up with the following tutorials to discover all Quickwit
 Run the following command from within Quickwit's installation directory.
 
 ```bash
-curl -o stackoverflow-index-config.yaml https://raw.githubusercontent.com/quickwit-oss/quickwit/main/config/tutorials/stackoverflow/index-config.yaml
+curl -o stackoverflow-index-config.yaml https://raw.githubusercontent.com/quickwit-oss/quickwit/v0.9.0/config/tutorials/stackoverflow/index-config.yaml
 ./quickwit index create --index-config ./stackoverflow-index-config.yaml
 curl -O https://quickwit-datasets-public.s3.amazonaws.com/stackoverflow.posts.transformed-10000.json
 ./quickwit index ingest --index stackoverflow --input-path ./stackoverflow.posts.transformed-10000.json --force

docs/get-started/tutorials/prometheus-metrics.md

@@ -17,27 +17,19 @@ Here’s the complete Docker Compose configuration:
 ```yaml
 services:
   quickwit:
-    image: quickwit/quickwit
-    environment:
-      QW_ENABLE_OPENTELEMETRY_OTLP_EXPORTER: "true"
-      OTEL_EXPORTER_OTLP_ENDPOINT: "http://localhost:7281"
+    image: quickwit/quickwit:0.9.0
     ports:
       - 7280:7280
     command: ["run"]
 
   grafana:
-    image: grafana/grafana-oss
+    image: grafana/grafana-oss:13.0.2
     container_name: grafana
     ports:
       - "${MAP_HOST_GRAFANA:-127.0.0.1}:3000:3000"
-    environment:
-      GF_INSTALL_PLUGINS: https://github.com/quickwit-oss/quickwit-datasource/releases/download/v0.4.6/quickwit-quickwit-datasource-0.4.6.zip;quickwit-quickwit-datasource
-      GF_AUTH_DISABLE_LOGIN_FORM: "true"
-      GF_AUTH_ANONYMOUS_ENABLED: "true"
-      GF_AUTH_ANONYMOUS_ORG_ROLE: Admin
 
   prometheus:
-    image: prom/prometheus:latest
+    image: prom/prometheus:v3.13.1
     container_name: prometheus
     volumes:
       - ./prometheus.yml:/etc/prometheus/prometheus.yml  # Ensure prometheus.yml exists in the same directory
@@ -80,10 +72,10 @@ This will launch Quickwit, Prometheus, and Grafana services.
 
 ## Step 4: Configure Grafana to Use Prometheus
 
-1. Open Grafana in your browser at `http://localhost:3000`.
-2. Navigate to **Configuration** > **Data Sources**.
-3. Click **Add Data Source**, select **Prometheus**, and set the URL to `http://prometheus:9090`.
-4. Click **Save & Test** to verify the connection.
+1. Open Grafana in your browser at `http://localhost:3000` and sign in with the default credentials (`admin` / `admin`). Grafana will ask you to choose a new password.
+2. Navigate to **Connections** > **Data sources**.
+3. Click **Add new data source**, select **Prometheus**, and set the URL to `http://prometheus:9090`.
+4. Click **Save & test** to verify the connection.
 
 ## Step 5: Create or Use Pre-Configured Dashboards
 
@@ -92,4 +84,3 @@ Now that Grafana is set up with Prometheus as a data source, you can create cust
 1. Go to the **Dashboards** section in Grafana.
 2. Import or create a new dashboard to visualize metrics.
 3. Alternatively, use one of Quickwit’s [pre-configured dashboards](../../operating/monitoring).
-