[Experiment] Antalya 26.6: Parquet v3 constant column skip

docs/design/parquet-v3-page-level-constant-column.md

@@ -0,0 +1,110 @@
+# Parquet v3: per-subgroup (page-level) constant-column detection
+
+## Motivation
+
+The current constant-column optimization (`detectConstantColumn`) fires only when a whole
+Parquet **column chunk** is single-valued (footer `min == max`, no nulls). Data that is sorted or
+clustered on a column is often constant over long **runs of pages** without the whole row group
+being constant, so chunk-level detection misses it.
+
+Parquet's **Column Index** stores per-page `min_values` / `max_values` / `null_pages` /
+`null_counts`. The v3 reader already loads and parses it (`applyColumnIndex`) for predicate
+push-down, so per-page "is this page constant?" is available at zero extra I/O. This lets us mark a
+column constant for the row range of an individual **row subgroup** (the unit that becomes one
+output `Chunk`) and materialize it as a `ColumnConst`, skipping the covered pages' reads and decode.
+
+## Tiers (both kept)
+
+| Tier | Source | Always present? | Exact flag? | Granularity |
+|------|--------|-----------------|-------------|-------------|
+| 1 (existing) | footer `ColumnMetaData.statistics` | yes | yes (`is_min_value_exact`) | whole chunk |
+| 2 (this doc) | Column Index (per page) | no (optional) | no | per subgroup |
+
+Tier 1 stays the always-on baseline and the **only** safe detector for `BYTE_ARRAY` /
+`FIXED_LEN_BYTE_ARRAY` (Column Index has no per-page exactness flag; 16-byte truncation can make two
+distinct strings compare equal). Tier 2 is opportunistic: only for fixed-width numeric/date/time,
+only when the Column Index is already loaded, only for chunks tier 1 did not already mark constant.
+
+## Key decision: do NOT change subgroup sizing
+
+Aligning subgroups to page boundaries would fragment the block stream into many tiny chunks
+(pages are far smaller than a row group), and a subgroup carries all columns so its size is bounded
+by the non-constant columns anyway. Instead, keep subgroup boundaries exactly as today and do
+**per-subgroup, per-column** detection: a column is constant for a subgroup iff every Column-Index
+page overlapping the subgroup's row range is constant with the *same* value (or every such page is
+`null_pages`). Chunk count is unchanged; we just catch subgroups that sit inside a constant run.
+
+## Phases
+
+- **Phase 0** — retain per-page constant info. `applyColumnIndex` currently discards the parsed
+  `parq::ColumnIndex`. Keep a compact per-page summary on `ColumnChunk` (value + `is_const` +
+  `all_null`), plus the page→`first_row_index` map already in the Offset Index. Populate only for
+  eligible types.
+- **Phase 1** — `detectConstantSubchunk(column, column_info, [start_row, end_row))`: scan the pages
+  overlapping the range; return constant + value when all are `is_const` and share one value;
+  all-null when all are `null_pages`.
+- **Phase 2** — call it in `intersectColumnIndexResultsAndInitSubgroups` for each subgroup /
+  primitive column that tier 1 didn't already mark constant; set `subchunk.is_constant` /
+  `is_all_null` / `constant_value` (the same fields `decodePrimitiveColumn` propagates).
+  `formOutputColumn` needs no change (already emits `ColumnConst`).
+- **Phase 3** — skip work for constant subchunks: `decodePrimitiveColumn` skips decode on
+  `subchunk.is_constant`; `determinePagesToPrefetch` skips fetching a page only when it is constant
+  in **every** subgroup that overlaps it.
+- **Phase 4** — stateless tests (page-run constant, all-null-per-page, byte-array negative/truncation
+  guard, `GROUP BY` correctness), with a new `ParquetConstantColumnSubchunks` ProfileEvent to prove
+  tier 2 fired.
+- **Phase 5** — ProfileEvents comparison on clustered data: expect further `S3GetObject` /
+  `ParquetFetchWaitTimeMicroseconds` drops with chunk count unchanged.
+
+## Optional force-load
+
+By default tier 2 only uses the Column Index when it is already loaded (columns with a predicate
+push-down). `input_format_parquet_use_column_index_for_constant_columns` (default off) extends it:
+the Column Index (+ Offset Index) is force-loaded for eligible read columns that have no predicate,
+so tier 2 can also fire on them. Cost is a small extra read of the (tiny, tail-contiguous, coalesced)
+index; worthwhile mainly for sorted / low-cardinality columns. `applyColumnIndex` records per-page
+constant info but skips predicate pruning when the column has no condition. A future `auto` mode
+could gate this on footer signals (row-group `sorting_columns`, low compressed-bytes-per-value,
+dictionary encoding stats) instead of an all-or-nothing switch.
+
+## Cast safety (future-proofing)
+
+The optimization only fires when the stats decoder needs no value-transforming conversion
+(`SchemaConverter` sets `allow_stats` accordingly), so today `input_type == output_type` for every
+constant column and no cast is applied. To keep the constant path correct if `allow_stats` is ever
+generalized to allow transforming casts, `formOutputColumn` builds the single-value constant in
+`input_type` and runs the same `castColumn` the per-row decode uses when `needs_cast` is set - a
+no-op today, O(1) on the `ColumnConst`, and it preserves const-ness. The all-null constant is
+synthesized directly in the output domain (Null / output default), so it is not cast. The mixed fill
+already goes through the normal `formOutputColumn` cast, so it is future-safe too. This does not
+touch the `allow_stats` decision itself.
+
+## Guardrails
+
+- All types, including `BYTE_ARRAY` / `FIXED_LEN_BYTE_ARRAY` strings. No truncation guard is needed
+  for the `min == max` case: statistics/Column-Index bounds are always valid
+  (`min <= every value <= max`) and truncation only widens them, so `min == max` proves a single
+  exact value (a truncated or multi-valued page yields `min < max`). This holds at both the chunk
+  level (tier 1) and per page (tier 2), so neither needs the `is_*_value_exact` flag.
+- Gate on the existing `input_format_parquet_use_constant_column_optimization` setting; the
+  force-load above is additionally gated by
+  `input_format_parquet_use_column_index_for_constant_columns`.
+- Partial-page subgroup boundaries are fine: a partial overlap of a constant page still yields that
+  value, as long as every overlapping page is constant with the shared value.
+
+## Mixed-topology fill (Approach B) — implemented, default off
+
+`input_format_parquet_fill_constant_pages` handles constant runs *shorter* than / straddling a
+subgroup: `fillConstantPagesAndDecodeRest` fills single-value pages from the Column Index and decodes
+only the varying ones, and `determinePagesToPrefetch` skips prefetching the filled pages (a page
+shared with a subgroup that decodes it normally is still fetched — `willFillConstantPages` is
+deterministic so both paths agree). The fill walks the rows that pass the filter (`row_subgroup.filter`), so it works under PREWHERE /
+row-level filters and page-pruning predicates too - it produces exactly `rows_pass` values for any
+filter, and because the decision no longer depends on `rows_pass` it is identical at prefetch time
+and decode time (so the prefetch-skip can never drop a page the decode needs). All-null pages are
+filled (nulls via the compact values + null map + `expand` path, or the output default under
+`null_as_default`); a subgroup with an all-null page falls back only when the output can represent
+neither null nor a default. The one remaining gate is `needs_cast`: the fill writes the Column Index
+value into the `decoded_type` column, valid only when no post-decode cast applies (resolving whether
+`decodeField` yields the decoded or the output value domain is build-gated). Experimental, off by
+default; needs a build + correctness tests before it can be trusted.

docs/en/engines/database-engines/datalake.md

@@ -59,6 +59,7 @@ The following settings are supported:
 | `region`                | AWS region for the service (e.g., `us-east-1`)                                          |
 | `dlf_access_key_id`     | Access key ID for DLF access                                                            |
 | `dlf_access_key_secret` | Access key Secret for DLF access                                                        |
+| `namespaces`            | Comma-separated list of namespaces, implemented for catalog types: `rest`, `glue` and `unity` |
 
 ## Examples {#examples}
 
@@ -81,4 +82,30 @@ SETTINGS
     onelake_client_secret = client_secret;
 SHOW TABLES IN database_name;
 SELECT count() from database_name.table_name;
-```
+```
+
+## Namespace filter {#namespace}
+
+By default, ClickHouse reads tables from all namespaces available in the catalog. You can limit this behavior using the `namespaces` database setting. The value should be a comma‑separated list of namespaces that are allowed to be read.
+
+Supported catalog types are `rest`, `glue` and `unity`.
+
+For example, if the catalog contains three namespaces - `dev`, `stage`, and `prod` - and you want to read data only from dev and stage, set:
+```
+namespaces='dev,stage'
+```
+
+### Nested namespaces {#namespace-nested}
+
+The Iceberg (`rest`) catalog supports nested namespaces. The `namespaces` filter accepts the following patterns:
+
+- `namespace` - includes tables from the specified namespace, but not from its nested namespaces.
+- `namespace.nested` - includes tables from the nested namespace, but not from the parent.
+- `namespace.*` - includes tables from all nested namespaces, but not from the parent.
+
+If you need to include both a namespace and its nested namespaces, specify both explicitly. For example:
+```
+namespaces='namespace,namespace.*'
+```
+
+The default value is '*', which means all namespaces are included.

programs/local/LocalServer.cpp

@@ -4,6 +4,7 @@
 #include <exception>
 #include <Common/Config/getLocalConfigPath.h>
 #include <Common/CurrentMemoryTracker.h>
+#include <Common/HostResolvePool.h>
 #include <Common/logger_useful.h>
 #include <Common/formatReadable.h>
 #include <Core/Defines.h>
@@ -161,6 +162,9 @@ namespace ServerSetting
     extern const ServerSettingsString mark_cache_policy;
     extern const ServerSettingsUInt64 mark_cache_size;
     extern const ServerSettingsDouble mark_cache_size_ratio;
+    extern const ServerSettingsString object_storage_identity_cache_policy;
+    extern const ServerSettingsUInt64 object_storage_identity_cache_size;
+    extern const ServerSettingsDouble object_storage_identity_cache_size_ratio;
     extern const ServerSettingsString unique_key_bitmap_cache_policy;
     extern const ServerSettingsUInt64 unique_key_bitmap_cache_size_bytes;
     extern const ServerSettingsDouble unique_key_bitmap_cache_size_ratio;
@@ -224,6 +228,7 @@ namespace ServerSetting
     extern const ServerSettingsUInt64 max_keep_alive_requests;
     extern const ServerSettingsBool asynchronous_metrics_enable_heavy_metrics;
     extern const ServerSettingsUInt32 asynchronous_heavy_metrics_update_period_s;
+    extern const ServerSettingsBool http_latency_aware_host_selection;
 }
 
 namespace ErrorCodes
@@ -419,6 +424,8 @@ void LocalServer::initialize(Poco::Util::Application & self)
         server_settings[ServerSetting::max_format_parsing_thread_pool_size],
         server_settings[ServerSetting::max_format_parsing_thread_pool_free_size],
         server_settings[ServerSetting::format_parsing_thread_pool_queue_size]);
+
+    HostResolver::setLatencyAwareSelection(server_settings[ServerSetting::http_latency_aware_host_selection]);
 }
 
 
@@ -1422,6 +1429,11 @@ void LocalServer::processConfig()
     }
     global_context->setMarkCache(mark_cache_policy, mark_cache_size, mark_cache_size_ratio);
 
+    global_context->setObjectStorageIdentityCache(
+        server_settings[ServerSetting::object_storage_identity_cache_policy],
+        server_settings[ServerSetting::object_storage_identity_cache_size],
+        server_settings[ServerSetting::object_storage_identity_cache_size_ratio]);
+
     /// UNIQUE KEY delete-bitmap cache. Zero size disables.
     String unique_key_bitmap_cache_policy_name = server_settings[ServerSetting::unique_key_bitmap_cache_policy];
     size_t unique_key_bitmap_cache_size = server_settings[ServerSetting::unique_key_bitmap_cache_size_bytes];

programs/server/Server.cpp

@@ -60,6 +60,7 @@
 #include <Common/FailPoint.h>
 #include <Common/CPUID.h>
 #include <Common/HTTPConnectionPool.h>
+#include <Common/HostResolvePool.h>
 #include <Common/NamedCollections/NamedCollectionsFactory.h>
 #include <Server/createServer.h>
 #include <Server/socketBindListen.h>
@@ -258,6 +259,7 @@ namespace ServerSetting
     extern const ServerSettingsUInt64 http_connections_warn_limit;
     extern const ServerSettingsUInt64 http_connections_rcvbuf;
     extern const ServerSettingsUInt64 http_connections_sndbuf;
+    extern const ServerSettingsBool http_latency_aware_host_selection;
     extern const ServerSettingsString index_mark_cache_policy;
     extern const ServerSettingsUInt64 index_mark_cache_size;
     extern const ServerSettingsDouble index_mark_cache_size_ratio;
@@ -298,6 +300,9 @@ namespace ServerSetting
     extern const ServerSettingsString mark_cache_policy;
     extern const ServerSettingsUInt64 mark_cache_size;
     extern const ServerSettingsDouble mark_cache_size_ratio;
+    extern const ServerSettingsString object_storage_identity_cache_policy;
+    extern const ServerSettingsUInt64 object_storage_identity_cache_size;
+    extern const ServerSettingsDouble object_storage_identity_cache_size_ratio;
     extern const ServerSettingsString unique_key_index_cache_policy;
     extern const ServerSettingsUInt64 unique_key_index_cache_size_bytes;
     extern const ServerSettingsDouble unique_key_index_cache_size_ratio;
@@ -2150,6 +2155,11 @@ try
     }
     global_context->setMarkCache(mark_cache_policy, mark_cache_size, mark_cache_size_ratio);
 
+    global_context->setObjectStorageIdentityCache(
+        server_settings[ServerSetting::object_storage_identity_cache_policy],
+        server_settings[ServerSetting::object_storage_identity_cache_size],
+        server_settings[ServerSetting::object_storage_identity_cache_size_ratio]);
+
     String unique_key_index_cache_policy_name = server_settings[ServerSetting::unique_key_index_cache_policy];
     size_t unique_key_index_cache_size = server_settings[ServerSetting::unique_key_index_cache_size_bytes];
     double unique_key_index_cache_size_ratio = server_settings[ServerSetting::unique_key_index_cache_size_ratio];
@@ -2749,6 +2759,8 @@ try
                     new_server_settings[ServerSetting::http_connections_sndbuf],
                 });
 
+            HostResolver::setLatencyAwareSelection(new_server_settings[ServerSetting::http_latency_aware_host_selection]);
+
             DNSResolver::instance().setFilterSettings(new_server_settings[ServerSetting::dns_allow_resolve_names_to_ipv4], new_server_settings[ServerSetting::dns_allow_resolve_names_to_ipv6]);
 
             if (global_context->isServerCompletelyStarted())

src/Access/Common/AccessType.h

@@ -318,6 +318,7 @@ enum class AccessType : uint8_t
     M(SYSTEM_DROP_ICEBERG_METADATA_CACHE, "SYSTEM CLEAR ICEBERG_METADATA_CACHE, SYSTEM DROP ICEBERG_METADATA_CACHE", GLOBAL, SYSTEM_DROP_CACHE) \
     M(SYSTEM_DROP_AVRO_SCHEMA_CACHE, "SYSTEM CLEAR AVRO SCHEMA CACHE, SYSTEM DROP AVRO SCHEMA CACHE, DROP AVRO SCHEMA CACHE", GLOBAL, SYSTEM_DROP_CACHE) \
     M(SYSTEM_DROP_PARQUET_METADATA_CACHE, "SYSTEM DROP PARQUET_METADATA_CACHE", GLOBAL, SYSTEM_DROP_CACHE) \
+    M(SYSTEM_DROP_OBJECT_STORAGE_IDENTITY_CACHE, "SYSTEM CLEAR OBJECT STORAGE IDENTITY CACHE, SYSTEM DROP OBJECT STORAGE IDENTITY CACHE", GLOBAL, SYSTEM_DROP_CACHE) \
     M(SYSTEM_PREWARM_PRIMARY_INDEX_CACHE, "SYSTEM PREWARM PRIMARY INDEX, PREWARM PRIMARY INDEX CACHE, PREWARM PRIMARY INDEX", GLOBAL, SYSTEM_DROP_CACHE) \
     M(SYSTEM_DROP_PRIMARY_INDEX_CACHE, "SYSTEM CLEAR PRIMARY INDEX CACHE, SYSTEM DROP PRIMARY INDEX, DROP PRIMARY INDEX CACHE, DROP PRIMARY INDEX", GLOBAL, SYSTEM_DROP_CACHE) \
     M(SYSTEM_DROP_UNCOMPRESSED_CACHE, "SYSTEM CLEAR UNCOMPRESSED CACHE, SYSTEM DROP UNCOMPRESSED, DROP UNCOMPRESSED CACHE, DROP UNCOMPRESSED", GLOBAL, SYSTEM_DROP_CACHE) \

src/Common/CurrentMetrics.cpp

@@ -337,6 +337,8 @@
     M(IcebergMetadataFilesCacheFiles, "Number of cached files in the Iceberg metadata cache") \
     M(ParquetMetadataCacheBytes, "Size of the Parquet metadata cache in bytes") \
     M(ParquetMetadataCacheFiles, "Number of cached files in the Parquet metadata cache") \
+    M(ObjectStorageIdentityCacheBytes, "Size of the object-storage identity cache in bytes") \
+    M(ObjectStorageIdentityCacheCells, "Number of entries in the object-storage identity cache") \
     M(AvroSchemaCacheBytes, "Size of the Avro schema cache in bytes") \
     M(AvroSchemaCacheCells, "Number of cached Avro schemas, including both registered and fetched schemas.") \
     M(AvroSchemaRegistryCacheBytes, "Size of the Avro schema registry cache in bytes") \

src/Common/ErrorCodes.cpp

@@ -658,6 +658,7 @@
     M(776, RESOURCE_LIMIT_EXCEEDED) \
     M(777, MEMORY_RESERVATION_KILLED) \
     M(778, MEMORY_RESERVATION_FAILED) \
+    M(779, CATALOG_NAMESPACE_DISABLED) \
 \
     M(900, DISTRIBUTED_CACHE_ERROR) \
     M(901, CANNOT_USE_DISTRIBUTED_CACHE) \