diff --git a/releases/release-8.5.8.md b/releases/release-8.5.8.md new file mode 100644 index 0000000000000..39dee12aec528 --- /dev/null +++ b/releases/release-8.5.8.md @@ -0,0 +1,120 @@ +--- +title: TiDB 8.5.8 Release Notes +summary: Learn about the improvements and bug fixes in TiDB 8.5.8. +--- + +# TiDB 8.5.8 Release Notes + +Release date: TBD + +TiDB version: 8.5.8 + +Quick access: [Quick start](https://docs.pingcap.com/tidb/v8.5/quick-start-with-tidb) | [Production deployment](https://docs.pingcap.com/tidb/v8.5/production-deployment-using-tiup) + +## Improvements + ++ TiDB + + - Improve optimizer row count estimation for composite indexes that include virtual columns by falling back to index statistics when column statistics are unavailable, helping TiDB choose indexes more accurately [#69134](https://github.com/pingcap/tidb/issues/69134) @[qw4990](https://github.com/qw4990) + ++ Tools + + + TiCDC + + - Improve TiCDC changefeed scan performance when delete events are ignored, reducing unnecessary DML decoding during historical catch-up for delete-heavy workloads [#5430](https://github.com/pingcap/ticdc/issues/5430) @[asddongmen](https://github.com/asddongmen) + - Improve TiCDC event service stability and throughput under memory pressure by introducing an adaptive scan window algorithm, reducing dispatcher starvation and reset events in DDL or sync point scenarios [#4172](https://github.com/pingcap/ticdc/issues/4172) @[asddongmen](https://github.com/asddongmen) + - Improve TiCDC Kafka Sink verification to make it more lightweight and complete by avoiding startup-only work during verification, checking encoder dependencies such as Schema Registry for existing topics, and validating `replication-factor` only when TiCDC needs to create a topic [#5618](https://github.com/pingcap/ticdc/issues/5618) [#5720](https://github.com/pingcap/ticdc/issues/5720) @[3AceShowHand](https://github.com/3AceShowHand) + - Reduce external storage client and connection usage for TiCDC Kafka Sinks with Claim-Check enabled by sharing a single `ClaimCheck` instance among all Encoders in the same Sink [#5719](https://github.com/pingcap/ticdc/issues/5719) @[3AceShowHand](https://github.com/3AceShowHand) + - Simplify and unify TiCDC Kafka Sink error handling by standardizing the classification and wrapping of configuration, Admin API, and producer errors, making retry classification and troubleshooting easier [#5790](https://github.com/pingcap/ticdc/issues/5790) @[3AceShowHand](https://github.com/3AceShowHand) + +## Bug fixes + ++ TiDB + + - Fix the issue that multi-schema changes containing `MODIFY COLUMN` fall back to transactional backfill instead of using ingest or distributed backfill [#70136](https://github.com/pingcap/tidb/issues/70136) @[joechenrh](https://github.com/joechenrh) + - Fix the issue that the progress of adding an index on a partitioned table might move backward during reorganization [#62496](https://github.com/pingcap/tidb/issues/62496) @[GMHDBJD](https://github.com/GMHDBJD) + - Fix the issue that `ADMIN ALTER DDL JOBS` cannot dynamically adjust the thread or batch size of a running transaction-mode backfill [#70138](https://github.com/pingcap/tidb/issues/70138) @[joechenrh](https://github.com/joechenrh) + - Fix the issue that TiDB might panic and terminate the user session when a query uses an `IN` subquery as an operand of a nested `NOT IN` expression [#64854](https://github.com/pingcap/tidb/issues/64854) @[hawkingrei](https://github.com/hawkingrei) + - Fix the issue that otherwise cache-safe queries using `JSON_EXTRACT()` cannot use the prepared or non-prepared plan cache [#69522](https://github.com/pingcap/tidb/issues/69522) @[winoros](https://github.com/winoros) + - Fix the issue that statistics for virtual generated columns added with `ALTER TABLE ADD COLUMN` might be initialized incorrectly, causing inaccurate `SHOW STATS_HISTOGRAMS` output and unnecessary loading of placeholder statistics [#69160](https://github.com/pingcap/tidb/issues/69160) @[qw4990](https://github.com/qw4990) + - Fix the issue that TiDB might consume unbounded connection memory when receiving repeated `COM_STMT_SEND_LONG_DATA` requests for prepared statements without executing or resetting them [#70349](https://github.com/pingcap/tidb/issues/70349) @[djshow832](https://github.com/djshow832) + - Fix the issue that TiDB might crash when parsing SQL statements or optimizer hints with excessively deep nested parentheses [#70192](https://github.com/pingcap/tidb/issues/70192) @[Debra-He](https://github.com/Debra-He) + - Fix the issue that parsing TiDB runtime traces might fail during batched TSO requests because the PD client emits invalid nested runtime trace regions [#69743](https://github.com/pingcap/tidb/issues/69743) @[YangKeao](https://github.com/YangKeao) + - Fix the issue that `COM_CHANGE_USER` authentication failure might leave the connection in an inconsistent session state [#69691](https://github.com/pingcap/tidb/issues/69691) @[bb7133](https://github.com/bb7133) + - Fix the issue that `COM_STMT_SEND_LONG_DATA` can accumulate unbounded parameter data for a single prepared statement; TiDB now limits the accumulated size using the session `max_allowed_packet` setting and returns a packet-too-large error when the limit is exceeded [#69693](https://github.com/pingcap/tidb/issues/69693) @[bb7133](https://github.com/bb7133) + - Fix the issue that `UNCOMPRESS()` might consume excessive untracked memory and exceed the query memory quota when processing specially crafted compressed input [#70198](https://github.com/pingcap/tidb/issues/70198) @[Debra-He](https://github.com/Debra-He) + - Fix the issue that non-privileged users can read other users' attributes through `INFORMATION_SCHEMA.USER_ATTRIBUTES` [#70277](https://github.com/pingcap/tidb/issues/70277) @[djshow832](https://github.com/djshow832) + - Fix the issue that recording OOM diagnostic goroutine profiles might prolong stop-the-world pauses and increase query latency when memory usage frequently exceeds the alarm ratio [#62080](https://github.com/pingcap/tidb/issues/62080) @[YangKeao](https://github.com/YangKeao) + - Fix the issue that `LOAD DATA LOCAL INFILE` in a pessimistic transaction might retry internally after a retryable lock conflict, desynchronize the client connection, and return an invalid sequence error instead of the original deadlock error [#69793](https://github.com/pingcap/tidb/issues/69793) @[lance6716](https://github.com/lance6716) + - Fix the issue that `ALTER TABLE ... REORGANIZE PARTITION` might rebuild global indexes without entries for rows in non-reorganized partitions that come after the reorganized partitions in partition order, causing queries using those indexes to miss rows and potentially allowing duplicate index values to be inserted [#70023](https://github.com/pingcap/tidb/issues/70023) @[mjonss](https://github.com/mjonss) + - Fix the issue that Join operations, `UPDATE` statements, and `DELETE` statements might allocate excessive memory for initial Chunks, especially under high concurrency or when processing wide rows [#68545](https://github.com/pingcap/tidb/issues/68545) @[solotzg](https://github.com/solotzg) + ++ TiKV + + - Fix the issue that TiKV performs unnecessary slow-log message formatting for fast Raftstore message batches, causing additional CPU overhead [#19861](https://github.com/tikv/tikv/issues/19861) @[pingyu](https://github.com/pingyu) + - Fix the issue that TiKV might panic when its coprocessor evaluates pushed-down `LIKE` expressions that involve malformed UTF-8 input or patterns, `BIT` values, or certain collations [#66597](https://github.com/pingcap/tidb/issues/66597) [#67082](https://github.com/pingcap/tidb/issues/67082) [#19811](https://github.com/tikv/tikv/issues/19811) @[jebter](https://github.com/jebter) + - Fix the issue that TiKV might generate inconsistent MVCC state during external SST ingestion when ingestion races with foreground writes, which might cause a panic in transaction status checks [#19891](https://github.com/tikv/tikv/issues/19891) @[gengliqi](https://github.com/gengliqi) + - Fix the issue that TiKV cannot be built with `ENABLE_FIPS=1` in FIPS environments [#19743](https://github.com/tikv/tikv/issues/19743) @[LykxSassinator](https://github.com/LykxSassinator) + - Fix the issue that enabling only background task resource control might cause a 5% to 10% performance regression in write-heavy workloads because TiKV unnecessarily switches the transaction scheduler to priority scheduling [#19858](https://github.com/tikv/tikv/issues/19858) @[glorv](https://github.com/glorv) + - Fix the issue that TiKV might crash during TiDB Lightning data import in `tidb` mode [#18671](https://github.com/tikv/tikv/issues/18671) @[Dog-Du](https://github.com/Dog-Du) + - Fix the issue that the TiKV In-Memory Engine might stop updating region labels after an etcd compaction error, causing the label watch to retry indefinitely [#19792](https://github.com/tikv/tikv/issues/19792) @[akashchakrabortymsc-cmd](https://github.com/akashchakrabortymsc-cmd) + - Fix the issue that Region leader transfer might be blocked indefinitely when in-memory engine cache warmup is stuck and repeated transfer requests arrive before the ACK deadline [#19776](https://github.com/tikv/tikv/issues/19776) @[overvenus](https://github.com/overvenus) + - Fix the issue that TiKV might apply unnecessary write flow control during transient RocksDB compaction spikes [#19667](https://github.com/tikv/tikv/issues/19667) @[hbisheng](https://github.com/hbisheng) + ++ PD + + - Fix the issue that PD `/metric/query` and `/metric/query_range` might be abused for SSRF or expose upstream response details @[rleungx](https://github.com/rleungx) + - Fix the issue that RU tokens might be allocated unevenly among TiDB instances in the same resource group when request rates are uneven, causing high RU wait duration and increased latency on high-demand instances [#9605](https://github.com/tikv/pd/issues/9605) @[JmPotato](https://github.com/JmPotato) + - Fix the issue that PD GlobalConfig gRPC APIs might access etcd keys outside the intended namespace when a client specifies an arbitrary `ConfigPath` or a path-like configuration name [#11079](https://github.com/tikv/pd/issues/11079) @[rleungx](https://github.com/rleungx) + - Fix the issue that PD might establish outbound gRPC connections to caller-selected addresses supplied through `pd-forwarded-host` instead of restricting forwarding targets to the current PD leader's advertised client URLs [#11070](https://github.com/tikv/pd/issues/11070) @[rleungx](https://github.com/rleungx) + - Fix the issue that the resource group client might permanently send `NaN` token requests when a newly created resource group controller races with the periodic state update [#11022](https://github.com/tikv/pd/issues/11022) @[JmPotato](https://github.com/JmPotato) + ++ Tools + + + Backup & Restore (BR) + + - Fix the issue that BR log restore does not enforce the configured rate limit during point-in-time recovery, causing log apply download speed to exceed the limit [#63505](https://github.com/pingcap/tidb/issues/63505) @[Leavrth](https://github.com/Leavrth) + - Fix the issue that BR point-in-time restore for tables with `AUTO_ID_CACHE=1` might cause duplicate-key errors on the first `INSERT` after restore [#69485](https://github.com/pingcap/tidb/issues/69485) @[vldmit](https://github.com/vldmit) + - Fix the issue that BR log backup leaves stale GC safepoints after a log backup task is stopped, which might affect cleanup and safepoint management [#19832](https://github.com/tikv/tikv/issues/19832) @[Leavrth](https://github.com/Leavrth) + - Fix the issue that BR cannot update the SST download rate limit correctly when multiple restore tasks run concurrently, which might cause one task's limit change not to take effect [#19454](https://github.com/tikv/tikv/issues/19454) @[Leavrth](https://github.com/Leavrth) + + + TiCDC + + - Fix the issue that TiCDC might create duplicate dispatchers during maintainer failover, causing downstream write conflicts [#5083](https://github.com/pingcap/ticdc/issues/5083) @[hongyunyan](https://github.com/hongyunyan) + - Fix the issue that TiCDC might produce inconsistent data between upstream and downstream after a Kafka controller failure [#5437](https://github.com/pingcap/ticdc/issues/5437) @[3AceShowHand](https://github.com/3AceShowHand) + - Fix the issue that changefeeds might disappear after upgrading TiCDC when existing changefeed ID metadata uses the legacy `namespace` field instead of `keyspace` in JSON [#4079](https://github.com/pingcap/ticdc/issues/4079) @[wlwilliamx](https://github.com/wlwilliamx) + - Fix the issue that TiCDC checkpoint advancement gets stuck when replicating `CREATE TABLE ... LIKE ...` and the referenced source table is filtered out by the changefeed [#5150](https://github.com/pingcap/ticdc/issues/5150) @[lidezhu](https://github.com/lidezhu) + - Fix the issue that TiCDC owner checkpoint timestamp and lag metrics remain stale after a changefeed is finished, stopped, removed, or moved to another owner [#5490](https://github.com/pingcap/ticdc/issues/5490) @[asddongmen](https://github.com/asddongmen) + - Fix the issue that a changefeed can stall after TiCDC failover or dispatcher reset in the TiCDC new architecture [#5553](https://github.com/pingcap/ticdc/issues/5553) @[asddongmen](https://github.com/asddongmen) + - Fix the issue that a TiCDC maintainer might continue to reschedule or recreate dispatchers after it is removed and shutdown handoff has started [#4827](https://github.com/pingcap/ticdc/issues/4827) @[wlwilliamx](https://github.com/wlwilliamx) + - Fix the issue that TiCDC stale-lock resolution might fail for residual async-commit locks because the `ScanLock` target timestamp can advance TiKV local MaxTS beyond the latest PD TSO [#5418](https://github.com/pingcap/ticdc/issues/5418) @[tenfyzhong](https://github.com/tenfyzhong) + - Fix the issue that TiCDC event service might repeatedly scan the same raw event after a table is deleted, causing scan progress to stall [#5040](https://github.com/pingcap/ticdc/issues/5040) @[asddongmen](https://github.com/asddongmen) + - Fix the issue that maintainer failover might leave in-flight scheduling or merge operators inconsistent and prevent TiCDC table scheduling from converging automatically [#4763](https://github.com/pingcap/ticdc/issues/4763) @[wlwilliamx](https://github.com/wlwilliamx) + - Fix the issue that TiCDC might hang during graceful shutdown when region event pushing is paused by flow control [#5608](https://github.com/pingcap/ticdc/issues/5608) @[lidezhu](https://github.com/lidezhu) + - Fix the issue that a TiCDC table-trigger checkpoint might advance past an add-table DDL before the maintainer acknowledgment arrives, causing newly added table dispatchers to skip subsequent table DDLs [#5401](https://github.com/pingcap/ticdc/issues/5401) @[hongyunyan](https://github.com/hongyunyan) + - Fix the issue that TiCDC might ignore unretryable changefeed errors when checkpoint progress advances in the same maintainer heartbeat, causing the changefeed to remain normal instead of entering the failed state [#5246](https://github.com/pingcap/ticdc/issues/5246) @[3AceShowHand](https://github.com/3AceShowHand) + - Fix the issue that TiCDC changefeeds using Avro or Debezium-Avro might continue to report a healthy status when the Schema Registry returns HTTP 500 errors; affected changefeeds now report a warning status with the registry error in `last_warning` [#5653](https://github.com/pingcap/ticdc/issues/5653) @[wk989898](https://github.com/wk989898) + - Fix the issue that TiCDC might spike memory and CPU usage when bulk-creating a large number of idle changefeeds because the coordinator does not honor the configured scheduler concurrency limit [#4831](https://github.com/pingcap/ticdc/issues/4831) @[wlwilliamx](https://github.com/wlwilliamx) + - Fix the issue that a TiCDC changefeed checkpoint might stall after capture replacement when a dropped view is incorrectly scheduled as a physical table and leaves an orphan dispatcher [#5710](https://github.com/pingcap/ticdc/issues/5710) @[asddongmen](https://github.com/asddongmen) + - Fix the issue that TiCDC might incorrectly reject newer PD, TiKV, or TiCDC versions in the new architecture because of hard-coded maximum compatible versions [#4681](https://github.com/pingcap/ticdc/issues/4681) @[wlwilliamx](https://github.com/wlwilliamx) + - Fix the issue that TiCDC Kafka changefeeds might create topics with broker default partition settings when a consumer starts immediately after changefeed creation, causing message delivery failures and increasing replication lag [#5896](https://github.com/pingcap/ticdc/issues/5896) @[3AceShowHand](https://github.com/3AceShowHand) + - Fix the issue that a stale TiCDC capture might continue writing downstream after losing its etcd session, which might cause duplicate or unsafe downstream writes during failover [#5202](https://github.com/pingcap/ticdc/issues/5202) @[asddongmen](https://github.com/asddongmen) + - Fix the issue that the TiCDC MySQL sink might hang when DDL and metadata operations are blocked by DML sessions after the shared downstream connection pool is exhausted [#5360](https://github.com/pingcap/ticdc/issues/5360) @[hongyunyan](https://github.com/hongyunyan) + - Fix the issue that TiCDC might mark a subscription span as initialized before all underlying Regions finish the initial scan, which might trigger initialization-dependent operations too early [#5658](https://github.com/pingcap/ticdc/issues/5658) @[lidezhu](https://github.com/lidezhu) + - Fix the issue that `cdc cli changefeed resume` reports success for running changefeeds and creates unnecessary temporary resume GC guards for invalid resume requests [#4893](https://github.com/pingcap/ticdc/issues/4893) @[wlwilliamx](https://github.com/wlwilliamx) + - Fix the issue that TiCDC might expose sensitive sink URI information in OpenAPI error messages and logs when sink URI validation fails [#5094](https://github.com/pingcap/ticdc/issues/5094) @[asddongmen](https://github.com/asddongmen) + - Fix the issue that the TiCDC Kafka, Pulsar, and storage consumers might ignore out-of-order replayed DML events after the checkpoint advances, causing downstream data inconsistency [#5713](https://github.com/pingcap/ticdc/issues/5713) @[wk989898](https://github.com/wk989898) + - Fix the issue that TiCDC downstream consumers might apply DML events before the corresponding DDL is processed during failover or message reordering, causing errors such as `Unknown column` [#5587](https://github.com/pingcap/ticdc/issues/5587) @[wk989898](https://github.com/wk989898) + - Fix the issue that TiCDC schema store initialization might retry forever when the initial schema snapshot has been lost by upstream GC, causing a changefeed to remain stuck after recovery from failures such as a network partition [#3249](https://github.com/pingcap/ticdc/issues/3249) @[asddongmen](https://github.com/asddongmen) + - Fix the issue that TiCDC might leak changefeed scan quota or panic when it receives malformed heartbeat or congestion-control messages [#5642](https://github.com/pingcap/ticdc/issues/5642) @[lidezhu](https://github.com/lidezhu) + - Fix the issue that TiCDC in DR Auto-Sync scenarios might trigger a TiKV panic during network partitions because of outdated tiflow dependencies [#5774](https://github.com/pingcap/ticdc/issues/5774) @[wk989898](https://github.com/wk989898) + - Fix the issue that TiCDC might run out of memory when redo apply is executed multiple times by supporting event collector batching based on both event count and total bytes and allowing batch overrides in changefeed configuration [#5950](https://github.com/pingcap/ticdc/issues/5950) @[3AceShowHand](https://github.com/3AceShowHand) + - Fix the issue that TiCDC changefeeds might stall after `TRUNCATE TABLE` on partitioned tables when the event-service scan window is pinned and the DDL barrier cannot make progress [#4365](https://github.com/pingcap/ticdc/issues/4365) @[asddongmen](https://github.com/asddongmen) + - Fix the issue that TiCDC event service scan progress might stall when the global scan window is pinned and a dispatcher has pending syncpoint barriers [#5546](https://github.com/pingcap/ticdc/issues/5546) @[asddongmen](https://github.com/asddongmen) + + + TiDB Lightning + + - Fix the issue that `IMPORT INTO` might report success with inconsistent indexes after a transient conflict-deletion commit error [#69792](https://github.com/pingcap/tidb/issues/69792) @[D3Hunter](https://github.com/D3Hunter) + - Fix the issue that `IMPORT INTO` might fail with the `lock held by current process` error after an import error or retry when local engine files are not cleaned up correctly [#65645](https://github.com/pingcap/tidb/issues/65645) @[D3Hunter](https://github.com/D3Hunter) + - Fix the issue that `IMPORT INTO` silently writes incorrect values for repeated dictionary-encoded Parquet `DECIMAL` values [#70365](https://github.com/pingcap/tidb/issues/70365) @[joechenrh](https://github.com/joechenrh) + - Fix the issue that `IMPORT INTO` or related local backend ingest tasks might hang when job generation is canceled [#69240](https://github.com/pingcap/tidb/issues/69240) @[D3Hunter](https://github.com/D3Hunter)