From ad1a06b4cce2564e4733353c98c4941f900dd76a Mon Sep 17 00:00:00 2001 From: JakeSCahill Date: Thu, 6 Aug 2026 17:40:41 +0100 Subject: [PATCH 1/2] docs: document how to reduce a topic's partition count (DOC-559) The docs said only that partition counts cannot be decreased, so users asking how to reduce partitions got no supported path. Add a validated copy-to-new-topic procedure using a Redpanda Connect pipeline that preserves record keys, with a lag-based completion check, and point the Kubernetes topic page at it. --- .../pages/manage-topics/config-topics.adoc | 50 +++++++++++++++++++ .../pages/kubernetes/k-manage-topics.adoc | 2 +- 2 files changed, 51 insertions(+), 1 deletion(-) diff --git a/modules/develop/pages/manage-topics/config-topics.adoc b/modules/develop/pages/manage-topics/config-topics.adoc index 5c1084508f..1adffa8dbe 100644 --- a/modules/develop/pages/manage-topics/config-topics.adoc +++ b/modules/develop/pages/manage-topics/config-topics.adoc @@ -134,6 +134,56 @@ Note that `--num <#>` is the number of partitions to _add_, not the total number include::develop:partial$balance-existing-topic-redistribution.adoc[] +=== Reduce the number of partitions + +You cannot reduce the number of partitions on an existing topic. The Kafka API does not support it, because records are distributed across partitions by key, and removing partitions would orphan their data. + +To move a topic's data to a smaller number of partitions, copy the data to a new topic: + +. Create a new topic with the target number of partitions: ++ +[,bash] +---- +rpk topic create --partitions +---- + +. Copy the data with a xref:connect:get-started:about.adoc[Redpanda Connect] pipeline. The following configuration copies every record from the beginning of the original topic and preserves record keys, so the default partitioner distributes records across the new partitions by key: ++ +.`reduce-partitions.yaml` +[,yaml] +---- +input: + redpanda: + seed_brokers: [""] + topics: [""] + consumer_group: partition-reduction-copy + start_from_oldest: true +output: + redpanda: + seed_brokers: [""] + topic: + key: ${! @kafka_key } +---- ++ +[,bash] +---- +rpk connect run reduce-partitions.yaml +---- + +. Wait for the copy to catch up. The copy is complete when the consumer group reports a lag of zero for every partition of the original topic and producers have stopped writing to it: ++ +[,bash] +---- +rpk group describe partition-reduction-copy +---- + +. Point your producers and consumers at the new topic. + +. When you no longer need the original topic, delete it to reclaim storage. See <>. + +CAUTION: The copy preserves record keys but not the global order of records across partitions. Stop producers before you switch over so that no records are written to the original topic after the copy catches up. + + ifndef::env-cloud[] [[change-the-replication-factor]] === Change the replication factor diff --git a/modules/manage/pages/kubernetes/k-manage-topics.adoc b/modules/manage/pages/kubernetes/k-manage-topics.adoc index a8297e1def..cc52d5f7a1 100644 --- a/modules/manage/pages/kubernetes/k-manage-topics.adoc +++ b/modules/manage/pages/kubernetes/k-manage-topics.adoc @@ -52,7 +52,7 @@ Valid names must consist of lowercase alphanumeric characters, hyphens (-), or p + - ``: The namespace in which to deploy the Topic resource. The Topic resource must be deployed in the same namespace as the Redpanda resource defined in `clusterRef.name`. - ``: The name of the Redpanda resource that defines the Redpanda cluster in which you want to create the topic. -- ``: The number of topic shards distributed across the brokers in a Redpanda cluster. This value cannot be decreased post-creation. Overrides the default cluster property xref:reference:cluster-properties.adoc#default_topic_partitions[`default_topic_partitions`]. +- ``: The number of topic shards distributed across the brokers in a Redpanda cluster. This value cannot be decreased after topic creation. To move a topic's data to fewer partitions, see xref:develop:manage-topics/config-topics.adoc#reduce-the-number-of-partitions[Reduce the number of partitions]. Overrides the default cluster property xref:reference:cluster-properties.adoc#default_topic_partitions[`default_topic_partitions`]. - ``: Specifies the number of topic replicas. The value must be an odd number. Overrides the default cluster property xref:reference:cluster-properties.adoc#default_topic_replications[`default_topic_replications`]. . Apply the manifest: From e094e58e6a8d4a28c91c20e55bcdae2675b0419c Mon Sep 17 00:00:00 2001 From: JakeSCahill Date: Sat, 8 Aug 2026 12:49:22 +0100 Subject: [PATCH 2/2] docs: add verified rpk output and address review feedback (DOC-559) Re-ran the procedure against a local single-broker cluster (rpk container start) and added the real output of each rpk command as collapsible example blocks: rpk topic describe on the source, rpk topic create, rpk connect run, rpk group describe at LAG 0, and a record-count comparison of both topics (9 records across 3 partitions -> 9 on 1). Review feedback: - The pipeline used `start_from_oldest`, which the redpanda input still accepts but reports as deprecated (`rpk connect lint --deprecated`); the documented field is `start_offset`, whose default is `earliest`. Verified against the component schema from `rpk connect create` (Connect 4.96.1). The original validation run passed despite the wrong field name precisely because the default is already `earliest`. - Added a step to inspect the source topic with `rpk topic describe` and carry over the replication factor and every `DYNAMIC_TOPIC_CONFIG` override, so the new topic does not silently fall back to cluster defaults for Tiered Storage, cleanup policy, or retention. - The consumer group name is now unique per copy (`-to-`), with a note that `start_offset: earliest` applies only when the group has no committed offset, so a reused name skips records. - Replaced the "every record from the beginning" claim with what `earliest` actually means: the oldest retained record, with retention and compaction still running during the copy. - Documented at-least-once delivery: a restart or retry can duplicate a record, and zero lag is a committed position, not proof of no duplicates. Added the record-count comparison as the check. - Made the cutover order explicit: stop producers, wait for zero lag, compare counts, switch clients, then delete the original topic. - Documented that consumer group offsets are per topic, so consumers replay the new topic from the beginning unless moved with `rpk group seek`. - Corrected the partition-assignment explanation: keyed records hash to a partition, unkeyed records go wherever the producer's partitioner puts them, and existing records stay in the partition they landed in. Co-Authored-By: Claude Opus 5 --- .../pages/manage-topics/config-topics.adoc | 137 ++++++++++++++++-- 1 file changed, 124 insertions(+), 13 deletions(-) diff --git a/modules/develop/pages/manage-topics/config-topics.adoc b/modules/develop/pages/manage-topics/config-topics.adoc index 1adffa8dbe..7eee27c97e 100644 --- a/modules/develop/pages/manage-topics/config-topics.adoc +++ b/modules/develop/pages/manage-topics/config-topics.adoc @@ -136,18 +136,67 @@ include::develop:partial$balance-existing-topic-redistribution.adoc[] === Reduce the number of partitions -You cannot reduce the number of partitions on an existing topic. The Kafka API does not support it, because records are distributed across partitions by key, and removing partitions would orphan their data. +You cannot reduce the number of partitions on an existing topic. The Kafka API does not support it: a record's partition is chosen when the record is produced, and records that have already been written stay in the partition where they landed, so removing a partition would orphan its data. Redpanda assigns a record that has a key to a partition by hashing the key, and leaves a record without a key to the producer's partitioner, which usually spreads such records across all available partitions. -To move a topic's data to a smaller number of partitions, copy the data to a new topic: +To move a topic's data to fewer partitions, copy it to a new topic and switch your clients over. Before you start, check that your applications can tolerate the following: -. Create a new topic with the target number of partitions: +* *Duplicates*: Redpanda Connect delivers records at least once, so a restart or a retry during the copy can write the same record to the new topic twice. A lag of zero shows only how far the copy has committed, not that the new topic is free of duplicates. Either make your consumers idempotent, or deduplicate on a record ID after the copy. +* *Ordering*: records keep their keys, so all records for a key still land on one partition and keep their order relative to each other, but the global order of records across partitions is not preserved. +* *Retention*: the copy starts at the oldest record that is still retained. Records that retention or compaction has already removed cannot be copied, and both keep running during the copy, so complete the copy well within the topic's retention period. +* *Consumer offsets*: consumer group offsets are stored per topic, so the offsets your consumers committed on the original topic do not carry over. Each consumer starts from the beginning of the new topic and replays what the copy wrote, unless you set its offsets explicitly with `rpk group seek`. +* *Write downtime*: producers must stop writing to the original topic before you switch clients over, so plan a window in which the topic accepts no writes. + +The following procedure reduces a topic named `orders` from three partitions to one. + +. Check the configuration of the original topic so that you can recreate it. Note the replication factor, and every row whose `SOURCE` is `DYNAMIC_TOPIC_CONFIG`, which is an override you must set on the new topic: ++ +[,bash] +---- +rpk topic describe orders +---- ++ +.Example output (abbreviated) +[%collapsible] +==== +[,bash,role="no-wrap"] +---- +SUMMARY +======= +NAME orders +PARTITIONS 3 +REPLICAS 1 + +CONFIGS +======= +KEY VALUE SOURCE +cleanup.policy delete DEFAULT_CONFIG +retention.bytes -1 DEFAULT_CONFIG +retention.local.target.ms 86400000 DEFAULT_CONFIG +retention.ms 604800000 DYNAMIC_TOPIC_CONFIG +segment.bytes 134217728 DEFAULT_CONFIG +---- +==== ++ +Here, only `retention.ms` is an override. If the topic has Tiered Storage settings, a custom cleanup policy, or other overrides, carry all of them over: a new topic created without them silently falls back to the cluster defaults. + +. Create the new topic with the target number of partitions, the replication factor of the original topic, and each override from the previous step: + [,bash] ---- -rpk topic create --partitions +rpk topic create orders-reduced --partitions 1 --replicas 1 --topic-config retention.ms=604800000 +---- ++ +.Example output +[%collapsible] +==== +[,bash,role="no-wrap"] +---- +TOPIC STATUS +orders-reduced OK ---- +==== -. Copy the data with a xref:connect:get-started:about.adoc[Redpanda Connect] pipeline. The following configuration copies every record from the beginning of the original topic and preserves record keys, so the default partitioner distributes records across the new partitions by key: +. Copy the data with a xref:connect:get-started:about.adoc[Redpanda Connect] pipeline. This configuration reads all records that are still available in the original topic and preserves record keys, so records for the same key land on the same partition of the new topic: + .`reduce-partitions.yaml` [,yaml] @@ -155,33 +204,95 @@ rpk topic create --partitions input: redpanda: seed_brokers: [""] - topics: [""] - consumer_group: partition-reduction-copy - start_from_oldest: true + topics: ["orders"] + consumer_group: orders-to-orders-reduced + start_offset: earliest output: redpanda: seed_brokers: [""] - topic: + topic: orders-reduced key: ${! @kafka_key } ---- + +Give the consumer group a name that is unique to this copy, such as `-to-`. `start_offset: earliest` applies only when the group has no committed offset, so a group name that has been used before resumes from where it left off and skips records. ++ [,bash] ---- rpk connect run reduce-partitions.yaml ---- ++ +.Example output +[%collapsible] +==== +[,bash,role="no-wrap"] +---- +level=info msg="Launching a Redpanda Connect instance, use CTRL+C to close" +level=info msg="Output type redpanda is now active" +level=info msg="Input type redpanda is now active" +---- +==== + +. Stop the producers that write to the original topic. Leave the pipeline running so that it copies the last records they wrote. -. Wait for the copy to catch up. The copy is complete when the consumer group reports a lag of zero for every partition of the original topic and producers have stopped writing to it: +. Wait for the copy to drain. It is complete when the consumer group reports a `LAG` of `0` for every partition of the original topic: ++ +[,bash] +---- +rpk group describe orders-to-orders-reduced +---- ++ +.Example output +[%collapsible] +==== +[,bash,role="no-wrap"] +---- +GROUP orders-to-orders-reduced +COORDINATOR-NODE 0 +COORDINATOR-PARTITION __consumer_offsets/0 +STATE Stable +BALANCER cooperative-sticky +MEMBERS 1 +TOTAL-LAG 0 + +TOPIC PARTITION CURRENT-OFFSET LOG-START-OFFSET LOG-END-OFFSET LAG MEMBER-ID CLIENT-ID HOST +orders 0 3 0 3 0 redpanda-connect-15d7a80f-590f-4cde-bc16-4854fa2754 redpanda-connect 10.0.0.1 +orders 1 3 0 3 0 redpanda-connect-15d7a80f-590f-4cde-bc16-4854fa2754 redpanda-connect 10.0.0.1 +orders 2 3 0 3 0 redpanda-connect-15d7a80f-590f-4cde-bc16-4854fa2754 redpanda-connect 10.0.0.1 +---- +==== + +. Compare the record counts of the two topics. For each topic, the number of available records is the sum of `HIGH-WATERMARK` minus `LOG-START-OFFSET` across its partitions. A higher count on the new topic means the copy wrote duplicates: + [,bash] ---- -rpk group describe partition-reduction-copy +rpk topic describe orders -p +rpk topic describe orders-reduced -p +---- ++ +.Example output +[%collapsible] +==== +[,bash,role="no-wrap"] ---- +PARTITION LEADER EPOCH REPLICAS LOG-START-OFFSET HIGH-WATERMARK +0 0 1 [0] 0 3 +1 0 1 [0] 0 3 +2 0 1 [0] 0 3 + +PARTITION LEADER EPOCH REPLICAS LOG-START-OFFSET HIGH-WATERMARK +0 0 1 [0] 0 9 +---- +==== ++ +Nine records across the three original partitions, and the same nine on the single partition of the new topic. + +. Point your producers and consumers at the new topic. Consumers start from the beginning of the new topic unless you set their offsets with `rpk group seek`. -. Point your producers and consumers at the new topic. +. Stop the pipeline with kbd:[Ctrl+C]. . When you no longer need the original topic, delete it to reclaim storage. See <>. -CAUTION: The copy preserves record keys but not the global order of records across partitions. Stop producers before you switch over so that no records are written to the original topic after the copy catches up. +CAUTION: Do not delete the original topic until the new topic holds the data you expect and your consumers are running against it. Deleting a topic deletes its data. ifndef::env-cloud[]