docs: document how to reduce a topic's partition count (DOC-559) - #1891
Open
JakeSCahill wants to merge 2 commits into
Open
docs: document how to reduce a topic's partition count (DOC-559)#1891JakeSCahill wants to merge 2 commits into
JakeSCahill wants to merge 2 commits into
Conversation
The docs said only that partition counts cannot be decreased, so users asking how to reduce partitions got no supported path. Add a validated copy-to-new-topic procedure using a Redpanda Connect pipeline that preserves record keys, with a lag-based completion check, and point the Kubernetes topic page at it.
✅ Deploy Preview for redpanda-docs-preview ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
Contributor
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThe documentation states that Kafka cannot decrease a topic’s partition count after creation. It documents creating a replacement topic with fewer partitions, copying records with Redpanda Connect, waiting for consumer lag to reach zero, switching producers and consumers, and deleting the original topic. It also notes that global cross-partition ordering is not preserved. Estimated code review effort: 1 (Trivial) | ~5 minutes Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 8
🧹 Nitpick comments (2)
modules/manage/pages/kubernetes/k-manage-topics.adoc (1)
55-55: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse an empty xref target.
Change the link to
xref:develop:manage-topics/config-topics.adoc#reduce-the-number-of-partitions[]so the target heading supplies the display text.Based on learnings, empty xref brackets keep link text synchronized with the referenced heading.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@modules/manage/pages/kubernetes/k-manage-topics.adoc` at line 55, Update the xref in the <number-of-partitions> documentation so the reduce-the-number-of-partitions target uses empty brackets, allowing the referenced heading to provide the link text while preserving the existing destination.Source: Learnings
modules/develop/pages/manage-topics/config-topics.adoc (1)
150-166: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winState the record-fidelity boundary.
The input exposes source timestamps and record headers. The output uses the current timestamp when
timestamp_msis empty, and output metadata headers are opt-in. This configuration explicitly preserves the key but does not create an exact Kafka-record copy for applications that depend on timestamps or headers. Preserve required metadata or state this limitation. (docs.redpanda.com)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@modules/develop/pages/manage-topics/config-topics.adoc` around lines 150 - 166, Update the Redpanda Connect example around the reduce-partitions configuration to state that it preserves record keys but is not an exact Kafka-record copy: timestamps may default to the current time when absent, and source headers require explicit output configuration. Either configure preservation of required timestamps and headers or clearly document this metadata limitation.Source: MCP tools
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@modules/develop/pages/manage-topics/config-topics.adoc`:
- Around line 173-184: Update the topic migration instructions around “Wait for
the copy to catch up” to explicitly require stopping producers before checking
lag. State the complete order: stop producers, wait for zero lag, verify the
original topic receives no new writes, switch producers and consumers to the new
topic, then delete the original topic; adjust the existing caution to avoid
implying a later producer stop is sufficient.
- Around line 143-148: Update the topic replacement procedure after the
source-topic identification step to inspect its configuration with rpk topic
describe, then extend the rpk topic create command with the source replication
factor and all non-default topic settings via -r/--topic-config, including
Tiered Storage, cleanup policy, and retention overrides.
- Around line 150-160: Update the Redpanda Connect copy instructions to state
that the pipeline copies all currently available records, rather than every
record from the beginning. Add guidance to complete the copy before source
retention or compaction removes the required data, while preserving the existing
start_from_oldest configuration.
- Line 139: Update the partition-assignment explanation in the existing-topic
guidance to distinguish null or blank-key records, which use round-robin
assignment, from keyed records, which use key hashing. State that existing
records remain tied to their current partitions, so partitions cannot be reduced
in place.
- Around line 180-182: Update the topic-switching procedure near “Point your
producers and consumers at the new topic” to define how existing consumer-group
offsets are handled for the new topic, including explicit migration or reset
steps, or clearly require consumers to accept replay/reset behavior. Clarify
that partition-reduction-copy tracks only the copy pipeline state, not consumer
offsets, before instructing users to delete the original topic.
- Around line 150-178: The partition-reduction procedure around the Redpanda
Connect copy and “Wait for the copy to catch up” steps must document
at-least-once delivery and possible duplicate records in <new-topic> after
restarts or retries. Clarify that zero consumer lag only reflects committed
offsets, then add guidance to identify or remove duplicates before cutover.
- Around line 159-160: Update the source-to-target copy configuration’s
consumer_group from the shared partition-reduction-copy value to a unique name
for this migration, and update the associated rpk group describe reference to
use the same group name. Preserve start_from_oldest: true.
- Line 160: Replace the unsupported start_from_oldest setting in the
input.redpanda configuration with the documented start_offset field set to
earliest, preserving the intended behavior of starting from the oldest available
messages.
---
Nitpick comments:
In `@modules/develop/pages/manage-topics/config-topics.adoc`:
- Around line 150-166: Update the Redpanda Connect example around the
reduce-partitions configuration to state that it preserves record keys but is
not an exact Kafka-record copy: timestamps may default to the current time when
absent, and source headers require explicit output configuration. Either
configure preservation of required timestamps and headers or clearly document
this metadata limitation.
In `@modules/manage/pages/kubernetes/k-manage-topics.adoc`:
- Line 55: Update the xref in the <number-of-partitions> documentation so the
reduce-the-number-of-partitions target uses empty brackets, allowing the
referenced heading to provide the link text while preserving the existing
destination.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 4d95b1d5-7867-4f9d-8127-9979fb29198d
📒 Files selected for processing (2)
modules/develop/pages/manage-topics/config-topics.adocmodules/manage/pages/kubernetes/k-manage-topics.adoc
Re-ran the procedure against a local single-broker cluster (rpk container start) and added the real output of each rpk command as collapsible example blocks: rpk topic describe on the source, rpk topic create, rpk connect run, rpk group describe at LAG 0, and a record-count comparison of both topics (9 records across 3 partitions -> 9 on 1). Review feedback: - The pipeline used `start_from_oldest`, which the redpanda input still accepts but reports as deprecated (`rpk connect lint --deprecated`); the documented field is `start_offset`, whose default is `earliest`. Verified against the component schema from `rpk connect create` (Connect 4.96.1). The original validation run passed despite the wrong field name precisely because the default is already `earliest`. - Added a step to inspect the source topic with `rpk topic describe` and carry over the replication factor and every `DYNAMIC_TOPIC_CONFIG` override, so the new topic does not silently fall back to cluster defaults for Tiered Storage, cleanup policy, or retention. - The consumer group name is now unique per copy (`<original-topic>-to-<new-topic>`), with a note that `start_offset: earliest` applies only when the group has no committed offset, so a reused name skips records. - Replaced the "every record from the beginning" claim with what `earliest` actually means: the oldest retained record, with retention and compaction still running during the copy. - Documented at-least-once delivery: a restart or retry can duplicate a record, and zero lag is a committed position, not proof of no duplicates. Added the record-count comparison as the check. - Made the cutover order explicit: stop producers, wait for zero lag, compare counts, switch clients, then delete the original topic. - Documented that consumer group offsets are per topic, so consumers replay the new topic from the beginning unless moved with `rpk group seek`. - Corrected the partition-assignment explanation: keyed records hash to a partition, unkeyed records go wherever the producer's partitioner puts them, and existing records stay in the partition they landed in. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Resolves DOC-559 (September 2024): the docs said only that partition counts cannot be decreased, so users (and Kapa) had no supported answer for reducing partitions on a topic with data.
Changes
develop/manage-topics/config-topics.adoc: new Reduce the number of partitions section after Add partitions. Explains why reduction is unsupported and documents the copy-to-new-topic procedure: create the target topic, copy with a Redpanda Connect pipeline that reads from the oldest offset and preserves record keys (key: ${! @kafka_key }), use consumer-group lag as the completion signal, switch clients over, delete the original. A CAUTION covers the two real gotchas: global cross-partition order is not preserved, and producers must stop before cutover.manage/kubernetes/k-manage-topics.adoc: the "cannot be decreased" bullet now links to the new section instead of dead-ending.Validation
The exact pipeline was executed against a local single-broker Redpanda (
rpk container start): 9 keyed records across 3 partitions copied to a 1-partition topic with all 9 records and all keys intact, andrpk group describeshowing LAG 0 on every source partition as the documented stop signal. The Connect xref targets theconnect:get-started:about.adocpage (component name verified against rp-connect-docs antora.yml).🤖 Generated with Claude Code