Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 4 additions & 4 deletions docs-data/property-overrides.json
Original file line number Diff line number Diff line change
Expand Up @@ -687,14 +687,14 @@
"config_scope": "cluster"
},
"default_leaders_preference": {
"description": "Default settings for preferred location of topic partition leaders. It can be either \"none\" (no preference), or \"racks:<rack1>,<rack2>,...\" (prefer brokers with rack ID from the list).\n\nThe list can contain one or more rack IDs. If you specify multiple IDs, Redpanda tries to distribute the partition leader locations equally across brokers in these racks.\n\nIf config_ref:enable_rack_awareness,true,properties/cluster-properties[] is set to `false`, leader pinning is disabled across the cluster.",
"description": "Default settings for preferred location of topic partition leaders. It can be either \"none\" (no preference), or \"racks:<rack1>,<rack2>,...\" (prefer brokers with rack ID from the list).\n\nThe list can contain one or more rack IDs. If you specify multiple IDs, Redpanda tries to distribute the partition leader locations equally across brokers in these racks.\n\nIf prop:enable_rack_awareness[link=true] is set to `false`, leader pinning is disabled across the cluster.",
"related_topics": [
"xref:develop:produce-data/leader-pinning.adoc[Leader pinning]"
],
"config_scope": "cluster"
},
"delete.retention.ms": {
"description": "The retention time for tombstone records in a compacted topic. Redpanda removes tombstone records after the retention limit is exceeded.\n\nIf you have enabled Tiered Storage and set <<redpandaremoteread,`redpanda.remote.read`>> or <<redpandaremotewrite,`redpanda.remote.write`>> for the topic, you cannot enable tombstone removal.\n\nIf both `delete.retention.ms` and the cluster property config_ref:tombstone_retention_ms,true,properties/cluster-properties[] are set, `delete.retention.ms` overrides the cluster level tombstone retention for an individual topic.\n\nThis property supports three states:\n\n* Positive value: Sets the milliseconds to retain tombstone records before removal.\n* 0: Tombstone records are immediately eligible for removal.\n* Negative value: Disables tombstone removal entirely for this topic.",
"description": "The retention time for tombstone records in a compacted topic. Redpanda removes tombstone records after the retention limit is exceeded.\n\nIf you have enabled Tiered Storage and set <<redpandaremoteread,`redpanda.remote.read`>> or <<redpandaremotewrite,`redpanda.remote.write`>> for the topic, you cannot enable tombstone removal.\n\nIf both `delete.retention.ms` and the cluster property prop:tombstone_retention_ms[link=true] are set, `delete.retention.ms` overrides the cluster level tombstone retention for an individual topic.\n\nThis property supports three states:\n\n* Positive value: Sets the milliseconds to retain tombstone records before removal.\n* 0: Tombstone records are immediately eligible for removal.\n* Negative value: Disables tombstone removal entirely for this topic.",
"related_topics": [
"xref:reference:properties/cluster-properties.adoc#tombstone_retention_ms[`tombstone_retention_ms`]",
"xref:manage:cluster-maintenance/compaction-settings.adoc#tombstone-record-removal[Tombstone record removal]"
Expand Down Expand Up @@ -1293,7 +1293,7 @@
"config_scope": "topic"
},
"max.message.bytes": {
"description": "The maximum size of a message or batch of a topic. If a compression type is enabled, `max.message.bytes` sets the maximum size of the compressed message or batch.\n\nIf `max.message.bytes` is set to a positive value, it overrides the cluster property xref:./cluster-properties.adoc#kafka_batch_max_bytes[`kafka_batch_max_bytes`] for the topic.\n\nSet an upper limit for `max.message.bytes` using the cluster property config_ref:kafka_max_message_size_upper_limit_bytes,true,properties/cluster-properties[`kafka_max_message_size_upper_limit_bytes`].",
"description": "The maximum size of a message or batch of a topic. If a compression type is enabled, `max.message.bytes` sets the maximum size of the compressed message or batch.\n\nIf `max.message.bytes` is set to a positive value, it overrides the cluster property xref:./cluster-properties.adoc#kafka_batch_max_bytes[`kafka_batch_max_bytes`] for the topic.\n\nSet an upper limit for `max.message.bytes` using the cluster property prop:kafka_max_message_size_upper_limit_bytes[link=true].",
"related_topics": [
"xref:reference:properties/cluster-properties.adoc#kafka_batch_max_bytes[`kafka_batch_max_bytes`]",
"xref:develop:produce-data/configure-producers.adoc#message-batching[Message batching]",
Expand Down Expand Up @@ -1639,7 +1639,7 @@
]
},
"redpanda.leaders.preference": {
"description": "The preferred location (rack) for partition leaders of a topic.\n\nThis property inherits the value from the config_ref:default_leaders_preference,true,properties/cluster-properties[] cluster configuration property. You may override the cluster-wide setting by specifying the value for individual topics.\n\nIf the cluster configuration property config_ref:enable_rack_awareness,true,properties/cluster-properties[] is set to `false`, Leader Pinning is disabled across the cluster.",
"description": "The preferred location (rack) for partition leaders of a topic.\n\nThis property inherits the value from the prop:default_leaders_preference[link=true] cluster configuration property. You may override the cluster-wide setting by specifying the value for individual topics.\n\nIf the cluster configuration property prop:enable_rack_awareness[link=true] is set to `false`, Leader Pinning is disabled across the cluster.",
"related_topics": [
"xref:develop:produce-data/leader-pinning.adoc[Leader pinning]"
],
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
:page-categories: Deployment
:page-aliases: deploy:deployment-option/self-hosted/manual/production/dev-deployment.adoc

You can deploy Redpanda using well-known configuration properties optimized for a development or test environment. This configuration uses less system resources and simplifies setup, but it's not suitable for a production deployment. For example, in xref:reference:rpk/rpk-redpanda/rpk-redpanda-mode.adoc#development-mode[development mode], the default `group_topic_partitions` is 3, and the default `storage_min_free_bytes` is 1 GiB. In production mode, the default `group_topic_partitions` is 16, and the default `storage_min_free_bytes` is 5 GiB.
You can deploy Redpanda using well-known configuration properties optimized for a development or test environment. This configuration uses less system resources and simplifies setup, but it's not suitable for a production deployment. For example, in xref:reference:rpk/rpk-redpanda/rpk-redpanda-mode.adoc#development-mode[development mode], the default prop:group_topic_partitions[] is 3, and the default prop:storage_min_free_bytes[] is 1 GiB. In production mode, the default prop:group_topic_partitions[] is 16, and the default prop:storage_min_free_bytes[] is 5 GiB.

[NOTE]
====
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -429,7 +429,7 @@ This is useful when using attached disks in instances with ephemeral operating s
|
| Set bucket name to enable Tiered Storage.

| `schema_registry_replication_factor`
| prop:schema_registry_replication_factor[]
| 1
| The replication factor of Schema Registry's internal storage topic.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -190,7 +190,7 @@ administrators can access the host.
* mTLS authentication: You can generate client and server x509 certificates
before starting Redpanda for the first time, refer to them in
`redpanda.yaml`, and use the client certificate when accessing the Admin API.
* Username/password authentication: Use the combination of `admin_api_require_auth`,
* Username/password authentication: Use the combination of prop:admin_api_require_auth[],
`superusers`, and `RP_BOOTSTRAP_USER` to access the Admin API username/password
authentication. You probably still want to enable TLS on the Admin API
endpoint to protect credentials in flight.
Expand All @@ -199,23 +199,23 @@ endpoint to protect credentials in flight.

Seed servers help new brokers join a cluster by directing requests from newly-started brokers to an existing cluster. The xref:reference:properties/broker-properties.adoc#seed_servers[`seed_servers`] broker property controls how Redpanda finds its peers when initially forming a cluster. It is dependent on the xref:reference:properties/broker-properties.adoc#empty_seed_starts_cluster[`empty_seed_starts_cluster`] broker property.

Starting with Redpanda version 22.3, you should explicitly set `empty_seed_starts_cluster` to `false` on every broker, and every broker in the cluster should have the same value set for `seed_servers`. With this set of configurations, Redpanda clusters form with these guidelines:
Starting with Redpanda version 22.3, you should explicitly set prop:empty_seed_starts_cluster[] to `false` on every broker, and every broker in the cluster should have the same value set for prop:seed_servers[]. With this set of configurations, Redpanda clusters form with these guidelines:

* When a broker starts and it is a seed server (its address is in the `seed_servers` list), it waits for all other seed servers to start up, and it forms a cluster with all seed servers as members.
* When a broker starts and it is a seed server (its address is in the prop:seed_servers[] list), it waits for all other seed servers to start up, and it forms a cluster with all seed servers as members.
* When a broker starts and it is not a seed server, it sends requests to the seed servers to join the cluster.

It is essential that all seed servers have identical values for the `seed_servers` list. Redpanda strongly recommends at least three seed servers when forming a cluster. Each seed server decreases the likelihood of unintentionally forming a split brain cluster. To ensure brokers can always discover the cluster, at least one seed server should be available at all times.
It is essential that all seed servers have identical values for the prop:seed_servers[] list. Redpanda strongly recommends at least three seed servers when forming a cluster. Each seed server decreases the likelihood of unintentionally forming a split brain cluster. To ensure brokers can always discover the cluster, at least one seed server should be available at all times.

By default, for backward compatibility, `empty_seed_starts_cluster` is set to `true`, and Redpanda clusters form with the guidelines used prior to version 22.3:
By default, for backward compatibility, prop:empty_seed_starts_cluster[] is set to `true`, and Redpanda clusters form with the guidelines used prior to version 22.3:

* When a broker starts with an empty `seed_servers` list, it creates a single broker cluster with itself as the only member.
* When a broker starts with a non-empty `seed_servers` list, it sends requests to the brokers in that list to join the cluster.
* When a broker starts with an empty prop:seed_servers[] list, it creates a single broker cluster with itself as the only member.
* When a broker starts with a non-empty prop:seed_servers[] list, it sends requests to the brokers in that list to join the cluster.

You should never have more than one broker with an empty `seed_servers` list, which would result in the creation of multiple clusters.
You should never have more than one broker with an empty prop:seed_servers[] list, which would result in the creation of multiple clusters.

IMPORTANT: Redpanda expects its storage to be persistent, and it's an error
to erase a broker's drive and restart it. However, in some environments (like when migrating to a different Node pool on Kubernetes), truly persistent storage is unavailable,
and brokers may find their data volumes erased. For such environments, Redpanda recommends setting `empty_seed_starts_cluster` to false and designating a set of seed servers such that they couldn't lose their storage simultaneously.
and brokers may find their data volumes erased. For such environments, Redpanda recommends setting prop:empty_seed_starts_cluster[] to false and designating a set of seed servers such that they couldn't lose their storage simultaneously.

=== Do not configure broker IDs

Expand All @@ -229,7 +229,7 @@ Redpanda assigns unique IDs automatically to prevent issues such as:
- Brokers with empty disks rejoining the cluster.
- Conflicts during recovery or scaling.

Manually setting or reusing `node_id` values, even for decommissioned brokers, can cause cluster inconsistencies and operational failures.
Manually setting or reusing prop:node_id[] values, even for decommissioned brokers, can cause cluster inconsistencies and operational failures.
====

include::deploy:partial$self-test.adoc[leveloffset=+1]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -106,7 +106,7 @@ redpanda-external NodePort 10.96.137.220 <none> 9644:31644/TCP,909
The headless Service associated with a StatefulSet gives the Pods their network identity in the form of a fully qualified domain name (FQDN). Both Redpanda brokers in the same Redpanda cluster and clients within the same Kubernetes cluster use this FQDN to communicate with each other.

An important requirement of distributed applications such as Redpanda is peer discovery: The ability for each broker to find other brokers in the same cluster.
When each Pod is rolled out, its `seed_servers` field is updated with the FQDN of each Pod in the cluster so that they can discover each other.
When each Pod is rolled out, its prop:seed_servers[] field is updated with the FQDN of each Pod in the cluster so that they can discover each other.

[,bash]
----
Expand Down
2 changes: 1 addition & 1 deletion modules/deploy/partials/requirements.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ endif::[]

- Each Redpanda broker must have at least 2 MB of memory for each topic partition replica.
+
The total memory available for partition replicas is determined as a percentage of the cluster's total memory, which is controlled by the xref:reference:tunable-properties.adoc#topic_partitions_memory_allocation_percent[`topic_partitions_memory_allocation_percent`] setting. Each partition replica consumes xref:reference:tunable-properties.adoc#topic_memory_per_partition[`topic_memory_per_partition`] bytes from this pool. If insufficient memory is available, topic operations will fail. You can adjust the allocation ratio using `topic_partitions_memory_allocation_percent`, but doing so is not recommended, as lowering it may lead to instability or degraded performance.
The total memory available for partition replicas is determined as a percentage of the cluster's total memory, which is controlled by the xref:reference:tunable-properties.adoc#topic_partitions_memory_allocation_percent[`topic_partitions_memory_allocation_percent`] setting. Each partition replica consumes xref:reference:tunable-properties.adoc#topic_memory_per_partition[`topic_memory_per_partition`] bytes from this pool. If insufficient memory is available, topic operations will fail. You can adjust the allocation ratio using prop:topic_partitions_memory_allocation_percent[], but doing so is not recommended, as lowering it may lead to instability or degraded performance.

*Recommendations*:

Expand Down
6 changes: 3 additions & 3 deletions modules/develop/pages/config-topics.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -138,17 +138,17 @@ To enable write caching by default in all user topics, set the cluster-level pro

`rpk cluster config set write_caching_default=true`

With `write_caching_default` set to true at the cluster level, Redpanda fsyncs to disk according to xref:reference:cluster-properties.adoc#raft_replica_max_pending_flush_bytes[`raft_replica_max_pending_flush_bytes`] and xref:reference:cluster-properties.adoc#raft_replica_max_flush_delay_ms[`raft_replica_max_flush_delay_ms`], whichever is reached first.
With prop:write_caching_default[] set to true at the cluster level, Redpanda fsyncs to disk according to xref:reference:cluster-properties.adoc#raft_replica_max_pending_flush_bytes[`raft_replica_max_pending_flush_bytes`] and xref:reference:cluster-properties.adoc#raft_replica_max_flush_delay_ms[`raft_replica_max_flush_delay_ms`], whichever is reached first.

endif::[]

==== Configure at topic level

To override the cluster-level setting at the topic level, set the topic-level property `write.caching`:
To override the cluster-level setting at the topic level, set the topic-level property prop:write.caching[]:

`rpk topic alter-config my_topic --set write.caching=true`

With `write.caching` enabled at the topic level, Redpanda fsyncs to disk according to `flush.ms` and `flush.bytes`, whichever is reached first.
With prop:write.caching[] enabled at the topic level, Redpanda fsyncs to disk according to prop:flush.ms[] and prop:flush.bytes[], whichever is reached first.

=== Remove a configuration setting

Expand Down
2 changes: 1 addition & 1 deletion modules/develop/pages/data-transforms/configure.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,7 @@ Set the following based on the number of functions you have and the amount of me
- xref:reference:properties/cluster-properties.adoc#data_transforms_per_core_memory_reservation[`data_transforms_per_core_memory_reservation`]: Increase this setting if you plan to deploy a large number of data transforms or if your transforms are memory-intensive. Reducing it may limit the number of concurrent transforms.
- xref:reference:properties/cluster-properties.adoc#data_transforms_per_function_memory_limit[`data_transforms_per_function_memory_limit`]: Adjust this setting if individual transform functions require more memory to process records efficiently. Reducing it may cause memory errors in complex transforms.

The maximum number of functions that can be deployed to a cluster is equal to `data_transforms_per_core_memory_reservation` / `data_transforms_per_function_memory_limit`. When that limit is hit, Redpanda cannot allocate memory for the VM and the transforms stay in `errored` states.
The maximum number of functions that can be deployed to a cluster is equal to prop:data_transforms_per_core_memory_reservation[] / prop:data_transforms_per_function_memory_limit[]. When that limit is hit, Redpanda cannot allocate memory for the VM and the transforms stay in `errored` states.
endif::[]

ifndef::env-cloud[]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ When you deploy a data transform to a Redpanda broker, it stores the Wasm byteco

- Each shard can run only one instance of the transform function at a time to ensure efficient resource utilization and prevent overload.
ifndef::env-cloud[]
- Memory for each function is reserved within the broker with the `data_transforms_per_core_memory_reservation` and `data_transforms_per_function_memory_limit` properties. See xref:develop:data-transforms/configure.adoc#resources[Configure memory for data transforms].
- Memory for each function is reserved within the broker with the prop:data_transforms_per_core_memory_reservation[] and prop:data_transforms_per_function_memory_limit[] properties. See xref:develop:data-transforms/configure.adoc#resources[Configure memory for data transforms].
endif::[]
- CPU time is dynamically allocated to the Wasm runtime to ensure that the code does not run forever and cannot block the broker from handling traffic or doing other work, such as Tiered Storage uploads.

Expand Down
2 changes: 1 addition & 1 deletion modules/develop/pages/kafka-clients.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,7 @@ endif::[]
* HTTP Proxy (pandaproxy): Unlike other REST proxy implementations in the Kafka ecosystem, Redpanda HTTP Proxy does not support topic and ACLs CRUD through the HTTP Proxy. HTTP Proxy is designed for clients producing and consuming data that do not perform administrative functions.
ifdef::env-cloud[]
+
* The `delete.retention.ms` topic configuration in Kafka is not supported. Tombstone markers are not removed for topics with a `compact` xref:get-started:config-topics.adoc#change-the-cleanup-policy[cleanup policy]. Redpanda only deletes tombstone markers when topics with a cleanup policy of `compact,delete` have reached their xref:get-started:create-topic.adoc[retention limits].
* The prop:delete.retention.ms[] topic configuration in Kafka is not supported. Tombstone markers are not removed for topics with a `compact` xref:get-started:config-topics.adoc#change-the-cleanup-policy[cleanup policy]. Redpanda only deletes tombstone markers when topics with a cleanup policy of `compact,delete` have reached their xref:get-started:create-topic.adoc[retention limits].
endif::[]
ifndef::env-cloud[]
+
Expand Down
Loading
Loading