From 3539c8a11277e30e8bf125628e1c54fff7b68c9b Mon Sep 17 00:00:00 2001 From: Ray Offiah Date: Wed, 27 May 2026 11:19:34 +0100 Subject: [PATCH 1/9] [DOC-14129]: Magma Compaction rate limiter documentation Added page detailing REST call for magma compaction. Update menu navigation and index page. Signed-off-by: Ray Offiah --- modules/ROOT/nav.adoc | 1 + .../rest-magma-compaction-rate-limiting.adoc | 109 ++++++++++++++++++ .../rest-memory-and-storage-table.adoc | 8 ++ 3 files changed, 118 insertions(+) create mode 100644 modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc diff --git a/modules/ROOT/nav.adoc b/modules/ROOT/nav.adoc index e58f9a4279..19c55f1ff8 100644 --- a/modules/ROOT/nav.adoc +++ b/modules/ROOT/nav.adoc @@ -433,6 +433,7 @@ include::cli:partial$cbcli/nav.adoc[] *** xref:rest-api:rest-compact-post.adoc[Performing Compaction Manually] *** xref:rest-api:rest-autocompact-global.adoc[Auto-Compaction: Global] *** xref:rest-api:rest-autocompact-per-bucket.adoc[Auto-Compaction: Per Bucket] + *** xref:rest-api:rest-magma-compaction-rate-limiting.adoc[Magma Compaction Rate Limiting] *** xref:rest-api:rest-magma-compression-per-bucket.adoc[Magma Compression] ** xref:rest-api:rest-rza.adoc[Server Groups API] diff --git a/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc b/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc new file mode 100644 index 0000000000..e3900b3c25 --- /dev/null +++ b/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc @@ -0,0 +1,109 @@ += Magma Compaction Rate Limiting +:description: This API is used to set the rate limit for Magma compaction threads. +:stem: asciimath +[abstract] +{description} + +== Description + +Use these API calls to set the rate limit (in MB per second). +This setting is a global rate limit for the Memcached process shared across buckets. +The strategy for this rate limiting is to set the block waiting time on the compaction threads to maintain the specified limits. + +Magma supports 2 types of compaction: + +Log-Structured Merge (LSM) Tree Compactions:: +These compactions are short-running and essential for preserving the structure of the Log-Structured Merge tree. +Delaying these operations can lead to increased read latency, as the read path may need to use multiple files, resulting in higher read amplification. +If this type of compaction is lagging for level-0 of the LSM Tree, the system performs level-0 compaction within the writer threads themselves + +Data Log Compactions:: +Data compactions can be de-prioritized compared to LSM Tree compactions. +The only consequence of slowing these operations is increased space usage. + +== Settings + +[cols="3,3,1"] +|=== +| Setting | Description| Range + +| `magma_compaction_rate_limit` +| Controls the I/O bandwidth +(given in bytes per second) +that the compactions across all the shards can take to consume. +Setting this value to 0 disables compaction rate limiting. +| stem:[0 … 2^64 - 1] +(default: 0) + +| `magma_enable_compaction_dataonly_ratelimiting` +| Controls whether the compaction rate limiting runs in `data-only` mode. +(only fragment reduction compactions are rate-limited) +| `true`, `false` (default:``false``) + +|=== + +== HTTP Methods and URIs + +[source, shell] +---- +GET /pools/default/settings/memcached/global +POST /pools/default/settings/memcached/global + +---- + +== Examples + +.Retrieve the compaction limit settings +[source, shell] +---- +curl -u Administrator:password -X GET http://localhost:8091/pools/default/settings/memcached/global +---- + +Running this command returns the global settings: + +[source, json] +---- +{ + "max_connections": 20000, + "system_connections": 5000, + "magma_enable_compaction_dataonly_ratelimiting": false +} +---- + +.Setting the compaction rate +[source, shell] +---- + curl -u Administrator:password -X POST localhost:8091/pools/default/settings/memcached/global \ +-d magma_compaction_rate_limit=500 | jq +---- + +The call executes and returns a string containing the new setting. + +[source, json] +---- +{ + "max_connections": 20000, + "system_connections": 5000, + "magma_compaction_rate_limit": 500, + "magma_enable_compaction_dataonly_ratelimiting": true +} +---- + +.Setting Compaction for data only rate limiting +[source,shell] +---- +curl -u Administrator:password -X POST localhost:8091/pools/default/settings/memcached/global \ +-d magma_enable_compaction_dataonly_ratelimiting=false | jq +---- + +The call executes and returns a string containing the new setting. + +[source, json] +---- +{ + "max_connections": 20000, + "system_connections": 5000, + "magma_compaction_rate_limit": 500, + "magma_enable_compaction_dataonly_ratelimiting": false +} +---- diff --git a/modules/rest-api/partials/rest-memory-and-storage-table.adoc b/modules/rest-api/partials/rest-memory-and-storage-table.adoc index 06a09eca8e..9f4feb7827 100644 --- a/modules/rest-api/partials/rest-memory-and-storage-table.adoc +++ b/modules/rest-api/partials/rest-memory-and-storage-table.adoc @@ -45,4 +45,12 @@ | `POST` | `/pools/default/buckets/[bucket-name]` | xref:rest-api:rest-magma-compression-per-bucket.adoc[Magma compression: Per Bucket] + +| `GET` +| /pools/default/settings/memcached/global +| xref:rest-api:rest-magma-compaction-rate-limiting.adoc[Magma Compaction Rate Limiting] + +| `POST` +| /pools/default/settings/memcached/global +| xref:rest-api:rest-magma-compaction-rate-limiting.adoc[Magma Compaction Rate Limiting] |=== From 006644f86a9c373ae642d644decfbde4ff0669e8 Mon Sep 17 00:00:00 2001 From: Ray Offiah <77050471+RayOffiah@users.noreply.github.com> Date: Wed, 27 May 2026 11:36:10 +0100 Subject: [PATCH 2/9] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc b/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc index e3900b3c25..5aeeabe479 100644 --- a/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc +++ b/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc @@ -6,7 +6,7 @@ == Description -Use these API calls to set the rate limit (in MB per second). +Use these API calls to set the rate limit (in bytes per second). This setting is a global rate limit for the Memcached process shared across buckets. The strategy for this rate limiting is to set the block waiting time on the compaction threads to maintain the specified limits. From 6c9d8f50626b7d4346ea459f2b448d49e2ca97c5 Mon Sep 17 00:00:00 2001 From: Ray Offiah Date: Wed, 27 May 2026 11:52:23 +0100 Subject: [PATCH 3/9] [DOC-14129]: Magma Compaction rate limiter documentation Changes made following co-pilot review. Signed-off-by: Ray Offiah --- .../pages/rest-magma-compaction-rate-limiting.adoc | 8 ++++---- .../rest-api/partials/rest-memory-and-storage-table.adoc | 4 ++-- 2 files changed, 6 insertions(+), 6 deletions(-) diff --git a/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc b/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc index 5aeeabe479..b909fe8ebe 100644 --- a/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc +++ b/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc @@ -77,7 +77,7 @@ Running this command returns the global settings: -d magma_compaction_rate_limit=500 | jq ---- -The call executes and returns a string containing the new setting. +The call executes and returns a JSON object containing the new setting. [source, json] ---- @@ -85,7 +85,7 @@ The call executes and returns a string containing the new setting. "max_connections": 20000, "system_connections": 5000, "magma_compaction_rate_limit": 500, - "magma_enable_compaction_dataonly_ratelimiting": true + "magma_enable_compaction_dataonly_ratelimiting": false } ---- @@ -93,7 +93,7 @@ The call executes and returns a string containing the new setting. [source,shell] ---- curl -u Administrator:password -X POST localhost:8091/pools/default/settings/memcached/global \ --d magma_enable_compaction_dataonly_ratelimiting=false | jq +-d magma_enable_compaction_dataonly_ratelimiting=true | jq ---- The call executes and returns a string containing the new setting. @@ -104,6 +104,6 @@ The call executes and returns a string containing the new setting. "max_connections": 20000, "system_connections": 5000, "magma_compaction_rate_limit": 500, - "magma_enable_compaction_dataonly_ratelimiting": false + "magma_enable_compaction_dataonly_ratelimiting": true } ---- diff --git a/modules/rest-api/partials/rest-memory-and-storage-table.adoc b/modules/rest-api/partials/rest-memory-and-storage-table.adoc index 9f4feb7827..14e1e4b886 100644 --- a/modules/rest-api/partials/rest-memory-and-storage-table.adoc +++ b/modules/rest-api/partials/rest-memory-and-storage-table.adoc @@ -47,10 +47,10 @@ | xref:rest-api:rest-magma-compression-per-bucket.adoc[Magma compression: Per Bucket] | `GET` -| /pools/default/settings/memcached/global +| `/pools/default/settings/memcached/global` | xref:rest-api:rest-magma-compaction-rate-limiting.adoc[Magma Compaction Rate Limiting] | `POST` -| /pools/default/settings/memcached/global +| `/pools/default/settings/memcached/global` | xref:rest-api:rest-magma-compaction-rate-limiting.adoc[Magma Compaction Rate Limiting] |=== From e606a3a02ff0b78eebe5e7f33bbf9237ea030e4a Mon Sep 17 00:00:00 2001 From: Ray Offiah <77050471+RayOffiah@users.noreply.github.com> Date: Wed, 27 May 2026 12:34:46 +0100 Subject: [PATCH 4/9] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc b/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc index b909fe8ebe..d336430036 100644 --- a/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc +++ b/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc @@ -96,7 +96,7 @@ curl -u Administrator:password -X POST localhost:8091/pools/default/settings/mem -d magma_enable_compaction_dataonly_ratelimiting=true | jq ---- -The call executes and returns a string containing the new setting. +The call executes and returns a JSON object containing the new setting. [source, json] ---- From 754336d70966a3c3d8f7f5ac76b706e3769cf926 Mon Sep 17 00:00:00 2001 From: Ray Offiah <77050471+RayOffiah@users.noreply.github.com> Date: Wed, 27 May 2026 12:35:26 +0100 Subject: [PATCH 5/9] Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc b/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc index d336430036..728dd2f2e5 100644 --- a/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc +++ b/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc @@ -30,7 +30,7 @@ The only consequence of slowing these operations is increased space usage. | `magma_compaction_rate_limit` | Controls the I/O bandwidth (given in bytes per second) -that the compactions across all the shards can take to consume. +that compactions across all shards can consume. Setting this value to 0 disables compaction rate limiting. | stem:[0 … 2^64 - 1] (default: 0) From 89f14c8ad0120407ae72d3f16e4c8ce6e74adcc0 Mon Sep 17 00:00:00 2001 From: Ray Offiah Date: Wed, 27 May 2026 12:36:51 +0100 Subject: [PATCH 6/9] [DOC-14129]: Magma Compaction rate limiter documentation Changes made following co-pilot review. --- .../rest-api/pages/rest-magma-compaction-rate-limiting.adoc | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc b/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc index 728dd2f2e5..52ac23db9e 100644 --- a/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc +++ b/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc @@ -30,14 +30,14 @@ The only consequence of slowing these operations is increased space usage. | `magma_compaction_rate_limit` | Controls the I/O bandwidth (given in bytes per second) -that compactions across all shards can consume. +that compactions across all shards consume. Setting this value to 0 disables compaction rate limiting. | stem:[0 … 2^64 - 1] (default: 0) | `magma_enable_compaction_dataonly_ratelimiting` | Controls whether the compaction rate limiting runs in `data-only` mode. -(only fragment reduction compactions are rate-limited) +(Only fragment reduction compactions are rate-limited) | `true`, `false` (default:``false``) |=== From 9e8c1b29a1c4c16c568a895d69dbb2d6064a0c03 Mon Sep 17 00:00:00 2001 From: Ray Offiah Date: Mon, 1 Jun 2026 10:15:10 +0100 Subject: [PATCH 7/9] [DOC-14129]: Magma Compaction rate limiter documentation Further changes following review. Signed-off-by: Ray Offiah --- .../rest-magma-compaction-rate-limiting.adoc | 47 ++++++++----------- 1 file changed, 20 insertions(+), 27 deletions(-) diff --git a/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc b/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc index 52ac23db9e..2116449dc8 100644 --- a/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc +++ b/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc @@ -6,9 +6,8 @@ == Description -Use these API calls to set the rate limit (in bytes per second). -This setting is a global rate limit for the Memcached process shared across buckets. -The strategy for this rate limiting is to set the block waiting time on the compaction threads to maintain the specified limits. +Use these API calls to limit the disk I/O bandwidth (in bytes per second) that Magma Storage Engine Compaction may consume. +The limit is global to the Data Service and shared across all buckets on the node Magma supports 2 types of compaction: @@ -21,9 +20,26 @@ Data Log Compactions:: Data compactions can be de-prioritized compared to LSM Tree compactions. The only consequence of slowing these operations is increased space usage. +== Usage + +Rate limiting helps wherever compaction can surge and contend with front-end I/O, for example: + +* Short-burst, write-heavy workloads, where spikes in writes drive spikes in compaction. +* Large collection drops, which mark a large amount of data for removal at once. +* Sudden large number document expirations, such as a batch of items with closer TTL ranges expiring together. + +The trade-off is that compaction runs more slowly, so disk usage may stay higher for longer. +Set the limit low enough to protect front-end write bursts, but high enough that compaction keeps pace with the rate at which data is written. + + +== Recommended Value + +A practical starting point is 100 MB/s (104857600 bytes/s), or about 10× your incoming write rate — estimated as writes per second per node × average document size. +For example, 10,000 writes/s of 1 KB documents is approximately 10 MB/s incoming, giving a ~100 MB/s limit. + == Settings -[cols="3,3,1"] +[cols="1,2,1"] |=== | Setting | Description| Range @@ -35,10 +51,6 @@ Setting this value to 0 disables compaction rate limiting. | stem:[0 … 2^64 - 1] (default: 0) -| `magma_enable_compaction_dataonly_ratelimiting` -| Controls whether the compaction rate limiting runs in `data-only` mode. -(Only fragment reduction compactions are rate-limited) -| `true`, `false` (default:``false``) |=== @@ -88,22 +100,3 @@ The call executes and returns a JSON object containing the new setting. "magma_enable_compaction_dataonly_ratelimiting": false } ---- - -.Setting Compaction for data only rate limiting -[source,shell] ----- -curl -u Administrator:password -X POST localhost:8091/pools/default/settings/memcached/global \ --d magma_enable_compaction_dataonly_ratelimiting=true | jq ----- - -The call executes and returns a JSON object containing the new setting. - -[source, json] ----- -{ - "max_connections": 20000, - "system_connections": 5000, - "magma_compaction_rate_limit": 500, - "magma_enable_compaction_dataonly_ratelimiting": true -} ----- From c91495687426d00c615d8c64d11a0bae81dfb531 Mon Sep 17 00:00:00 2001 From: Ray Offiah Date: Mon, 6 Jul 2026 10:48:06 +0100 Subject: [PATCH 8/9] [DOC-14129]: Magma Compaction rate limiter documentation Further changes following review. Signed-off-by: Ray Offiah --- .../rest-magma-compaction-rate-limiting.adoc | 28 ++++++++++++------- 1 file changed, 18 insertions(+), 10 deletions(-) diff --git a/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc b/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc index 2116449dc8..45e28f356e 100644 --- a/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc +++ b/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc @@ -6,9 +6,15 @@ == Description -Use these API calls to limit the disk I/O bandwidth (in bytes per second) that Magma Storage Engine Compaction may consume. +Use these API calls to limit the disk I/O bandwidth (in *bytes per second*) that the Magma Storage Engine Compaction may consume. The limit is global to the Data Service and shared across all buckets on the node +NOTE: The limit setting is global, +even though the application of the setting is on a per-node basis. +This allows compactions on each node to consume a maximum of `compaction-rate-limit` bytes per second. + +Compaction is a background maintenance activity that reclaims space from stale and deleted data and keeps data files efficient to read. It operates on the same data files that front-end reads and writes use, so large bursts of compaction can consume enough disk I/O to slow front-end operations. Rate limiting smooths those bursts into a steady background rate, keeping front-end performance predictable — at the cost of slower compaction. + Magma supports 2 types of compaction: Log-Structured Merge (LSM) Tree Compactions:: @@ -20,22 +26,25 @@ Data Log Compactions:: Data compactions can be de-prioritized compared to LSM Tree compactions. The only consequence of slowing these operations is increased space usage. -== Usage +=== Usage Rate limiting helps wherever compaction can surge and contend with front-end I/O, for example: -* Short-burst, write-heavy workloads, where spikes in writes drive spikes in compaction. +* Short burst, write heavy workloads, where spikes in writes drive spikes in compaction. + * Large collection drops, which mark a large amount of data for removal at once. -* Sudden large number document expirations, such as a batch of items with closer TTL ranges expiring together. + +* Sudden large number document expirations, such as a batch of items with closer TTL range expiring together. The trade-off is that compaction runs more slowly, so disk usage may stay higher for longer. -Set the limit low enough to protect front-end write bursts, but high enough that compaction keeps pace with the rate at which data is written. +Set the limit low enough to protect front-end write bursts, but high enough that compaction keeps pace with the write rate. + == Recommended Value A practical starting point is 100 MB/s (104857600 bytes/s), or about 10× your incoming write rate — estimated as writes per second per node × average document size. -For example, 10,000 writes/s of 1 KB documents is approximately 10 MB/s incoming, giving a ~100 MB/s limit. +For example, 10,000 writes/s of 1 KB documents is roughly 10 MB/s incoming, giving a ~100 MB/s limit. == Settings @@ -46,7 +55,7 @@ For example, 10,000 writes/s of 1 KB documents is approximately 10 MB/s incoming | `magma_compaction_rate_limit` | Controls the I/O bandwidth (given in bytes per second) -that compactions across all shards consume. +that compactions on all the shards on one node consume. Setting this value to 0 disables compaction rate limiting. | stem:[0 … 2^64 - 1] (default: 0) @@ -78,7 +87,7 @@ Running this command returns the global settings: { "max_connections": 20000, "system_connections": 5000, - "magma_enable_compaction_dataonly_ratelimiting": false + "magma_compaction_rate_limit": 100 } ---- @@ -96,7 +105,6 @@ The call executes and returns a JSON object containing the new setting. { "max_connections": 20000, "system_connections": 5000, - "magma_compaction_rate_limit": 500, - "magma_enable_compaction_dataonly_ratelimiting": false + "magma_compaction_rate_limit": 500 } ---- From 14330e3354be821cf9d1be64811fefe9f2200241 Mon Sep 17 00:00:00 2001 From: Ray Offiah Date: Wed, 8 Jul 2026 09:48:30 +0100 Subject: [PATCH 9/9] [DOC-14129]: Magma Compaction rate limiter documentation Removed section that stated two types of compaction are supported. Signed-off-by: Ray Offiah --- .../rest-magma-compaction-rate-limiting.adoc | 16 +++------------- 1 file changed, 3 insertions(+), 13 deletions(-) diff --git a/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc b/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc index 45e28f356e..40871e2278 100644 --- a/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc +++ b/modules/rest-api/pages/rest-magma-compaction-rate-limiting.adoc @@ -13,18 +13,9 @@ NOTE: The limit setting is global, even though the application of the setting is on a per-node basis. This allows compactions on each node to consume a maximum of `compaction-rate-limit` bytes per second. -Compaction is a background maintenance activity that reclaims space from stale and deleted data and keeps data files efficient to read. It operates on the same data files that front-end reads and writes use, so large bursts of compaction can consume enough disk I/O to slow front-end operations. Rate limiting smooths those bursts into a steady background rate, keeping front-end performance predictable — at the cost of slower compaction. - -Magma supports 2 types of compaction: - -Log-Structured Merge (LSM) Tree Compactions:: -These compactions are short-running and essential for preserving the structure of the Log-Structured Merge tree. -Delaying these operations can lead to increased read latency, as the read path may need to use multiple files, resulting in higher read amplification. -If this type of compaction is lagging for level-0 of the LSM Tree, the system performs level-0 compaction within the writer threads themselves - -Data Log Compactions:: -Data compactions can be de-prioritized compared to LSM Tree compactions. -The only consequence of slowing these operations is increased space usage. +Compaction is a background maintenance activity that reclaims space from stale and deleted data and keeps data files efficient to read. +It operates on the same data files that front-end reads and writes use, so large bursts of compaction can consume enough disk I/O to slow front-end operations. +Rate limiting smooths those bursts into a steady background rate, keeping front-end performance predictable — at the cost of slower compaction. === Usage @@ -40,7 +31,6 @@ The trade-off is that compaction runs more slowly, so disk usage may stay higher Set the limit low enough to protect front-end write bursts, but high enough that compaction keeps pace with the write rate. - == Recommended Value A practical starting point is 100 MB/s (104857600 bytes/s), or about 10× your incoming write rate — estimated as writes per second per node × average document size.