fix(keys): reclaim expired shared-dict entries in remove_expired_keys()

[AlinsRan]

Problem

prometheus-metrics free_space drops in steps and never recovers when metrics are registered with an expire (apache/apisix#13658). A reporter's dump showed 747970 of 751496 keys were stale __ngx_prom__key_N index slots against 28 live ones, 388MB of a 512MB dict, with very few active series.

KeyIndex:remove_expired_keys() only cleared the worker-local keys / index / expire_keys tables. It issued no writes to the shared dict, so the expired entries themselves — both the index slots and the metric value keys, which share the dict — stayed physically allocated.

Nothing else reclaimed them:

• Index slots are never reused. Once a metric's slot has expired, add() takes the not-found branch and allocates a fresh slot at key_count+1. key_count is monotonic, so under label churn with an exptime the slot count grows without bound.
• The passive expiry scan cannot reach them. shdict checks only the LRU tail on each write and stops at the first non-expired entry. A permanent entry — the error metric, or any metric registered without an exptime — inevitably goes cold and ends up sitting there, blocking the scan for good.

Reproduced standalone against 0.20250302 and 0.20260623 with identical output: 1000 active series held constant, free_space dropping one step per burst and never recovering, and a single explicit flush_expired(0) at the end reclaiming 7000 entries and restoring free_space to its initial value. A separate experiment confirmed the LRU-tail blocking: with a permanent entry at the tail, 1000 writes reclaimed nothing; without it, everything was reclaimed.

This is also what feeds #12275 — the unbounded growth is what fills the dict in the first place, which is the precondition for the spin fixed in #16.

Fix

Call self.dict:flush_expired() once per remove_expired_keys() timer round.

Expired entries are indistinguishable from absent ones through every shared-dict API (ngx_http_lua_shdict_lookup() returns NGX_DONE for them, and every caller treats that as a miss), so reclaiming them cannot change any observable behaviour — /metrics output, metric values and expire semantics are unchanged. Only free_space and eviction pressure change.

Live series are not at risk: an active metric's index slot TTL is refreshed on every operation via the lookup_or_create() cache-hit path, which calls key_index:add() with the exptime whenever exptime > 0, and counter value TTLs are refreshed by the counter sync each round. Only series that genuinely saw no writes for the whole expire window can be flushed, which is exactly the semantics of the feature.

Cost is a locked linear scan of the dict per worker per timer round (3600s under the APISIX wiring). At steady state that is microseconds to milliseconds. An instance that has already accumulated the backlog pays one larger scan on the first round after upgrade and then converges — no manual flush or restart needed.

Tests

Three regression tests, all of which fail without the one-line change and pass with it:

• TestKeyIndex:testRemoveExpiredKeysReclaimsSharedDict — an expired index slot and an expired value key are both physically gone after a round; a non-expired entry survives.
• TestKeyIndex:testExpiredSlotsDoNotAccumulate — repeated expire/re-add churn leaves at most one index slot in the dict.
• TestPrometheus:testExpiredMetricIsReclaimedFromDict — same, driven through the public counter API.

Two supporting changes to the test harness:

• SimpleDict:flush_expired() added, matching the real semantics (scans everything, removes only expired entries, returns the count; the optional argument caps removals, not scans).
• SimpleDict:ttl() no longer physically prunes the entry it reads. The real ttl reports an expired entry as not found but leaves it allocated; the mock's pruning would have made the reclamation assertions vacuous, since remove_expired_keys() calls ttl on exactly the slots under test.

The assertions read dict.dict directly rather than going through get(), since get() prunes as well — physical presence is the whole point here and cannot be observed through the dict API by design.

CI

prometheus_test.lua was never executed by CI, only linted — luaunit was installed and unused, so the tests added in #14 and #16 have never actually gated anything. Added a step to run it, which is what makes the above regression tests meaningful.

Notes

• Local run under LuaJIT: 44/45 pass. The one failure is TestPrometheus.testPrintfTable, which asserts # on a table with holes ({nil,2,nil,"foo",nil}) — undefined behaviour that differs between 5.1 and 5.2. It reproduces identically on unmodified main and passes under the Lua 5.2 the CI matrix uses. Unrelated to this change.
• No if self.dict.flush_expired guard: both consumers (apache/apisix and the API7 EE gateway) pass a real ngx.shared dict, and the test mock now implements it.
• This should ship in the same release as fix(keys): prevent infinite loop in KeyIndex:add() when key_count falls behind occupied slots #16 — they are the same failure chain, and fix(prometheus): bump nginx-lua-prometheus-api7 to 0.20260716 to fix 100% CPU loop on full shared dict apache/apisix#13708 is still open waiting on a bump, so both can land in one version bump instead of two.

[coderabbitai]

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 3 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 9555cc1f-9218-4367-8b51-e5cc75a3c6b3

📥 Commits

Reviewing files that changed from the base of the PR and between 613577e and 4bf2ede.

📒 Files selected for processing (3)

• .github/workflows/tests.yml
• prometheus_keys.lua
• prometheus_test.lua

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/flush-expired-shm-reclaim

---

_{Comment @coderabbitai help to get the list of available commands.}