control-plane: warn when the scheduler is stuck reapplying a non-converging plan#6585
control-plane: warn when the scheduler is stuck reapplying a non-converging plan#6585aanufriev-celonis wants to merge 3 commits into
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
https://github.com/quickwit-oss/quickwit/blob/f2848637b69a5c1e29c9fc7d560c44a184508b96/quickwit-control-plane/src/indexing_scheduler/mod.rs#L400-L401
This reset only covers the control-loop path where node IDs differ, but normal model changes also schedule a fresh plan via ControlPlane::handle(RebuildPlan) calling rebuild_plan directly (control_plane.rs:453, and similarly rebalance at :1081). After a prior stuck plan has crossed the threshold, applying a new plan through those paths leaves num_consecutive_reapplies high, so the first normal post-reschedule mismatch is logged as a repeated non-convergence warning even though it is the first reapply for a different plan. Reset the counter when a new physical plan is actually applied/scheduled rather than only in this branch.
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
@codex review |
|
Codex Review: Didn't find any major issues. Keep them coming! Reviewed commit: ℹ️ About Codex in GitHubCodex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback". |
1 participant
Summary
Make the indexing scheduler's "reapply last plan" control loop observable so a non-converging cluster can be detected and alerted on, instead of failing silently for an unbounded amount of time.
Part of #6601.
Problem
The control plane runs a periodic control loop (
control_running_plan) that compares the running plan reported by indexers against the last applied plan. When the tasks differ (same nodes), the loop re-sends the identical plan.If the last applied plan can never converge, for example, an indexer that never picks up its assigned tasks, or a stale/un-runnable plan referencing a shard that no longer exists, the loop reapplies the same plan every cycle indefinitely. Ingestion for the affected shards stalls, and currently this is invisible: the loop logs the same
infoline every ~30s with no counter, so a transient one-cycle propagation lag is indistinguishable from a cluster that has been stuck for hours, and there is nothing to alert on.Change
Add
num_consecutive_reappliestoIndexingSchedulerState. It increments on each reapply and resets to 0 on convergence or when a new plan is scheduled.Once it reaches
REAPPLY_LOOP_WARN_THRESHOLD(5 cycles, ~2.5 min at the 30s production interval), the per-cycle log escalates frominfotowarn, including the plan diff. The missing-vs-unplanned breakdown in the diff distinguishes an un-runnable task from a stale pipeline.This is purely observability, no scheduling behavior changes.
Testing
New unit test
test_control_running_plan_counts_consecutive_reappliesdrives the reapply branch repeatedly, asserts the counter grows, then converges the indexer and asserts it resets to 0.cargo test -p quickwit-control-planepasses.cargo +nightly fmtandcargo clippy -p quickwit-control-plane --all-features --testsare clean.