Make EKS upgrade scoring deterministic across runs#2
Open
kahhaw9368 wants to merge 1 commit into
Open
Conversation
The same cluster scored 72% and 88% on two runs because three scoring decisions were left to model judgment rather than pinned to rules: 1. Deprecated v1beta3 APF objects: steering both said "count them" and "they're harmless system objects." Added deterministic Step 3b two-test filter — a FlowSchema/PriorityLevelConfiguration counts only if its stored (live) apiVersion is a removed version OR a user tool (kubectl/helm/argocd/flux) wrote the removed version. Objects already stored on v1 whose only v1beta3 trace comes from internal APF controllers (api-priority-and-fairness-config-*, eks-internal) are false positives and do not count. Verified against the live cluster: eks-exempt / eks-leader-election / eks-workload-high are stored on v1 with only internal-controller v1beta3 history -> correctly excluded. Referenced from report-generation Category 2 and breaking-changes. 2. Add-on compatibility: "compatible" was assignable without checking. Made `describe-addon-versions` comparison mandatory with a bright-line verdict table — no COMPATIBLE without running the check. 3. Anonymous-auth restriction fit two categories (Breaking vs Behavioral). Pinned its single scoring home to Breaking Changes; added do-not-double- count guards in both files. Also added a score-reconciliation hard gate: headline score must equal 100 minus the Master Finding List sum, and every "always flag" item must appear as a row. Catches header/table drift automatically. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
The same cluster (
test-old-cluster, 1.31→1.32) scored 72% on one run and 88% on another with no change to the cluster. The scoring arithmetic is deterministic, but the finding list feeding it was built from model judgment in three spots, so two runs legitimately diverged.Root causes & fixes
1. Deprecated
v1beta3APF objects (10-pt swing). The steering both said "count them" and "they're harmless system objects." Replaced with a deterministic two-test filter indeprecated-apis.mdStep 3b:...k8s.io/v1, exclude it — av1beta3string inmanagedFieldsis stale edit-history, not live config.v1beta3is an internal APF controller (api-priority-and-fairness-config-*,eks-internal), exclude it. Count only when a user tool (kubectl/helm/argocd/flux) wrote the removed version.Verified live:
eks-exempt/eks-leader-election/eks-workload-highare stored onv1with only internal-controller history → correctly excluded (0 pts).2. Add-on compatibility (4-pt swing). "COMPATIBLE" was assignable without checking. Made
aws eks describe-addon-versionscomparison mandatory inaddon-compatibility.mdwith a bright-line verdict table — no COMPATIBLE without running the check.3. Anonymous-auth restriction (2-pt swing). The finding fit both Breaking Changes and Behavioral Changes. Pinned its single scoring home to Breaking Changes (Category 1, 4 pts); added do-not-double-count guards in both files.
4. Safety net. Added a score-reconciliation hard gate in
report-generation.md: headline score must equal 100 minus the Master Finding List sum, and every "always flag" item must appear as a row. Catches header/table drift automatically.Result
The cluster now scores a stable 82% on every run — the two prior runs were over-penalized (72, counted the flowcontrol false positives) and under-penalized (88, also dropped the real add-on and anon-auth findings) respectively. The corrected run keeps the genuine deductions and drops only the false positives.
Files changed
.claude/skills/eks-upgrade/steering/deprecated-apis.md.claude/skills/eks-upgrade/steering/breaking-changes.md.claude/skills/eks-upgrade/steering/addon-compatibility.md.claude/skills/eks-upgrade/steering/report-generation.md🤖 Generated with Claude Code