Skip to content

feat(scoring): drop density, swap base score to exponential saturation#1352

Open
seroperson wants to merge 1 commit into
entrius:testfrom
seroperson:density-drop
Open

feat(scoring): drop density, swap base score to exponential saturation#1352
seroperson wants to merge 1 commit into
entrius:testfrom
seroperson:density-drop

Conversation

@seroperson
Copy link
Copy Markdown
Contributor

@seroperson seroperson commented May 23, 2026

Summary

Replaces the density-based base score with an exponential saturation curve on source token score, per the discussion and agreed shape in #1339:

base_score = MERGED_PR_BASE_SCORE * (1 - exp(-src_tok / SRC_TOK_SATURATION_SCALE))
           + min(total_score / CONTRIBUTION_SCORE_FOR_FULL_BONUS, 1) * MAX_CONTRIBUTION_BONUS

The new curve is monotonic in src_tok, concave from the origin, and ruled by one knob (SRC_TOK_SATURATION_SCALE = 58, per-repo overridable).

Per-repo override range is [10, 500], enforced by the loader and pinned by the live-config test in tests/validator/test_load_weights.py.

Requirements

# Requirement Status
1 Ship formula base_score = 25 * (1 - exp(-src_tok / 58)) + min(total_score / 1500, 1) * 5
2 Drop contribution bonus cap to 5 instead of removing the term (keep tests/non-code non-zero, 5:1 source-vs-rest ratio)
3 Land formula + bonus change in the same PR
4 Delete MAX_CODE_DENSITY_MULTIPLIER
5 Delete MIN_TOKEN_SCORE_FOR_BASE_SCORE from scoring.py (the cliff)
6 Delete MIN_TOKEN_SCORE_FOR_BASE_SCORE from credibility.py (valid_merged_count filter)
7 Eligibility becomes a pure count of merged PRs, per-repo overridable
8 Add SRC_TOK_SATURATION_SCALE = 58
9 Make SRC_TOK_SATURATION_SCALE per-repo configurable via master_repositories.json scoring block
10 MAX_CONTRIBUTION_BONUS: 25 → 5
11 Keep MERGED_PR_BASE_SCORE = 25
12 Keep CONTRIBUTION_SCORE_FOR_FULL_BONUS = 1500
13 Keep TEST_FILE_CONTRIBUTION_WEIGHT = 0.05
14 Keep the full multiplier chain untouched
15 Enforce [10, 500] range for SRC_TOK_SATURATION_SCALE via the load weights test suite

Behavior changes

New score bounds:

  • initial_base_score: 28.75 -> 25
  • Total base_score: 53.75 -> 30

Also:

  • Contribution bonus cap: MAX_CONTRIBUTION_BONUS 25 -> 5.
  • Eligibility (check_eligibility) is now a pure count of merged PRs; per-repo min_token_score_for_base_score field removed.
  • _build_solving_pr_cache no longer pre-filters by token score - every merged PR is reusable for issue-discovery lookups.
  • PullRequest.code_density / ScoredMirrorPR.code_density default flips 0.0 -> 1.0. The field is no longer populated by the scorer (density is gone), but the column survives in the DB and is read by downstream consumers as a neutral multiplier - 1.0 keeps it neutral, 0.0 would zero anything multiplying by it.

Constants

Constant Before After
MAX_CODE_DENSITY_MULTIPLIER 1.15 removed
MIN_TOKEN_SCORE_FOR_BASE_SCORE 5 removed
MAX_CONTRIBUTION_BONUS 25 5
SRC_TOK_SATURATION_SCALE n/a 58.0 (per-repo, [10, 500])
MERGED_PR_BASE_SCORE 25 25
CONTRIBUTION_SCORE_FOR_FULL_BONUS 1500 1500
TEST_FILE_CONTRIBUTION_WEIGHT 0.05 0.05

The full multiplier chain (time decay, review quality, label, issue, spam, credibility) is untouched.

Per-repo src_tok_saturation_scale

Optional knob on each master_repositories.json scoring block:

"scoring": { "src_tok_saturation_scale": 80.0 }

Loader rejects values outside [10, 500]. No entry in master_repositories.json sets the knob today; default is SRC_TOK_SATURATION_SCALE = 58.0.

Open concerns / follow-ups

  • code_density column lingers in the DB. Density is gone from scoring, but the column survives on pull_requests (written by the upsert in gittensor/validator/storage/queries.py) and the field survives on PullRequest / ScoredMirrorPR. The default flip to 1.0 keeps downstream consumers safe, but I'm unsure how DB schema removal is handled in this repo. Open question: schema-level removal as a follow-up PR, or leave the column as a permanent neutral 1.0 write?
  • Eligibility no longer requires source-code change. The saturation curve suppresses score for trivial PRs but not count toward MIN_VALID_MERGED_PRS = 3. A miner can clear the gate with three test-only / non-code merged PRs. Per direction (comment), eligibility is intentionally "a pure count of 3 merged PRs", but still worth tracking: should there be maybe an opt-in require_source_for_eligibility per-repo knob so test-only/non-code-only PRs count toward credibility but not the minimum-merged gate?

Related Issues

Closes #1339.

Type of Change

  • New feature
  • Bug fix
  • Refactor
  • Documentation
  • Other (describe below)

Testing

  • Tests added/updated
  • Manually tested using gitt miner score

Checklist

  • Code follows project style guidelines
  • Self-review completed
  • Changes are documented (gittensor-docs-ui update to follow on test)

Test plan

uv run pre-commit run --all-files
uv run pre-commit run --all-files --hook-stage pre-push

New coverage:

  • TestPerRepoSaturationScaleOverride - default scale + per-repo override reshapes the curve in both directions.
  • test_loader_rejects_out_of_range_saturation_scale + test_loader_accepts_saturation_scale_at_bounds - parametrized [10, 500] bounds.
  • test_live_mirror_scoring_fields_have_valid_shape - live master_repositories.json pinned to the same bound for every repo.

@xiao-xiao-mao xiao-xiao-mao Bot added the enhancement New feature or request label May 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Drop density from score calculation

1 participant