Skip to content

docs: audit and correct attributes.md against actual assessor code#408

Draft
jwm4 wants to merge 3 commits intomainfrom
docs/attributes-accuracy-audit
Draft

docs: audit and correct attributes.md against actual assessor code#408
jwm4 wants to merge 3 commits intomainfrom
docs/attributes-accuracy-audit

Conversation

@jwm4
Copy link
Copy Markdown
Contributor

@jwm4 jwm4 commented May 8, 2026

Summary

Full accuracy audit of docs/attributes.md found widespread drift between the documentation and the actual assessor implementations. This PR corrects everything found.

Weight corrections

Pre-v2.0.0 weights that were never updated in the doc:

Attribute Doc said Actual
readme_structure 10% 5%
type_annotations 10% 8%
standard_layout 10% 5%
lock_files 10% 5%
cyclomatic_complexity 3% 2%
structured_logging 3% 2%
issue_pr_templates Tier 4 / 1% Tier 3 / 1.5%

Criteria corrections

The doc described checks the assessors do not actually perform:

  • claude_md_file: doc claimed 6 required content sections; code only checks the file exists at ≥50 bytes. Added AGENTS.md as a fully equivalent alternative (scores 100).
  • readme_structure: doc listed 8 required sections; code checks 3 (installation, usage, development), pass ≥75.
  • test_execution: doc described 70/80/90% coverage thresholds; assessor does not measure coverage at all — it checks whether test config files are present.
  • lock_files: added freshness check (15pt deduction if >6 months old) and requirements.txt pin quality validation; removed gitignore claim that is not checked.
  • deterministic_enforcement: doc required CI enforcement; assessor only checks .pre-commit-config.yaml (60pts), .claude/settings.json hooks (30pts), .husky (10pts). Pass on pre-commit config alone.
  • gitignore_completeness: replaced category-based description with actual pattern-based mechanism (70% of language-specific patterns from hardcoded list).
  • container_setup: added Containerfile as alternative, .dockerignore scoring, and conditionality (not_applicable if no container file found).

New content

Full write-ups added for three Tier 1 attributes that previously had only one-liners:

  • ci_quality_gates (5%)
  • single_file_verification (5%)
  • dependency_security (5%)

New full entry for repomix_config (Tier 3, 2%) — recently registered assessor, not yet documented.

Other fixes

  • separation_concernsseparation_of_concerns (attribute ID was wrong throughout)
  • Tier 3 attribute count: 5 → 6
  • Implementation status table updated: all tiers are fully implemented; removed stale "active development / planned" labels

Assessor code fixes

default_weight values in four assessors were out of sync with the yaml (yaml is authoritative):

File Field Was Now
stub_assessors.py DependencyPinningAssessor 0.10 0.05
security.py DependencySecurityAssessor 0.04 0.05
code_quality.py CyclomaticComplexityAssessor 0.03 0.02
code_quality.py StructuredLoggingAssessor 0.015 0.02

Test plan

  • 1152 passed, 21 skipped — no regressions
  • black . && isort . && ruff check . — clean

🤖 Generated with Claude Code under the supervision of Bill Murdock

Full accuracy audit of docs/attributes.md found weights, criteria, and
structural information that had drifted from the implementation. All
discrepancies corrected.

Weight corrections (pre-v2.0.0 values updated):
- readme_structure: 10% -> 5%
- type_annotations: 10% -> 8%
- standard_layout: 10% -> 5%
- lock_files: 10% -> 5%
- cyclomatic_complexity: 3% -> 2%
- structured_logging: 3% -> 2%
- issue_pr_templates: Tier 4 / 1% -> Tier 3 / 1.5%

Criteria corrections (doc described checks the code does not perform):
- claude_md_file: remove 6-section requirement; actual check is file
  exists at >=50 bytes; add AGENTS.md as fully equivalent alternative
- readme_structure: replace 8-section list with actual 3 sections
  checked (installation, usage, development); pass >=75
- test_execution: remove 70/80/90% coverage thresholds; assessor checks
  config presence only (test files, runner, coverage config, enforcement)
- lock_files: add freshness check and requirements.txt pin quality;
  remove gitignore claim that is not checked
- deterministic_enforcement: remove CI enforcement requirement; actual
  scoring is .pre-commit-config.yaml (60pts), .claude/settings.json
  hooks (30pts), .husky (10pts); pass >=60
- gitignore_completeness: replace category-based with pattern-based
  description; pass >=70% of language-specific patterns
- container_setup: add Containerfile, .dockerignore, conditionality

New full write-ups added for attributes previously listed as one-liners:
- ci_quality_gates (Tier 1, 5%)
- single_file_verification (Tier 1, 5%)
- dependency_security (Tier 1, 5%)

New entry added: repomix_config (Tier 3, 2%)

Other fixes:
- separation_concerns -> separation_of_concerns (attribute ID corrected)
- Tier 3 count: 5 -> 6 attributes
- Implementation status table: all tiers now show fully implemented

Assessor default_weight values synced to yaml (yaml is authoritative):
- DependencyPinningAssessor: 0.10 -> 0.05
- DependencySecurityAssessor: 0.04 -> 0.05
- CyclomaticComplexityAssessor: 0.03 -> 0.02
- StructuredLoggingAssessor: 0.015 -> 0.02

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@coderabbitai
Copy link
Copy Markdown

coderabbitai Bot commented May 8, 2026

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: ccb37d3d-dd5e-494c-9d05-0578bb15b4c7

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Warning

.coderabbit.yaml has a parsing error

The CodeRabbit configuration file in this repository has a parsing error and default settings were used instead. Please fix the error(s) in the configuration file. You can initialize chat with CodeRabbit to get help with the configuration file.

💥 Parsing errors (1)
Validation error: String must contain at most 250 character(s) at "tone_instructions"
⚙️ Configuration instructions
  • Please see the configuration documentation for more information.
  • You can also validate your configuration using the online YAML validator.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch docs/attributes-accuracy-audit

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 8, 2026

📈 Test Coverage Report

Branch Coverage
This PR 72.5%
Main 72.5%
Diff ✅ +0%

Coverage calculated from unit tests only

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@jwm4 jwm4 marked this pull request as draft May 8, 2026 16:32
Replace vague research claims, padded impact bullet lists, and
inline bold-header bullets with direct, specific prose.

- Remove unsourced "~40% prompt engineering reduction" claim
- Replace "Impact on AI Agents" bullet lists with single focused paragraphs
- Remove overview's inline bold-header format (Research-backed/Measurable/etc.)
- Replace vague "Research shows" with specific mechanism explanations
- Remove significance inflation phrases
- Fix one em dash

All scoring criteria, thresholds, commands, and citations unchanged.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant