Restore accurate C# semantic token kinds

[Widthdom]

Summary

• add a shared C# semantic-token classifier for decoded LSP tokens, excerpts, and inspect output
• restore accurate keyword/modifier, namespace/type, method/property/event, parameter/variable/field, enum-member, and literal token kinds across source and indexed fallbacks
• bound indexed-source reconstruction and preserve useful visible-range excerpt tokens for large files
• expand cross-surface regression coverage for contextual and escaped identifiers, attributes, accessors, using declarations, explicit-interface members, record structs, interpolation, and numeric member access

Root cause and impact

The LSP and excerpt paths used separate coarse mappings, so C# identifiers were commonly flattened to variable, and source-unavailable fallbacks diverged further. The shared classifier now gives clients consistent semantic kinds while retaining bounded behavior for indexed content.

Validation

• dotnet build CodeIndex.sln -c Release --no-restore -p:UseSharedCompilation=false (0 warnings, 0 errors)
• dotnet format CodeIndex.sln --verify-no-changes --no-restore
• targeted Issue Restore accurate C# semantic token kinds in LSP and excerpt output #4852 tests on Release net8.0 and net9.0
• semantic-token / excerpt test filter on Release net8.0 and net9.0 (21 passed per TFM)
• full Release suite: net8.0 10,637 passed / 7 skipped; net9.0 10,164 passed / 419 skipped; 0 failed
• dotnet run --project tools/CodeIndex.Changelog -- check
• git diff --check
• two Codex adversarial review rounds; all findings addressed

Documentation and changelog

• updated the English and Japanese semantic-token documentation in README.md, DEVELOPER_GUIDE.md, and TESTING_GUIDE.md
• added changelog.d/unreleased/4852.fixed.md

Follow-up candidates

None.

Fixes #4852