Skip to content

Add CJK coverage per writing system - #236

Open
DraftingDreamer wants to merge 1 commit into
braver:gh-pagesfrom
DraftingDreamer:cjk-coverage
Open

Add CJK coverage per writing system#236
DraftingDreamer wants to merge 1 commit into
braver:gh-pagesfrom
DraftingDreamer:cjk-coverage

Conversation

@DraftingDreamer

Copy link
Copy Markdown

Implements the CJK coverage data discussed in #231. Adds cjk.py, which measures how much of each East Asian writing system a font covers, and wires it into info.py so the numbers land in fonts.json.

Every existing file is touched by pure addition — no line in info.py, fonts-schema.json or README.md is modified or removed.

The front end is untouched. No changes to filters.js, index.html or the styling — this only provides the data to filter on.

Two places where I did something different from your comment

You proposed storing a percentage per facet, written only above a threshold of maybe 30%. I implemented counts, always written. Both are one-line changes if you prefer your version — I'd rather show you the reasoning than quietly do it my way:

Counts instead of percentages. It matches how languages already works, so the front end reads both the same way, and fonts.json stays readable when someone reviews a PR by eye. It also survives a corrected denominator: if a tier total ever needs fixing, counts can be re-divided, percentages can't. To switch, divide by cjk.TOTALS[key] in cjk.coverage().

No write-time threshold. A threshold in info.py bakes the decision into the data, so changing 30% to 20% means re-running the script over all 189 fonts. In the front end it's a comparison against cjk_total[key], changeable any time — the same reason lang_count lives in the front end today. To switch, drop entries below the cutoff in coverage().

One thing worth knowing before you pick a threshold: a single cutoff means very different things per facet. 30% of Big5 level 2 is a reasonable "has some rarer hanzi". 30% of Hangul is useless — Korean composes syllables, so a font with 3,000 of the 11,172 fails on ordinary words. 30% of hiragana is 26 kana, which can't write anything. If you want one knob, the level-1 tiers and the syllabaries want something close to 95%, while the level-2 tiers are informative anywhere above ~20%.

Where the numbers come from

The four Han tiers are the character sets national standards define, enumerated by decoding that standard's byte ranges with the matching stdlib codec — no new dependency, no data file to maintain. Hangul and the kana are complete Unicode blocks, which for them genuinely is the practical set.

You asked to be able to trace the hardcoded ranges to an authoritative source. Rather than ask you to trust a comment, cjk_verify.py rebuilds the sets from Unicode's own data and compares: Unihan records which national standards each ideograph came from (UAX #38 kIRG_*Source). Run python3 cjk_verify.py — it downloads ~8MB from unicode.org, prints the comparison, and exits non-zero if anything differs unexpectedly. It is not part of the build; nothing on the site depends on it.

Current result against UCD 17.0:

tier vs Unihan result
GB/T 2312 G0, 6,763 identical
JIS X 0208 J0, 6,356 Unihan has one extra, 仝 (U+4EDD), which sits in JIS X 0208's symbol rows rather than the two kanji levels
KS X 1001 K0, 4,888 same count; the two pick different compatibility code points for 郎/郞 and 隸/隷
Big5 T1+T2, 13,064 13,049 in common; Unihan has no Big5 tag, so this compares against CNS 11643 planes 1–2

cjk.py also checks each tier against the count its standard specifies when it builds the sets, so if a future Python changes its codec tables the script stops with an error instead of silently shifting every percentage on the site.

Checked before opening this

  • cjk_verify.py against UCD 17.0: all four comparisons as expected, exit 0
  • cjk.py over 29 fonts from this repo; 11 have CJK data, output parses as JSON
  • info.py --name end to end on three fonts; fonts.json validates against the updated schema with no errors
  • the schema rejects a count above a tier total, an unknown tier key, a zero, and an empty cjk object
  • replacing a font with one that has no CJK removes the stale cjk entry rather than leaving it behind

Not included

fonts.json data. Running python3 info.py regenerates it, and I'd rather not put a 189-font diff in the same PR as the code — also avoids colliding with #235.

One caveat on reading the numbers

A count says whether a font can set text in a language, not whether it does so well. Han characters are shared between these standards but drawn to different regional conventions, and only the cmap is inspected, so a font can cover Big5 with glyphs drawn to Japanese conventions. Worth keeping in mind if the filter ever grows a "best for" label.

Records how much of each East Asian writing system a font covers, as a
count per tier in a new "cjk" object in fonts.json.

The Han tiers are the character sets national standards define, enumerated
by decoding their byte ranges with the matching stdlib codec, so this adds
no dependency beyond the fonttools already in requirements.txt. Hangul and
the kana are complete Unicode blocks, which for them is the practical set.

cjk_verify.py re-derives the same sets from Unicode's own Unihan data
(UAX braver#38 kIRG_*Source) and compares, so the hardcoded byte ranges can be
checked rather than taken on trust. It is a one-off tool, not part of the
build.

Refs braver#231
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant