Add CJK coverage per writing system - #236
Open
DraftingDreamer wants to merge 1 commit into
Open
Conversation
Records how much of each East Asian writing system a font covers, as a count per tier in a new "cjk" object in fonts.json. The Han tiers are the character sets national standards define, enumerated by decoding their byte ranges with the matching stdlib codec, so this adds no dependency beyond the fonttools already in requirements.txt. Hangul and the kana are complete Unicode blocks, which for them is the practical set. cjk_verify.py re-derives the same sets from Unicode's own Unihan data (UAX braver#38 kIRG_*Source) and compares, so the hardcoded byte ranges can be checked rather than taken on trust. It is a one-off tool, not part of the build. Refs braver#231
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements the CJK coverage data discussed in #231. Adds
cjk.py, which measures how much of each East Asian writing system a font covers, and wires it intoinfo.pyso the numbers land infonts.json.Every existing file is touched by pure addition — no line in
info.py,fonts-schema.jsonorREADME.mdis modified or removed.The front end is untouched. No changes to
filters.js,index.htmlor the styling — this only provides the data to filter on.Two places where I did something different from your comment
You proposed storing a percentage per facet, written only above a threshold of maybe 30%. I implemented counts, always written. Both are one-line changes if you prefer your version — I'd rather show you the reasoning than quietly do it my way:
Counts instead of percentages. It matches how
languagesalready works, so the front end reads both the same way, andfonts.jsonstays readable when someone reviews a PR by eye. It also survives a corrected denominator: if a tier total ever needs fixing, counts can be re-divided, percentages can't. To switch, divide bycjk.TOTALS[key]incjk.coverage().No write-time threshold. A threshold in
info.pybakes the decision into the data, so changing 30% to 20% means re-running the script over all 189 fonts. In the front end it's a comparison againstcjk_total[key], changeable any time — the same reasonlang_countlives in the front end today. To switch, drop entries below the cutoff incoverage().One thing worth knowing before you pick a threshold: a single cutoff means very different things per facet. 30% of Big5 level 2 is a reasonable "has some rarer hanzi". 30% of Hangul is useless — Korean composes syllables, so a font with 3,000 of the 11,172 fails on ordinary words. 30% of hiragana is 26 kana, which can't write anything. If you want one knob, the level-1 tiers and the syllabaries want something close to 95%, while the level-2 tiers are informative anywhere above ~20%.
Where the numbers come from
The four Han tiers are the character sets national standards define, enumerated by decoding that standard's byte ranges with the matching stdlib codec — no new dependency, no data file to maintain. Hangul and the kana are complete Unicode blocks, which for them genuinely is the practical set.
You asked to be able to trace the hardcoded ranges to an authoritative source. Rather than ask you to trust a comment,
cjk_verify.pyrebuilds the sets from Unicode's own data and compares: Unihan records which national standards each ideograph came from (UAX #38kIRG_*Source). Runpython3 cjk_verify.py— it downloads ~8MB from unicode.org, prints the comparison, and exits non-zero if anything differs unexpectedly. It is not part of the build; nothing on the site depends on it.Current result against UCD 17.0:
G0, 6,763J0, 6,356K0, 4,888T1+T2, 13,064cjk.pyalso checks each tier against the count its standard specifies when it builds the sets, so if a future Python changes its codec tables the script stops with an error instead of silently shifting every percentage on the site.Checked before opening this
cjk_verify.pyagainst UCD 17.0: all four comparisons as expected, exit 0cjk.pyover 29 fonts from this repo; 11 have CJK data, output parses as JSONinfo.py --nameend to end on three fonts;fonts.jsonvalidates against the updated schema with no errorscjkobjectcjkentry rather than leaving it behindNot included
fonts.jsondata. Runningpython3 info.pyregenerates it, and I'd rather not put a 189-font diff in the same PR as the code — also avoids colliding with #235.One caveat on reading the numbers
A count says whether a font can set text in a language, not whether it does so well. Han characters are shared between these standards but drawn to different regional conventions, and only the
cmapis inspected, so a font can cover Big5 with glyphs drawn to Japanese conventions. Worth keeping in mind if the filter ever grows a "best for" label.