OPENNLP-1894: Add dictionary-based tokenization for Japanese, Korean, and Chinese - #1191
Open
krickert wants to merge 14 commits into
Open
OPENNLP-1894: Add dictionary-based tokenization for Japanese, Korean, and Chinese#1191krickert wants to merge 14 commits into
krickert wants to merge 14 commits into
Conversation
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Jul 21, 2026
krickert
marked this pull request as ready for review
July 24, 2026 18:57
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Jul 24, 2026
…ENNLP-1895 recorded Restate the map against apache main a864230, cut as 3.0.0-M5 on 2026-07-24. apache#1177 (OPENNLP-1870) merged upstream and moves into the merged box, apache#1190 and apache#1191 are marked ready for review, and OPENNLP-1895 (quantized embedding tables) joins the diagram in its own colour: filed in JIRA with the pull request deliberately held until apache#1165 and apache#1152 move. Statuses now carry the measured GitHub draft flag and how far each head sits behind main, which surfaces three things the old text did not: apache#1182 is a draft again, apache#1167 is based on main rather than on apache#1155 and carries the seam and isBlank commits as copies, and apache#1152 reports conflicts only because its apache-hosted sentencepiece base has diverged from the refreshed head.
krickert
force-pushed
the
OPENNLP-1894-lattice-cjk
branch
from
July 24, 2026 19:27
65579e9 to
db5f6cc
Compare
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Jul 27, 2026
…preview-docs, record OPENNLP-1897 The 2026-07-24 map refresh (PR-head rebase record, apache#1190/apache#1191 ready, morfologik-fsa, OPENNLP-1895) was committed directly on kristian-3.x-features and would have been discarded by the next regeneration; preview-docs is the durable home. Also adds OPENNLP-1897-term-vectors to the held-PR section and diagram, and moves the state line to 2026-07-26 (apache main unchanged since the M5 cut).
…ctionaries A Viterbi decoder over word and connection costs segments languages written without spaces; the same engine serves Japanese and Korean because the language lives entirely in the dictionary. Unknown text is handled through the dictionary's character categories, and every span stays in original text coordinates. An installer fetches and unpacks a user-chosen dictionary archive at install time: nothing is bundled, no location is built in, and entry names are flattened so no archive path escapes the target directory. (cherry picked from commit a699c8a)
Common-prefix lookup walks a trie built at load time instead of probing substrings per length, terminating on the first missing prefix and allocating nothing per position. (cherry picked from commit e10ce4b)
A Viterbi search maximizing summed word log-probabilities segments Chinese and similar scripts from a plain word-count lexicon, with unlisted characters falling back to single-character words. The user supplies the lexicon and thereby accepts its license; nothing is bundled. (cherry picked from commit bff3f23)
…nigram segmenters, corrected javadoc
…s, add an EUC-JP loading example
…ng download helper
…e their category run, and validate context ids at load
…expressible dictionary values The lattice tokenizer rescanned the same-category run from every position, so a run of L characters cost on the order of L squared category lookups; a 16,000-character katakana run measured around half a second. One right-to-left pass per stretch now fixes every position's category and run end, and the same 16,000-character run tokenizes in about half a millisecond at 31 million characters per second. The character table holds Category instances instead of names, so the per-character path compares by identity with no name-map lookup, and a char.def mapping to an undefined category now fails at load naming the code point. The lexicon trie's children are sorted character arrays found by binary search, so a descent no longer boxes a Character per step. matrix.def loading rejects connection costs outside the 16-bit range instead of silently truncating them, and dimension products beyond the addressable array size fail at the header. The unigram segmenter's unknown-character fallback advances one code point, never one code unit, so an unknown supplementary character is stepped over whole and no span can split its surrogate halves.
…recoded labels The lexicon trie's per-node child lookup, a binary search over the node's fan-out, paid about a dozen comparisons at the root of a real dictionary; the classic base/check double-array makes every transition one array read and one comparison. Characters are recoded into dense labels ordered by descending frequency before the array is built, so the array stays compact although CJK surfaces draw on tens of thousands of distinct characters, and a character the lexicon never uses misses in the recode table before the array is consulted. On the IPADIC harness the prefix walk now matches the fastest previous implementation at 5.6M chars/s with strictly constant-time transitions, and building the array adds about a quarter second to the 392k-entry load.
…er-position lists The Viterbi lattice held one ArrayList per text position plus one fresh candidate list per position, pure allocation churn on long stretches. Nodes ending at a position now chain through their own link field behind a single head reference per position, and candidate gathering fills one scratch list reused across positions, so building the lattice allocates nothing besides the nodes themselves. IPADIC throughput on the 400k-character harness rises from 5.6M to 6.5M characters per second with identical output.
…example Add a lattice tokenizer section to the manual citing LatticeUsageExampleTest.
…lexicon accessors
…e tokenizer overrides
The frequency lexicon was trimmed with String.trim(), which strips only ASCII
control characters and the space. A line starting with an ideographic space
(U+3000), ordinary in hand-edited CJK text files, therefore kept that space as
part of the word and pushed the count field one token to the right, so the load
failed as a malformed count. The lexicon reader now trims with
StringUtil.trimUnicodeWhitespace, matching the White_Space convention the rest
of the tokenizer already scans by, and a test pins the leading U+3000 case.
The mecab reader's line and numeric-field trims move to the same call so one
class does not mix two whitespace judgments; those fields are ASCII in valid
dictionaries, so the behavior there is unchanged.
Both tokenizer views also gain {@inheritdoc} and their null contract, and the
unknown-candidate helper drops a static modifier it did not need.
…fold fixture duplication - Document the private lattice helpers decode, relax, and candidates, and the installer's boundedStream, with the parameter, return, and exception contracts the review expects every method to carry. - Document the WordEntry and Category record components and the double-array builder's findBase and ensureCapacity helpers. - Record on analyze, tokenize, and tokenizePos that a unk.def without a DEFAULT template leaves the lattice disconnected and makes them throw IllegalStateException. - State on readLines that it never returns an empty list, which is what lets the matrix.def header be read before the emptiness check. - Rename the Tokenizer override parameter from s to text in LatticeTokenizer and UnigramSegmenter, so the javadoc names a parameter that exists. - Hoist the matrix.def, char.def, and unk.def file names, the DEFAULT category name, the 0x code point prefix, the .. range separator, and the flag value into named constants in MecabDictionary, and let LatticeTokenizer reach the DEFAULT name through MecabDictionary instead of repeating the literal. - Name the tar block size, header field offsets, and field lengths in the TarGzArchives test helper instead of writing 512, 124, and 148 inline. - Replace the boolean[1] capture in candidates with a check that the candidate list is still empty, which is the same signal without the array. - Track the best boundary total in decode instead of recomputing the incumbent's connection cost on every comparison. - Drop the categories map field from MecabDictionary, which nothing read once the constructor resolved the DEFAULT category out of it. - Match the char.def code point prefix once, case insensitively, rather than testing 0x and 0X separately, and cut the range at the separator's own length. - Trim the matrix.def header before parsing it and report an empty first line as an empty matrix.def, since readLines never yields the empty list the previous check was looking for. - Split the omnibus malformed-dictionary test into named cases that pin the messages for a missing definition file, a char.def without DEFAULT, a lexicon with no entries, and an empty matrix.def. - Parameterize the malformed char.def cases and the malformed unigram lexicon cases, which were repeated assertThrows calls over one fixture shape. - Add a Morpheme test pinning the null and empty argument rejections and the defensive copy of the feature list. - Extend the invalid-argument tests to the entry points that were uncovered: MecabDictionary.load with a null directory or charset, the installer's null target, UnigramSegmenter's path and stream overloads, and both tokenize methods of each tokenizer. - Fold the repeated Files.write fixture calls into one write helper and hoist the shared lexicon, matrix, char.def, and unk.def fixture text into constants. - Correct dev/README-mecab-dictionaries.md to say that dicrc is the configuration file the distributions ship alongside the csv and def files a MecabDictionary reads, rather than implying the dictionary reads dicrc itself.
krickert
force-pushed
the
OPENNLP-1894-lattice-cjk
branch
from
July 28, 2026 15:15
b02948c to
6716542
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Viterbi lattice segmentation over user-supplied MeCab-format dictionaries, covering the Japanese (IPADIC) and Korean (mecab-ko-dic) dictionary layouts, plus a frequency-driven unigram segmenter for Chinese.
The lexicon is held in a double-array trie with frequency-recoded labels, and lattice nodes chain intrusively so the hot path allocates no per-position lists. Measured on the real IPADIC dictionary: about 5 million characters per second single-threaded, with 392k entries loading in under one second. Segmentation of the standard connection-cost-sensitive test sentences matches the reference implementation's output on IPADIC.
Dictionaries are always user-supplied and never bundled; test fixtures are project-authored miniature lexicons written by the tests themselves, and
dev/README-mecab-dictionaries.mdplus a checksum-verifying download script document dictionary acquisition. The tokenizer manual gains a section whose example is asserted byLatticeUsageExampleTest.