diff --git a/README.md b/README.md index d5bfa01..278ae72 100644 --- a/README.md +++ b/README.md @@ -6,3 +6,12 @@ libime [![Documentation](https://codedocs.xyz/fcitx/libime.svg)](https://codedocs.xyz/fcitx/libime/) This is a library to support generic input method implementation. + +Documentation: + +- [Pinyin dictionary text format](src/libime/pinyin/pinyindictionary-text-format.md) +- [拼音词典文本格式](src/libime/pinyin/pinyindictionary-text-format_zh.md) +- [Shuangpin profile text format](src/libime/pinyin/shuangpin-profile-format.md) +- [双拼方案文本格式](src/libime/pinyin/shuangpin-profile-format_zh.md) +- [Table text format](src/libime/table/table-text-format.md) +- [码表文本格式](src/libime/table/table-text-format_zh.md) diff --git a/src/libime/pinyin/pinyindictionary-text-format.md b/src/libime/pinyin/pinyindictionary-text-format.md new file mode 100644 index 0000000..358d0eb --- /dev/null +++ b/src/libime/pinyin/pinyindictionary-text-format.md @@ -0,0 +1,158 @@ +# Pinyin dictionary text format + +This document describes the plain-text format used by `PinyinDictionary`. + +For in-repository examples, see `test/testpinyindictionary.cpp` and the +generated dictionary source used by tests such as `data/dict_sc.txt`. + +## Overview + +The format is line-based. Each dictionary entry is written on one line in one +of these forms: + +```text + + +``` + +Examples: + +```text +倪辉 ni'hui 0.0 +小企鹅 xiao'qi'e +X光 X'guang +``` + +## Fields + +### 1. Hanzi field + +The first field is the word or phrase text. + +Examples: + +```text +你 +小企鹅 +X光 +``` + +This field may be quoted when it contains whitespace, quotes, backslashes, or +escaped newlines. + +Example: + +```text +"你\n好 不\"" ni +``` + +That entry represents the text: + +```text +你 +好 不" +``` + +### 2. Full pinyin field + +The second field is the full pinyin spelling. + +Examples: + +```text +ni +ni'hui +xiao'qi'e +X'guang +``` + +Important details: + +- syllables are separated with `'` +- upper-case letters are allowed when the spelling needs them, for example + `X'guang` +- the pinyin must be a valid full pinyin spelling accepted by libime + +### 3. Weight field + +The third field is optional and is parsed as a floating-point number. + +Examples: + +```text +0 +0.0 +1.25 +-3 +``` + +If the weight is omitted, the entry is loaded with `0.0`. + +When the dictionary is saved back to text, every entry is written with an +explicit weight field, so an entry that was originally: + +```text +小企鹅 xiao'qi'e +``` + +will be written back as: + +```text +小企鹅 xiao'qi'e 0 +``` + +## Escaping and quoting + +Fields are tokenized with support for quoted and escaped values. + +Use quotes when a field contains whitespace or special characters: + +```text +"你\n好 不\"" ni +``` + +Supported escaped forms are the usual backslash-escaped forms accepted by the +value parser, including: + +- `\"` for a literal quote +- `\\` for a literal backslash +- `\n` for an embedded newline + +Each entry still occupies one physical line in the file. If you need embedded +newlines inside the text field, represent them with escapes such as `\n`. + +## Line behavior + +- Blank lines are ignored. +- Lines with anything other than 2 or 3 parsed fields are ignored. +- There is no section syntax in this format. +- Comment syntax is not part of the format. A line beginning with `#` is simply + treated as an invalid entry and ignored unless it happens to parse as a valid + 2-field or 3-field record. + +## Validation behavior + +An entry is skipped if: + +- the pinyin field is invalid +- the optional weight field is present but is not a valid floating-point number + +The loader continues reading later lines after such errors. + +## Saved text format + +When written back as text, each entry is saved in this form: + +```text + +``` + +Examples: + +```text +X光 X'guang 0 +"你\\n好 不\\\"" ni 0 +倪辉 ni'hui 0 +``` + +The hanzi field is automatically escaped when needed. diff --git a/src/libime/pinyin/pinyindictionary-text-format_zh.md b/src/libime/pinyin/pinyindictionary-text-format_zh.md new file mode 100644 index 0000000..70902d2 --- /dev/null +++ b/src/libime/pinyin/pinyindictionary-text-format_zh.md @@ -0,0 +1,152 @@ +# 拼音词典文本格式 + +本文档说明 `PinyinDictionary` 使用的纯文本格式。 + +仓库内的示例可参考 `test/testpinyindictionary.cpp`,以及测试所使用的生成词典 +源文件,例如 `data/dict_sc.txt`。 + +## 概览 + +该格式按行组织。每个词典条目占一行,支持以下两种形式: + +```text + + +``` + +例如: + +```text +倪辉 ni'hui 0.0 +小企鹅 xiao'qi'e +X光 X'guang +``` + +## 字段说明 + +### 1. 汉字字段 + +第一个字段是词或短语文本。 + +例如: + +```text +你 +小企鹅 +X光 +``` + +当该字段中包含空白、引号、反斜杠或转义换行时,可以使用引号包裹。 + +例如: + +```text +"你\n好 不\"" ni +``` + +这个条目表示的文本是: + +```text +你 +好 不" +``` + +### 2. 全拼字段 + +第二个字段是完整拼音。 + +例如: + +```text +ni +ni'hui +xiao'qi'e +X'guang +``` + +需要注意: + +- 音节之间使用 `'` 分隔 +- 在需要时允许使用大写字母,例如 `X'guang` +- 该拼音必须是 libime 接受的合法全拼形式 + +### 3. 权重字段 + +第三个字段是可选的,按浮点数解析。 + +例如: + +```text +0 +0.0 +1.25 +-3 +``` + +如果省略该字段,则按 `0.0` 载入。 + +当词典再保存回文本格式时,每个条目都会显式写出权重字段。因此原本写成: + +```text +小企鹅 xiao'qi'e +``` + +保存后会变成: + +```text +小企鹅 xiao'qi'e 0 +``` + +## 转义与引号 + +字段在分词时支持带引号和带转义的值。 + +当字段中包含空白或特殊字符时,请使用引号: + +```text +"你\n好 不\"" ni +``` + +支持的转义形式包括常见的反斜杠转义,例如: + +- `\"` 表示字面意义上的双引号 +- `\\` 表示字面意义上的反斜杠 +- `\n` 表示嵌入式换行 + +每个条目仍然只占用文件中的一个物理行。如果需要在文本字段内部包含换行, +应使用 `\n` 这样的转义形式表示。 + +## 行行为 + +- 空行会被忽略。 +- 解析后字段数不是 2 个或 3 个的行会被忽略。 +- 该格式没有分节语法。 +- 注释不是该格式的一部分。以 `#` 开头的行不会被当作注释处理;除非它恰好 + 不能解析为合法的 2 字段或 3 字段记录,否则它只是普通输入。 + +## 校验行为 + +在以下情况下,条目会被跳过: + +- 拼音字段不合法 +- 提供了可选权重字段,但它不是合法的浮点数 + +即使某一行出错,后续行仍然会继续读取。 + +## 保存后的文本格式 + +当以文本格式写出时,每个条目都会保存为: + +```text + +``` + +例如: + +```text +X光 X'guang 0 +"你\\n好 不\\\"" ni 0 +倪辉 ni'hui 0 +``` + +汉字字段在需要时会自动转义。 diff --git a/src/libime/pinyin/pinyindictionary.cpp b/src/libime/pinyin/pinyindictionary.cpp index 3a8a870..047ee3c 100644 --- a/src/libime/pinyin/pinyindictionary.cpp +++ b/src/libime/pinyin/pinyindictionary.cpp @@ -266,6 +266,9 @@ PinyinDictionary::TrieType loadTextImpl(std::istream &in) { } catch (const std::invalid_argument &e) { LIBIME_ERROR() << "Skipped line " << lineNo << ", exception: " << e.what(); + } catch (const std::out_of_range &e) { + LIBIME_ERROR() + << "Skipped line " << lineNo << ", exception: " << e.what(); } } } diff --git a/src/libime/pinyin/shuangpin-profile-format.md b/src/libime/pinyin/shuangpin-profile-format.md new file mode 100644 index 0000000..a9a22e1 --- /dev/null +++ b/src/libime/pinyin/shuangpin-profile-format.md @@ -0,0 +1,294 @@ +# Shuangpin profile text format + +This document describes the text format for a custom shuangpin profile such +as `sp.dat`. + +For an in-repository example, see `test/testshuangpinprofile.cpp`. + +## Overview + +A shuangpin profile is a UTF-8 text file. In practice it is usually written +with these sections: + +1. `[方案]` +2. `[零声母标识]` +3. `[声母]` +4. `[韵母]` +5. `[音节]` + +Example: + +```text +[方案] +方案名称=自定义 + +[零声母标识] +=O* + +[声母] +ch=I +sh=U +zh=V + +[韵母] +ai=L +an=J +ang=H +ao=K +ei=Z +en=F +eng=G +er=R +ia=W +ian=M +iang=D +iao=C +ie=X +in=N +ing=Y +iong=S +iu=Q +ong=S +ou=B +ua=W +uai=Y +uan=R +uang=D +ue=T +ui=V +un=P +ve=T +uo=O + +[音节] +ju=jv +shi=ui +e=ee +``` + +## Line rules + +- The file is line-based. +- Blank lines are ignored. +- Lines whose first non-space character is `#` are comments. +- Leading and trailing whitespace is ignored. +- Keys on the right-hand side are case-insensitive; they are normalized to + lowercase. + +Section headers such as `[方案]` and `[韵母]` are recommended for readability, +but the meaningful content is carried by the mapping lines themselves. + +## 1. Scheme name + +Use: + +```text +方案名称= +``` + +Example: + +```text +方案名称=自定义 +``` + +For a custom profile, use a custom name such as `自定义` or any other non-built-in +name. + +These built-in names are special: + +- `自然码` +- `微软` +- `紫光` +- `拼音加加` +- `中文之星` +- `智能ABC` +- `小鹤` + +If `方案名称=` is set to one of those names, the remaining custom mapping lines in +the file are ignored. For a custom profile, avoid using those names. + +## 2. Zero-initial marker + +Use: + +```text += +``` + +Examples: + +```text +=O* +=a +=o +``` + +Meaning: + +- every non-`*` character in `` can be used as the first key of a + zero-initial syllable +- the value is case-insensitive + +Examples: + +- `=o` allows `o` as a zero-initial marker +- `=a` allows `a` as a zero-initial marker +- `=O*` means `o` is a zero-initial marker and `*` enables the special + Ziranma/Xiaohe-style zero-initial rules described below + +### `*` special behavior + +If the marker string contains `*`, zero-initial syllables are additionally +generated in this style: + +1. one-letter finals use a doubled key, such as `aa`, `ee` +2. two-letter finals keep their quanpin form, such as `ou`, `en` +3. longer finals use the first letter of full pinyin plus the mapped final key + +For example, if `ang=g`, then zero-initial `ang` can be entered as: + +```text +ag +``` + +### Default behavior + +If no valid zero-initial marker line is provided, the default marker is `o`. + +A line containing only: + +```text += +``` + +does not define an empty marker; it leaves the default behavior unchanged. + +## 3. Initial mappings + +Use: + +```text += +``` + +Examples: + +```text +ch=u +sh=i +zh=v +``` + +The right-hand side must be exactly one character. + +Multiple mappings are allowed: + +```text +ch=u +ch=; +zh=o +zh=v +``` + +You do not need to list initials whose shuangpin key is already the same as +their normal one-letter initial. For example, `b=b` and `m=m` are unnecessary. + +## 4. Final mappings + +Use: + +```text += +``` + +Examples: + +```text +iu=q +ian=w +uang=z +ui=; +v=v +``` + +The right-hand side must be exactly one character. + +Multiple mappings are allowed: + +```text +ua=a +ua=m +uo=p +uo=b +``` + +Single-letter finals such as `a`, `e`, `i`, `o`, `u`, `v` do not always need to +be listed. They already work as direct inputs unless you want to override or +add aliases. + +## 5. Explicit syllable mappings + +Use: + +```text += +``` + +Examples: + +```text +ju=jv +e=ee +e=e; +shi=is +zhong=os +yue=yb +``` + +This is used for explicit full-syllable overrides or aliases that cannot be +described well enough by only separate initial/final mappings. + +Rules: + +- the left-hand side must be a valid full pinyin syllable +- the right-hand side must be exactly two characters +- the two-key input is case-insensitive +- multiple explicit mappings are allowed for the same syllable + +This section is especially useful for: + +- zero-initial syllables such as `e`, `ou`, `er` +- syllables with multiple accepted shuangpin forms +- compatibility aliases +- custom disambiguation choices + +## How the sections work together + +The profile is built from three kinds of information: + +1. initial mappings +2. final mappings +3. explicit full-syllable mappings + +In addition: + +- one-letter initials are accepted directly even if they are not listed +- one-letter finals are also available directly +- explicit syllable mappings add extra accepted two-key forms +- multiple mappings for the same initial, final, or syllable are allowed + +## Supported characters on the right-hand side + +The key characters are not limited to letters. Punctuation such as `;` is also +allowed, as shown in `test/testshuangpinprofile.cpp`. + +Examples: + +```text +ch=; +ue=; +ui=; +``` + +Each mapping must stay on a single line. diff --git a/src/libime/pinyin/shuangpin-profile-format_zh.md b/src/libime/pinyin/shuangpin-profile-format_zh.md new file mode 100644 index 0000000..7f233c5 --- /dev/null +++ b/src/libime/pinyin/shuangpin-profile-format_zh.md @@ -0,0 +1,287 @@ +# 双拼方案文本格式 + +本文档说明自定义双拼方案配置文件(例如 `sp.dat`)的文本格式。 + +仓库内的示例可参考 `test/testshuangpinprofile.cpp`。 + +## 概览 + +双拼方案文件是一个 UTF-8 文本文件。通常会按下面的结构编写: + +1. `[方案]` +2. `[零声母标识]` +3. `[声母]` +4. `[韵母]` +5. `[音节]` + +示例: + +```text +[方案] +方案名称=自定义 + +[零声母标识] +=O* + +[声母] +ch=I +sh=U +zh=V + +[韵母] +ai=L +an=J +ang=H +ao=K +ei=Z +en=F +eng=G +er=R +ia=W +ian=M +iang=D +iao=C +ie=X +in=N +ing=Y +iong=S +iu=Q +ong=S +ou=B +ua=W +uai=Y +uan=R +uang=D +ue=T +ui=V +un=P +ve=T +uo=O + +[音节] +ju=jv +shi=ui +e=ee +``` + +## 行规则 + +- 文件按行解析。 +- 空行会被忽略。 +- 去掉前导空白后,以 `#` 开头的行会被当作注释。 +- 每行前后的空白会被忽略。 +- 等号右侧的按键大小写不敏感,内部会统一转换为小写。 + +`[方案]`、`[韵母]` 这样的分节标题主要用于增强可读性,真正有意义的是 +其中的映射行。 + +## 1. 方案名称 + +写法: + +```text +方案名称= +``` + +例如: + +```text +方案名称=自定义 +``` + +对于自定义方案,请使用 `自定义` 或其他非内建方案名。 + +下面这些内建名称具有特殊含义: + +- `自然码` +- `微软` +- `紫光` +- `拼音加加` +- `中文之星` +- `智能ABC` +- `小鹤` + +如果 `方案名称=` 被设置为上述内建名称之一,那么文件里后面的自定义映射 +内容会被忽略。因此自定义方案不要使用这些名称。 + +## 2. 零声母标识 + +写法: + +```text += +``` + +例如: + +```text +=O* +=a +=o +``` + +含义: + +- `` 中每个不是 `*` 的字符,都可以作为零声母音节的首键 +- 该值大小写不敏感 + +例如: + +- `=o` 表示允许 `o` 作为零声母标识 +- `=a` 表示允许 `a` 作为零声母标识 +- `=O*` 表示 `o` 是零声母标识,同时 `*` 会启用下面描述的自然码/小鹤式 + 零声母规则 + +### `*` 的特殊行为 + +如果零声母标识中包含 `*`,则还会额外生成这种形式的零声母输入: + +1. 单字母韵母使用重复按键,例如 `aa`、`ee` +2. 双字母韵母保持全拼形式,例如 `ou`、`en` +3. 更长的韵母使用“全拼首字母 + 该韵母映射键” + +例如,如果 `ang=g`,那么零声母 `ang` 可以写成: + +```text +ag +``` + +### 默认行为 + +如果没有提供有效的零声母标识行,默认标识为 `o`。 + +只有下面这一行: + +```text += +``` + +并不表示空零声母标识;它会保持默认行为不变。 + +## 3. 声母映射 + +写法: + +```text += +``` + +例如: + +```text +ch=u +sh=i +zh=v +``` + +等号右侧必须正好是一个字符。 + +允许一对多映射: + +```text +ch=u +ch=; +zh=o +zh=v +``` + +如果某个声母的双拼键本来就和它自身的一字母声母相同,通常不需要显式列出。 +例如 `b=b`、`m=m` 一般没有必要写。 + +## 4. 韵母映射 + +写法: + +```text += +``` + +例如: + +```text +iu=q +ian=w +uang=z +ui=; +v=v +``` + +等号右侧必须正好是一个字符。 + +允许一对多映射: + +```text +ua=a +ua=m +uo=p +uo=b +``` + +像 `a`、`e`、`i`、`o`、`u`、`v` 这样的单字母韵母并不总是需要显式列出。 +除非你要覆盖默认行为或增加别名,否则它们通常可以直接输入。 + +## 5. 显式音节映射 + +写法: + +```text += +``` + +例如: + +```text +ju=jv +e=ee +e=e; +shi=is +zhong=os +yue=yb +``` + +这一部分用于定义完整音节的显式覆盖或别名,适合那些仅靠声母映射和韵母映射 +还不足以表达的情况。 + +规则: + +- 左侧必须是合法的完整拼音音节 +- 右侧必须正好是两个字符 +- 这两个按键大小写不敏感 +- 同一个音节允许定义多个显式映射 + +这一部分尤其适用于: + +- `e`、`ou`、`er` 这类零声母音节 +- 一个音节有多个可接受双拼形式的情况 +- 兼容性别名 +- 自定义消歧选择 + +## 各部分如何共同生效 + +一个双拼方案由三类信息共同构成: + +1. 声母映射 +2. 韵母映射 +3. 显式音节映射 + +此外: + +- 单字母声母即使不写出来也会被直接接受 +- 单字母韵母也会直接可用 +- 显式音节映射会额外增加可接受的两键输入形式 +- 同一个声母、韵母或音节都允许多个映射 + +## 等号右侧支持的字符 + +按键字符不局限于字母,也允许使用 `;` 这样的标点符号,这在 +`test/testshuangpinprofile.cpp` 中就有示例。 + +例如: + +```text +ch=; +ue=; +ui=; +``` + +每条映射都必须写在单独的一行中。 diff --git a/src/libime/table/table-text-format.md b/src/libime/table/table-text-format.md new file mode 100644 index 0000000..4e762ba --- /dev/null +++ b/src/libime/table/table-text-format.md @@ -0,0 +1,312 @@ +# Table file format + +A table file is a UTF-8 text file with up to four parts, in this order: + +1. config +2. optional rule section +3. data section +4. optional phrase section + +Both English and Chinese keywords are accepted: + +| English | Chinese | +| --- | --- | +| `KeyCode=` | `键码=` | +| `Length=` | `码长=` | +| `InvalidChar=` | `规避字符=` | +| `Pinyin=` | `拼音=` | +| `PinyinLength=` | `拼音长度=` | +| `Prompt=` | `提示=` | +| `ConstructPhrase=` | `构词=` | +| `[Rule]` | `[组词规则]` | +| `[Data]` | `[数据]` | +| `[Phrase]` | `[词组]` | + +## Example + +```text +KeyCode=abcdefghijklmnopqrstuvwxy +Length=4 +Pinyin=@ +[Rule] +e2=p11+p12+p21+p22 +e3=p11+p21+p31+p32 +a4=p11+p21+p31+n11 +[Data] +xycq 统 +yfh 计 +nnkd 局 +[Phrase] +统计局 +``` + +## Typical use cases + +### Chinese table input methods + +For Chinese table input methods such as Wubi, Cangjie, Zhengma, or other +shape-based systems, a table file often needs more than simple `code -> text` +entries. + +Common needs: + +- single-character entries in `[Data]` +- phrase generation rules in `[Rule]` +- optional `[Phrase]` entries for common multi-character words +- optional `Pinyin=` entries for pinyin-assisted lookup +- optional `Prompt=` entries for decomposition or radical hints +- optional `ConstructPhrase=` entries when phrase generation should use a + dedicated per-character construction code + +This is the typical full-featured table format. + +### Other languages or simpler code tables + +For non-Chinese input methods, symbol tables, transliteration schemes, or any +other simple code-based lookup table, usually only the basic configuration and +`[Data]` section are needed. + +Typical cases: + +- custom symbol input +- shorthand expansion +- transliteration-based word lookup +- small code-to-text dictionaries + +In these cases, you can often omit: + +- `[Rule]` +- `[Phrase]` +- `Pinyin=` +- `Prompt=` +- `ConstructPhrase=` + +The minimal practical form is: + +```text +KeyCode=abcdefghijklmnopqrstuvwxyz +Length=6 +[Data] +ab example +abc another +``` + +## 1. Configuration + +The file starts with configuration lines. Leading and trailing whitespace is +ignored. Lines whose first non-space character is `#` are comments. This comment +rule applies only to the configuration and rule sections, not to the `[Data]` +and `[Phrase]` sections. + +Supported keys: + +- `KeyCode=`: required set of valid input-code characters. +- `Length=`: maximum code length for normal entries. +- `InvalidChar=`: code characters that should not be used as the first + character of a single-character entry when building the per-character lookup + used for phrase generation. This is mainly useful for special leading codes in + some tables. +- `Pinyin=`: one character used as the prefix for pinyin entries in `[Data]`. +- `Prompt=`: one character used as the prefix for prompt/hint entries in + `[Data]`. +- `ConstructPhrase=`: one character used as the prefix for phrase-construction + entries in `[Data]`. +- `PinyinLength=`: reserved for compatibility. + +`[Data]` ends the config section and starts the main dictionary data. + +`[Rule]` switches to the rule section. A later `[Data]` is still required. + +## 2. Rule section + +The rule section is optional. It is only needed when you want phrase codes to +be generated automatically. + +For Wubi, `wbx.txt` uses this rule set: + +```text +[组词规则] +e2=p11+p12+p21+p22 +e3=p11+p21+p31+p32 +a4=p11+p21+p31+n11 +``` + +Rule syntax: + +- `eN=...`: apply when phrase length is exactly `N`. +- `aN=...`: apply when phrase length is at least `N`. +- The right-hand side is a `+`-joined list of rule entries. +- Each rule entry is three characters: + - `p` or `n`: pick the phrase character from the front or the back. + - the second character: which phrase character to use (`1`-`9`). + - the third character: which code position to use from that character's + code: + - `1`-`9`: from the front + - `z`: last code element + - `y`: second last + - `x`: third last + - and so on +- `p00` is a placeholder and contributes nothing. + +For example, `p1z` means "take the last code element of the first character in +the phrase". + +### Wubi rule meaning + +These three rules are the standard phrase-building rules used by `wbx.txt`: + +- `e2=p11+p12+p21+p22` + - 2-character phrase + - take the first two code elements from the first character + - then the first two code elements from the second character +- `e3=p11+p21+p31+p32` + - 3-character phrase + - take the first code element from the first, second, and third characters + - then take the second code element from the third character +- `a4=p11+p21+p31+n11` + - phrase length 4 or more + - take the first code element from the first three characters + - then take the first code element from the last character + +### Wubi example: `百合 -> djwg` + +Assume the table already contains these single-character codes from `wbx.txt`: + +```text +dj 百 +wgk 合 +``` + +For a 2-character phrase, Wubi uses: + +```text +e2=p11+p12+p21+p22 +``` + +This rule means: + +- `e2`: apply to a 2-character phrase +- `p11`: take the 1st code element from the 1st character +- `p12`: take the 2nd code element from the 1st character +- `p21`: take the 1st code element from the 2nd character +- `p22`: take the 2nd code element from the 2nd character + +Applying it to `百合`: + +- `百` has code `dj`, so it contributes `d` and `j` +- `合` has code `wgk`, so it contributes `w` and `g` + +The generated phrase code is therefore: + +```text +百合 -> djwg +``` + +`[Data]` ends the rule section. + +## 3. Data section + +Each data line contains a key and a value, separated by whitespace: + +```text + +``` + +Examples: + +```text +a 日 +aa 昌 +abac 暝 +``` + +Normal entries use only characters listed by `KeyCode=` in ``. + +If `Pinyin=`, `Prompt=`, or `ConstructPhrase=` is configured, a key may start +with that prefix character: + +```text +@ni 你 +&a 日 +^cq 统 +``` + +These mean: + +- `@ni 你`: pinyin entry for `你` +- `&a 日`: prompt/hint text for code `a` +- `^cq 统`: construction code for phrase generation + +Important details: + +- The key must not be empty. +- `Prompt=` entries provide hint text, not normal dictionary candidates. +- `ConstructPhrase=` entries are used only for phrase-code generation. They are + not inserted as normal candidates by themselves. +- Blank or malformed data lines are ignored. +- In `[Data]`, `#` is **not** a comment marker; it is just text unless it makes + the line malformed. + +## 4. Phrase section + +`[Phrase]` is optional and may appear only after `[Data]`. It is valid only +when rules are present. + +Each following line is a phrase value only: + +```text +[Phrase] +统计局 +中华人民共和国 +``` + +The phrase code is generated from `[Rule]` plus the single-character entries +already defined earlier in the file. + +After `[Phrase]`, every remaining line is treated as a phrase. + +## Escaping + +Values in `[Data]` and lines in `[Phrase]` may be quoted. Use quoting when a +value contains whitespace, `"` or `\`. + +Each record in the file is still line-based: one entry per physical line. +Literal multi-line entries are not written as raw multi-line text inside the +file. + +If you need a value to contain an embedded newline, write it with escaping +inside a quoted string, for example `\n`. + +Examples: + +```text +aaaa "工 " +f "\"" +note "first line\nsecond line" +[Phrase] +"统计局" +``` + +Quoted values may use backslash escaping, including: + +- `\"` for a literal quote +- `\\` for a literal backslash +- `\n` for an embedded newline + +So this: + +```text +note "first line\nsecond line" +``` + +represents a single value containing two lines, even though the table file +itself still stores it as one physical line. + +## Requirements and restrictions + +- The file must be valid UTF-8. +- `KeyCode=` must not contain the configured `Pinyin=`, `Prompt=`, or + `ConstructPhrase=` prefix character. +- Data-entry keys must not be empty. +- `[Data]` is required. +- `[Phrase]` requires a rule section. diff --git a/src/libime/table/table-text-format_zh.md b/src/libime/table/table-text-format_zh.md new file mode 100644 index 0000000..4b5cdfd --- /dev/null +++ b/src/libime/table/table-text-format_zh.md @@ -0,0 +1,294 @@ +# 码表文件格式 + +码表文件是一个 UTF-8 文本文件,最多可以包含以下四个部分,顺序如下: + +1. 配置部分 +2. 可选的组词规则部分 +3. 数据部分 +4. 可选的词组部分 + +文件同时支持英文和中文关键字: + +| 英文 | 中文 | +| --- | --- | +| `KeyCode=` | `键码=` | +| `Length=` | `码长=` | +| `InvalidChar=` | `规避字符=` | +| `Pinyin=` | `拼音=` | +| `PinyinLength=` | `拼音长度=` | +| `Prompt=` | `提示=` | +| `ConstructPhrase=` | `构词=` | +| `[Rule]` | `[组词规则]` | +| `[Data]` | `[数据]` | +| `[Phrase]` | `[词组]` | + +## 示例 + +```text +KeyCode=abcdefghijklmnopqrstuvwxy +Length=4 +Pinyin=@ +[Rule] +e2=p11+p12+p21+p22 +e3=p11+p21+p31+p32 +a4=p11+p21+p31+n11 +[Data] +xycq 统 +yfh 计 +nnkd 局 +[Phrase] +统计局 +``` + +## 典型使用场景 + +### 中文码表输入法 + +对于五笔、仓颉、郑码以及其他形码输入法来说,码表文件通常不只是简单的 +`编码 -> 文本` 对应关系。 + +常见需求包括: + +- 在 `[Data]` 中定义单字条目 +- 在 `[Rule]` 中定义组词规则 +- 在 `[Phrase]` 中可选地加入常用多字词 +- 可选地使用 `Pinyin=` 提供拼音辅助查询 +- 可选地使用 `Prompt=` 提供拆字或字根提示 +- 可选地使用 `ConstructPhrase=` 为组词提供专门的单字构词编码 + +这是最常见的完整功能码表格式。 + +### 其他语言或更简单的编码表 + +对于非中文输入法、符号码表、转写方案,或者任何其他简单的编码查找表, +通常只需要基本配置和 `[Data]` 部分。 + +典型场景包括: + +- 自定义符号输入 +- 简写展开 +- 基于转写的词语查找 +- 小型编码到文本字典 + +这些场景通常可以省略: + +- `[Rule]` +- `[Phrase]` +- `Pinyin=` +- `Prompt=` +- `ConstructPhrase=` + +最小可用形式如下: + +```text +KeyCode=abcdefghijklmnopqrstuvwxyz +Length=6 +[Data] +ab example +abc another +``` + +## 1. 配置部分 + +文件开头是配置行。每一行前后的空白会被忽略。去掉前导空白后,如果一行 +的第一个字符是 `#`,则该行会被视为注释。这个注释规则只适用于配置和规则部分, +不适用于 `[Data]` 和 `[Phrase]` 部分。 + +支持的键包括: + +- `KeyCode=`:必填,定义可用于编码的字符集合。 +- `Length=`:普通条目的最大编码长度。 +- `InvalidChar=`:定义某些编码字符不应作为单字条目的首码参与单字反查表 + 的构建。它主要用于某些码表中的特殊前导编码。 +- `Pinyin=`:定义 `[Data]` 中拼音条目的前缀字符。 +- `Prompt=`:定义 `[Data]` 中提示条目的前缀字符。 +- `ConstructPhrase=`:定义 `[Data]` 中构词条目的前缀字符。 +- `PinyinLength=`:保留作兼容用途。 + +`[Data]` 表示配置部分结束,并开始主数据部分。 + +`[Rule]` 表示切换到规则部分。之后仍然必须出现 `[Data]`。 + +## 2. 组词规则部分 + +规则部分是可选的。只有在你希望自动生成词组编码时才需要。 + +对于五笔,`wbx.txt` 使用下面这组规则: + +```text +[组词规则] +e2=p11+p12+p21+p22 +e3=p11+p21+p31+p32 +a4=p11+p21+p31+n11 +``` + +规则语法如下: + +- `eN=...`:当词组长度恰好为 `N` 时使用。 +- `aN=...`:当词组长度大于等于 `N` 时使用。 +- 等号右侧是由 `+` 连接起来的一组规则项。 +- 每个规则项都由三个字符组成: + - `p` 或 `n`:从词组前面或后面取字。 + - 第二个字符:取第几个字(`1`-`9`)。 + - 第三个字符:取该字编码中的第几个码元: + - `1`-`9`:从前往后数 + - `z`:最后一个码元 + - `y`:倒数第二个码元 + - `x`:倒数第三个码元 + - 依此类推 +- `p00` 是占位符,不产生任何码元。 + +例如,`p1z` 表示“取词组第一个字编码的最后一个码元”。 + +### 五笔规则含义 + +这三条规则就是 `wbx.txt` 使用的标准组词规则: + +- `e2=p11+p12+p21+p22` + - 适用于二字词 + - 取第一个字的前两个码元 + - 再取第二个字的前两个码元 +- `e3=p11+p21+p31+p32` + - 适用于三字词 + - 依次取第一、第二、第三个字的第一个码元 + - 然后再取第三个字的第二个码元 +- `a4=p11+p21+p31+n11` + - 适用于四字及以上词组 + - 取前三个字的第一个码元 + - 再取最后一个字的第一个码元 + +### 五笔示例:`百合 -> djwg` + +假设码表中已经包含来自 `wbx.txt` 的以下单字编码: + +```text +dj 百 +wgk 合 +``` + +对于一个二字词,五笔使用: + +```text +e2=p11+p12+p21+p22 +``` + +这条规则的含义是: + +- `e2`:适用于 2 个字的词组 +- `p11`:取第 1 个字的第 1 个码元 +- `p12`:取第 1 个字的第 2 个码元 +- `p21`:取第 2 个字的第 1 个码元 +- `p22`:取第 2 个字的第 2 个码元 + +应用到 `百合` 上时: + +- `百` 的编码是 `dj`,因此提供 `d` 和 `j` +- `合` 的编码是 `wgk`,因此提供 `w` 和 `g` + +因此,生成的词组编码就是: + +```text +百合 -> djwg +``` + +`[Data]` 表示规则部分结束。 + +## 3. 数据部分 + +每一条数据行都由一个 key 和一个 value 组成,中间以空白字符分隔: + +```text + +``` + +例如: + +```text +a 日 +aa 昌 +abac 暝 +``` + +普通条目的 `` 只能使用 `KeyCode=` 中列出的字符。 + +如果配置了 `Pinyin=`、`Prompt=` 或 `ConstructPhrase=`,那么 key 可以以前缀 +字符开头: + +```text +@ni 你 +&a 日 +^cq 统 +``` + +这些分别表示: + +- `@ni 你`:`你` 的拼音条目 +- `&a 日`:编码 `a` 的提示文本 +- `^cq 统`:用于组词的构词编码 + +需要注意: + +- key 不能为空。 +- `Prompt=` 条目提供提示文本,不属于普通候选词条。 +- `ConstructPhrase=` 条目只用于组词编码生成,本身不会作为普通候选词条插入。 +- 空行或格式错误的数据行会被忽略。 +- 在 `[Data]` 中,`#` **不是** 注释标记;除非它让这一行格式非法,否则它就只是普通文本。 + +## 4. 词组部分 + +`[Phrase]` 是可选的,并且只能出现在 `[Data]` 之后。只有在规则部分存在 +时,它才有效。 + +后续每一行都只写词组本身: + +```text +[Phrase] +统计局 +中华人民共和国 +``` + +词组编码会根据 `[Rule]` 和前面已经定义好的单字条目自动生成。 + +一旦进入 `[Phrase]`,后面所有行都会被当作词组处理。 + +## 转义 + +`[Data]` 中的 value,以及 `[Phrase]` 中的整行内容,都可以使用带引号的形式。 +当值中包含空白字符、`"` 或 `\` 时,建议使用引号。 + +文件中的每一条记录仍然是按“行”组织的:一个条目占用一个物理行。 +真正的多行条目不能直接以原始多行文本写在文件中。 + +如果希望一个值内部包含换行,可以在带引号的字符串中使用转义,例如 `\n`。 + +例如: + +```text +aaaa "工 " +f "\"" +note "first line\nsecond line" +[Phrase] +"统计局" +``` + +带引号的值支持反斜杠转义,包括: + +- `\"`:字面意义上的双引号 +- `\\`:字面意义上的反斜杠 +- `\n`:嵌入式换行 + +所以: + +```text +note "first line\nsecond line" +``` + +表示的是一个内部包含两行内容的单个值,虽然码表文件本身仍然只占一行。 + +## 要求与限制 + +- 文件必须是合法的 UTF-8。 +- `KeyCode=` 不能包含已经配置给 `Pinyin=`、`Prompt=` 或 `ConstructPhrase=` 的前缀字符。 +- 数据条目的 key 不能为空。 +- `[Data]` 是必需的。 +- `[Phrase]` 必须依赖规则部分。