Add doc for some text file format.

README.md

@@ -6,3 +6,12 @@ libime
 [![Documentation](https://codedocs.xyz/fcitx/libime.svg)](https://codedocs.xyz/fcitx/libime/)
 
 This is a library to support generic input method implementation.
+
+Documentation:
+
+- [Pinyin dictionary text format](src/libime/pinyin/pinyindictionary-text-format.md)
+- [拼音词典文本格式](src/libime/pinyin/pinyindictionary-text-format_zh.md)
+- [Shuangpin profile text format](src/libime/pinyin/shuangpin-profile-format.md)
+- [双拼方案文本格式](src/libime/pinyin/shuangpin-profile-format_zh.md)
+- [Table text format](src/libime/table/table-text-format.md)
+- [码表文本格式](src/libime/table/table-text-format_zh.md)

src/libime/pinyin/pinyindictionary-text-format.md

@@ -0,0 +1,158 @@
+# Pinyin dictionary text format
+
+This document describes the plain-text format used by `PinyinDictionary`.
+
+For in-repository examples, see `test/testpinyindictionary.cpp` and the
+generated dictionary source used by tests such as `data/dict_sc.txt`.
+
+## Overview
+
+The format is line-based. Each dictionary entry is written on one line in one
+of these forms:
+
+```text
+<hanzi> <full-pinyin>
+<hanzi> <full-pinyin> <weight>
+```
+
+Examples:
+
+```text
+倪辉 ni'hui 0.0
+小企鹅 xiao'qi'e
+X光 X'guang
+```
+
+## Fields
+
+### 1. Hanzi field
+
+The first field is the word or phrase text.
+
+Examples:
+
+```text
+你
+小企鹅
+X光
+```
+
+This field may be quoted when it contains whitespace, quotes, backslashes, or
+escaped newlines.
+
+Example:
+
+```text
+"你\n好 不\"" ni
+```
+
+That entry represents the text:
+
+```text
+你
+好 不"
+```
+
+### 2. Full pinyin field
+
+The second field is the full pinyin spelling.
+
+Examples:
+
+```text
+ni
+ni'hui
+xiao'qi'e
+X'guang
+```
+
+Important details:
+
+- syllables are separated with `'`
+- upper-case letters are allowed when the spelling needs them, for example
+  `X'guang`
+- the pinyin must be a valid full pinyin spelling accepted by libime
+
+### 3. Weight field
+
+The third field is optional and is parsed as a floating-point number.
+
+Examples:
+
+```text
+0
+0.0
+1.25
+-3
+```
+
+If the weight is omitted, the entry is loaded with `0.0`.
+
+When the dictionary is saved back to text, every entry is written with an
+explicit weight field, so an entry that was originally:
+
+```text
+小企鹅 xiao'qi'e
+```
+
+will be written back as:
+
+```text
+小企鹅 xiao'qi'e 0
+```
+
+## Escaping and quoting
+
+Fields are tokenized with support for quoted and escaped values.
+
+Use quotes when a field contains whitespace or special characters:
+
+```text
+"你\n好 不\"" ni
+```
+
+Supported escaped forms are the usual backslash-escaped forms accepted by the
+value parser, including:
+
+- `\"` for a literal quote
+- `\\` for a literal backslash
+- `\n` for an embedded newline
+
+Each entry still occupies one physical line in the file. If you need embedded
+newlines inside the text field, represent them with escapes such as `\n`.
+
+## Line behavior
+
+- Blank lines are ignored.
+- Lines with anything other than 2 or 3 parsed fields are ignored.
+- There is no section syntax in this format.
+- Comment syntax is not part of the format. A line beginning with `#` is simply
+  treated as an invalid entry and ignored unless it happens to parse as a valid
+  2-field or 3-field record.
+
+## Validation behavior
+
+An entry is skipped if:
+
+- the pinyin field is invalid
+- the optional weight field is present but is not a valid floating-point number
+
+The loader continues reading later lines after such errors.
+
+## Saved text format
+
+When written back as text, each entry is saved in this form:
+
+```text
+<escaped-hanzi> <full-pinyin> <weight>
+```
+
+Examples:
+
+```text
+X光 X'guang 0
+"你\\n好 不\\\"" ni 0
+倪辉 ni'hui 0
+```
+
+The hanzi field is automatically escaped when needed.

src/libime/pinyin/pinyindictionary-text-format_zh.md

@@ -0,0 +1,152 @@
+# 拼音词典文本格式
+
+本文档说明 `PinyinDictionary` 使用的纯文本格式。
+
+仓库内的示例可参考 `test/testpinyindictionary.cpp`，以及测试所使用的生成词典
+源文件，例如 `data/dict_sc.txt`。
+
+## 概览
+
+该格式按行组织。每个词典条目占一行，支持以下两种形式：
+
+```text
+<hanzi> <full-pinyin>
+<hanzi> <full-pinyin> <weight>
+```
+
+例如：
+
+```text
+倪辉 ni'hui 0.0
+小企鹅 xiao'qi'e
+X光 X'guang
+```
+
+## 字段说明
+
+### 1. 汉字字段
+
+第一个字段是词或短语文本。
+
+例如：
+
+```text
+你
+小企鹅
+X光
+```
+
+当该字段中包含空白、引号、反斜杠或转义换行时，可以使用引号包裹。
+
+例如：
+
+```text
+"你\n好 不\"" ni
+```
+
+这个条目表示的文本是：
+
+```text
+你
+好 不"
+```
+
+### 2. 全拼字段
+
+第二个字段是完整拼音。
+
+例如：
+
+```text
+ni
+ni'hui
+xiao'qi'e
+X'guang
+```
+
+需要注意：
+
+- 音节之间使用 `'` 分隔
+- 在需要时允许使用大写字母，例如 `X'guang`
+- 该拼音必须是 libime 接受的合法全拼形式
+
+### 3. 权重字段
+
+第三个字段是可选的，按浮点数解析。
+
+例如：
+
+```text
+0
+0.0
+1.25
+-3
+```
+
+如果省略该字段，则按 `0.0` 载入。
+
+当词典再保存回文本格式时，每个条目都会显式写出权重字段。因此原本写成：
+
+```text
+小企鹅 xiao'qi'e
+```
+
+保存后会变成：
+
+```text
+小企鹅 xiao'qi'e 0
+```
+
+## 转义与引号
+
+字段在分词时支持带引号和带转义的值。
+
+当字段中包含空白或特殊字符时，请使用引号：
+
+```text
+"你\n好 不\"" ni
+```
+
+支持的转义形式包括常见的反斜杠转义，例如：
+
+- `\"` 表示字面意义上的双引号
+- `\\` 表示字面意义上的反斜杠
+- `\n` 表示嵌入式换行
+
+每个条目仍然只占用文件中的一个物理行。如果需要在文本字段内部包含换行，
+应使用 `\n` 这样的转义形式表示。
+
+## 行行为
+
+- 空行会被忽略。
+- 解析后字段数不是 2 个或 3 个的行会被忽略。
+- 该格式没有分节语法。
+- 注释不是该格式的一部分。以 `#` 开头的行不会被当作注释处理；除非它恰好
+  不能解析为合法的 2 字段或 3 字段记录，否则它只是普通输入。
+
+## 校验行为
+
+在以下情况下，条目会被跳过：
+
+- 拼音字段不合法
+- 提供了可选权重字段，但它不是合法的浮点数
+
+即使某一行出错，后续行仍然会继续读取。
+
+## 保存后的文本格式
+
+当以文本格式写出时，每个条目都会保存为：
+
+```text
+<escaped-hanzi> <full-pinyin> <weight>
+```
+
+例如：
+
+```text
+X光 X'guang 0
+"你\\n好 不\\\"" ni 0
+倪辉 ni'hui 0
+```
+
+汉字字段在需要时会自动转义。

src/libime/pinyin/pinyindictionary.cpp

@@ -266,6 +266,9 @@ PinyinDictionary::TrieType loadTextImpl(std::istream &in) {
             } catch (const std::invalid_argument &e) {
                 LIBIME_ERROR()
                     << "Skipped line " << lineNo << ", exception: " << e.what();
+            } catch (const std::out_of_range &e) {
+                LIBIME_ERROR()
+                    << "Skipped line " << lineNo << ", exception: " << e.what();
             }
         }
     }