cut: support multibyte characters in non-UTF-8 locales

[sylvestre]

No description provided.

[codspeed-hq]

Merging this PR will degrade performance by 4.67%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
❌ 2 regressed benchmarks
✅ 342 untouched benchmarks
⏩ 46 skipped benchmarks¹

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

	Mode	Benchmark	BASE	HEAD	Efficiency
❌	Simulation	cut_characters	17.1 ms	23.3 ms	-26.55%
❌	Simulation	du_deep_tree[(100, 3)]	2.3 ms	2.3 ms	-3.67%
⚡	Simulation	df_with_path	699.4 µs	571.2 µs	+22.44%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.

---

_{Comparing sylvestre:fix-cut-mb-non-utf8 (e34f4db) with main (a0cb024)}

Footnotes

1. 46 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

[github-actions]

GNU testsuite comparison:

Skipping an intermittent issue tests/pr/bounded-memory (passes in this run but fails in the 'main' branch)
Skipping an intermittent issue tests/tail/tail-n0f (passes in this run but fails in the 'main' branch)
Congrats! The gnu test tests/cut/mb-non-utf8 is no longer failing!
Note: The gnu test tests/cp/link-heap is now being skipped but was previously passing.
Note: The gnu test tests/rm/many-dir-entries-vs-OOM is now being skipped but was previously passing.

[sylvestre]

@codspeedbot fix this regression

[codspeed-hq]

@codspeedbot fix this regression

I reproduced the cut_characters regression locally (32.6 ms, matching the 32.8 ms reported here) and pushed a fix: 32.8 ms → 12.0 ms.

What caused it

With a UTF-8 LANG (the case on the CI runners), -c no longer takes the plain byte path, and the new cut_chars loop was doing two expensive things per line:

• it called write_all once per character instead of once per range — ~26 buffered writes per line on this benchmark;
• it restarted the character walk from the beginning of the line for every range.

What changed

src/uu/cut/src/cut.rs only:

• the line is walked once for all ranges (Range::from_list returns them sorted and disjoint, so each range maps to a contiguous slice) and each range is written with a single write_all;
• when a range reaches past the end of the line, the remaining characters are not decoded at all — a character can never end before its own byte offset, so everything left is covered. This also makes the very common cut -c1-N on short lines nearly free;
• ASCII bytes are handled inline, keeping them out of the decoder — they are single-byte characters in every encoding handled by charmap.

Correctness

Output is byte-for-byte identical to the current implementation:

• 800k randomised cases (lines with valid/invalid/truncated multi-byte sequences, multiple ranges, -c and -b -n, with and without an output delimiter);
• 360 invocations of the old vs new binary over UTF-8, GB18030, EUC-JP and single-byte locales, on the benchmark data, GB18030 data, 200 KB of random bytes and an empty file, covering ranges, complements and custom output delimiters;
• the cut test suite (61 tests) passes, along with clippy and fmt.

Remaining gap

cut_characters stays above the 7.4 ms of main because main never looks at character boundaries for -c — that decoding is the point of this PR. Locally the whole cut -c5-30 run costs 30.1M instructions versus 19.0M for the byte path; a few further variants I tried (word-at-a-time ASCII scanning, #[inline] on mb_char_len, specialising the loop on -c vs -b -n) all measured worse, so I kept the simplest version.

One note: uucore::i18n::charmap::mb_chars is no longer used by cut after this change. I left it in place as a public helper — say the word and I'll remove it.

Summary

• Warming up the performance wizard...
• Investigating cut_characters regression — Root cause identified: the new -c path writes one character at a time and rescans the line per range
• Building and measuring the baseline — Reproduced the regression locally: cut_characters at 32.6 ms with a UTF-8 locale
• Optimizing the multibyte cut path — cut_characters: 32.8 ms -> 12.0 ms, output unchanged; commit pushed to the PR branch
• Handling comment — Comment handling finished successfully

_{Was this response helpful? React with 👍 or 👎 to help us improve CodSpeedBot.}