test(windows): make the suite runnable on Windows by JamBalaya56562 · Pull Request #771 · jdx/usage

JamBalaya56562 · 2026-08-02T04:24:18Z

Groundwork for running the test suite on Windows. Today it gives 15 failures and one file that never runs, and every one of them is the test's own doing rather than a bug in usage.

Measured on a windows-latest runner after these changes: 538 passed, 0 failed, 0 skipped.

What was wrong

Where	Failures	Cause
`cli/tests/complete_word.rs`	8	the skip guard probes `sh`, but the fixtures need `bash`
`cli/tests/shell_completions_integration.rs`	5	`bash` is the WSL launcher, and Windows paths get eaten
`cli/tests/examples.rs`	1	spawns a `.sh` directly
`lib/tests/sdk_compile.rs`	1	a Windows path inside a Python string literal
`cli/tests/shell_override.rs`	—	`#![cfg(unix)]`, so 5 tests never run at all

The last one is the sharpest: those are the tests for USAGE_SHELL_<SHELL> from #767, a feature that exists for Windows, and they only ever ran on Linux.

`bash` on Windows is not a POSIX bash

The Win32 executable search order puts the system directory ahead of PATH, and installing WSL puts bash.exe there. So on such a machine bash is the WSL launcher no matter what else is installed, and it cannot open a Windows path:

/bin/bash: C:UsersJamAppDataLocalTempusage_bash_test_14636test.sh: No such file or directory

Two problems in one line. WSL cannot read C:\…, and the backslashes were already gone before it looked — the path was interpolated into a script body where \ is an escape character, so C:\Users\Jam arrived as C:UsersJam.

This is not exotic; a stock GitHub runner does it too. On windows-latest:

> where bash
C:\Program Files\Git\bin\bash.exe
C:\Windows\System32\bash.exe

Git Bash is listed first, and yet Command::new("bash") reaches System32 — because where searches PATH while CreateProcess searches the system directory before it. Unset, usage bash there exits 1 with Windows Subsystem for Linux has no installed distributions. The two disagreeing is exactly what USAGE_SHELL_<SHELL> exists to settle.

The skip guards were checking the wrong thing

skip_if_shell_missing asked whether <shell> --version spawns. WSL's bash answers that perfectly well and then fails at everything the tests need, which is why five bash tests failed where zsh and fish correctly skipped — those are simply absent on Windows.

skip_if_posix_shell_missing in complete_word.rs had the same shape of gap for a subtler reason: it probes sh, but the mount fixtures are #!/usr/bin/env -S usage bash scripts. On Windows the two need not be the same family — sh resolves to Git Bash while a bare bash is the WSL launcher — so sh starting proves nothing about whether the shebang will work.

What changed

Tests only. No library or CLI source is touched.

Guards now probe the precondition, not a proxy. skip_if_shell_missing runs a script out of the temp directory and requires it to exit cleanly; skip_if_posix_shell_missing runs a real mount fixture through usage bash at the absolute path sh would hand its shebang. Both keep the existing rule of panicking under CI rather than skipping. On Linux nothing changes; on Windows an unusable shell now skips instead of failing.

The fixture path is built from CARGO_MANIFEST_DIR rather than fs::canonicalize, whose \\?\ prefix usage cannot open — probing with one made the guard fail on the shape of the path rather than on the shell, and quietly skipped all eight mount tests on a machine where they run.

Shell resolution goes through USAGE_SHELL_<SHELL> — the same variable shell_program_override reads in cli/src/env.rs, so a developer configures one thing and the whole suite follows. Unset, which is every platform but Windows, means the bare name as before.

One script_command builds every invocation, so the guard and the tests it guards cannot drift apart — not in which program they run, nor in how. pwsh is where they would have: it needs -File to take a script path, and -NoProfile -NonInteractive so a developer's profile cannot add output to a run whose stdout is compared against a single token.

Three places hand a Windows path to something that parses it, and each needed a different answer:

Shell script bodies — normalized to /, because \ is an escape character there.
$PATH entries — through cygpath, because $PATH is colon-separated, so C:/Users/… splits into C and /Users/… and neither resolves. Only the shell in use knows whether the answer is /c/…, /cygdrive/c/…, or a mount point from fstab, so the conversion is asked of it rather than of a cygpath that may not be on Windows' own PATH. Applies to bash and zsh; fish's set -gx PATH a b is space-separated and PowerShell's separator is ;.
A Python string literal — sdk_compile.rs put the SDK directory inside one, where C:\Users\RUNNER~1\… fails to parse at all: \U opens a unicode escape. That one now travels as argv, clear of any escaping rules rather than escaped around.

The zsh pty test's timeout wrapper also goes through the shell now. Spawning it directly walks into the same trap as bash: Windows keeps an unrelated timeout.exe — a sleep, not a watchdog — in the system directory.

test_empty_defaults_example runs through usage bash, like the nine other tests in its file. It was the only one relying on the shebang, and Windows cannot execute a .sh at all — the spawn fails with os error 193 before the shebang is read.

shell_override.rs loses #![cfg(unix)]. Two changes make it portable:

The substitute program is $CARGO instead of /bin/echo, which has no Windows counterpart. Cargo sets the variable for the processes it spawns, so it is the one executable a test can name without knowing the platform, and handed a path it does not recognize it repeats it back — no such subcommand \…`— which is what makes the substitution observable. (Not theusage` binary itself: its top level has a positional argument, so a path lands there and never reaches the error message.)
The three fallback tests compare against a baseline run with nothing overridden, instead of asserting the script's output. Whether the default bash can run the script is a property of the machine, and on the platform this feature exists for it may well not — but "same as unset" is the property those tests are actually about, and it holds either way. The positive case is covered on every platform by the_override_replaces_the_program.

Verified

On a windows-latest runner, with USAGE_SHELL_BASH naming Git Bash and core.autocrlf off: 538 passed, 0 failed, 0 skipped — every shell-dependent test genuinely runs rather than skipping past.

Locally on Windows, with and without the override:

	override set	unset
`complete_word`	34 pass, 0 skip	26 pass, 8 skip
`shell_completions_integration`	19 pass	3 pass, 16 skip
`examples`	10 pass	10 pass
`shell_override`	5 pass	5 pass

Linux: cargo test --all --all-features and cargo clippy --all --all-features --all-targets -- -D warnings unchanged.

Follow-up: the Windows CI job

Deliberately not in this PR, but no longer guesswork — I ran it on the fork to find out what it needs, and all three of these are measured rather than assumed:

core.autocrlf false, set before checkout. git defaults it to true on Windows and the repo has no .gitattributes, so fixtures arrive with CRLF and seven help tests plus two markdown tests fail on an ending nobody can see — the printed diff showed identical text.
USAGE_SHELL_BASH pointed at Git Bash, for the where versus CreateProcess reason above.
The CI panic in skip_if_shell_missing relaxed on Windows. The rule exists because the Linux job installs zsh and fish on purpose; nothing installs them on Windows, so their absence is expected rather than a configuration bug. mise draws the same line — its Windows jobs install no POSIX shells at all. That is a one-line change I left out of this PR, since it is about CI policy rather than about the tests.

Happy to open that as a second PR whenever you want it.

This pull request was generated by Claude Code.

Summary by CodeRabbit

Tests
- Improved cross-platform coverage for shell completions, shell overrides, and example execution.
- Added validation that configured shells can successfully run mounted fixtures.
- Expanded support for Bash, Zsh, Fish, and PowerShell environments, including path and timeout handling.
- Updated unavailable-shell handling: CI reports failures, while non-CI environments skip affected tests.
- Added coverage for shell overrides, fallback behavior, argument passing, and ignored environment variables.

coderabbitai · 2026-08-02T04:24:39Z

Note

Currently processing new changes in this PR. This may take a few minutes, please wait...

⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: 327552c6-020a-457b-b64f-df7b235acc32

📥 Commits

Reviewing files that changed from the base of the PR and between d1a3d02 and 4fe17a5.

📒 Files selected for processing (5)

cli/tests/complete_word.rs
cli/tests/examples.rs
cli/tests/shell_completions_integration.rs
cli/tests/shell_override.rs
lib/tests/sdk_compile.rs

 ____________________________________________________________________
< Never go full rewrite. You don't buy that? Ask Netscape Navigator. >
 --------------------------------------------------------------------
  \
   \   \
        \ /\
        ( )
      .( o ).

📝 Walkthrough

Walkthrough

Shell tests now validate fixture execution, use configurable shell executables, normalize paths across platforms, and expand shell override coverage beyond Unix-only assumptions.

Changes

Cross-platform shell tests

Layer / File(s)	Summary
Shell probing and fixture execution `cli/tests/complete_word.rs`, `cli/tests/examples.rs`	Shell checks now require successful `sh` and mounted-fixture execution. Example tests invoke the compiled binary through Cargo without manual `PATH` setup.
Shell path and invocation migration `cli/tests/shell_completions_integration.rs`	Integration tests use configurable shells, normalized paths, platform-specific `PATH` conversion, and shell-based timeout execution.
Portable shell override coverage `cli/tests/shell_override.rs`	Override tests run on all platforms with a portable substitute program, baseline comparisons, and platform-independent failure assertions.

Estimated code review effort: 4 (Complex) | ~45 minutes

Possibly related PRs

jdx/usage#760: Updates related shell completion integration behavior.
jdx/usage#765: Updates related shell availability and portable shell execution tests.
jdx/usage#767: Updates related shell override coverage.

Poem

A rabbit checks each shell in line,
With paths in POSIX and Windows design.
Fixtures run and overrides stay clear,
Baselines guide each test engineer.
Hop, hop—the portable checks are here!

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main change: making the test suite runnable on Windows.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

greptile-apps · 2026-08-02T04:31:07Z

Greptile Summary

This PR makes the shell-oriented test suite portable to Windows without changing production code.

Probes whether configured shells can execute actual test scripts before running shell-dependent tests.
Routes test shell invocations through USAGE_SHELL_<SHELL> and normalizes Windows paths for each shell environment.
Runs the examples consistently through usage bash.
Enables shell-override tests on Windows using portable executables and baseline comparisons.

Confidence Score: 5/5

The PR appears safe to merge.

No blocking failure remains; the configured-shell probe and every guarded shell invocation now resolve through the same override-aware helper, including the zsh timeout path.

Important Files Changed

Filename	Overview
cli/tests/shell_completions_integration.rs	The configured shell override is now used consistently by probes, integration-test invocations, and the zsh timeout wrapper, resolving the previously reported mismatch.
cli/tests/complete_word.rs	The POSIX-shell guard now validates that an actual mounted fixture can execute before shell-dependent tests proceed.
cli/tests/examples.rs	The empty-defaults example now runs through the built usage binary instead of relying on direct shebang execution.
cli/tests/shell_override.rs	Shell-override coverage is made cross-platform with Cargo as the substitute executable and machine-independent baseline comparisons.

_{Reviews (2): Last reviewed commit: "test(windows): make the suite runnable o..." | Re-trigger Greptile}

coderabbitai

Actionable comments posted: 1

🧹 Nitpick comments (1)

cli/tests/shell_override.rs (1)
79-98: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Make the PATH override observable.

USAGE_SHELL_BASH="bash" is indistinguishable from the fallback. This test passes if the override is ignored. It also passes if PATH resolution is broken but the default shell has the same result.

Prepend the directory that contains substitute_program() to PATH. Set USAGE_SHELL_BASH to that program’s file name. Then assert that its stderr contains SCRIPT and that the fixture output is absent.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@cli/tests/shell_override.rs` around lines 79 - 98, Update
the_override_may_name_a_program_on_path to prepend the directory containing
substitute_program() to PATH and set USAGE_SHELL_BASH to that program’s filename
instead of bash. Assert the run succeeds with stderr containing SCRIPT, and
verify the normal fixture output is absent so the PATH-resolved override is
observable.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@cli/tests/shell_completions_integration.rs`:
- Around line 103-129: Update shell_can_run_a_script so the pwsh branch invokes
the shell with -NoProfile, -NonInteractive, and -File before the script path,
matching test_powershell_completion_integration. Keep the existing invocation
for other shells and preserve the current stdout validation.

---

Nitpick comments:
In `@cli/tests/shell_override.rs`:
- Around line 79-98: Update the_override_may_name_a_program_on_path to prepend
the directory containing substitute_program() to PATH and set USAGE_SHELL_BASH
to that program’s filename instead of bash. Assert the run succeeds with stderr
containing SCRIPT, and verify the normal fixture output is absent so the
PATH-resolved override is observable.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

Push a commit to this branch (recommended)
Create a new PR with the fixes

ℹ️ Review info

⚙️ Run configuration

Configuration used: Central YAML (base), Organization UI (inherited)

Review profile: CHILL

Plan: Pro Plus

Run ID: fa2773fa-e178-4806-87f9-7c7bb26ced9a

📥 Commits

Reviewing files that changed from the base of the PR and between 6cbc931 and 8818c3b.

📒 Files selected for processing (4)

cli/tests/complete_word.rs
cli/tests/examples.rs
cli/tests/shell_completions_integration.rs
cli/tests/shell_override.rs

Dismissed because a newer commit was pushed; Greptile will re-review the current head.

`cargo test --all --all-features` on a Windows machine with WSL gives 15 failures, and `cli/tests/shell_override.rs` never runs at all. None of it is a bug in usage; the tests are. `bash` there is the WSL launcher. The Win32 executable search order puts the system directory ahead of PATH and installing WSL puts `bash.exe` in it, so a bare `bash` cannot open the Windows paths these tests write. GitHub's windows-latest is the same: `where bash` lists Git Bash first, but `Command::new("bash")` still reaches System32, and without an override it exits 1 with "Windows Subsystem for Linux has no installed distributions". Both skip guards missed this by probing a proxy rather than the precondition: `--version` spawns fine under WSL bash, and in `complete_word.rs` the probe asked about `sh` while the mount fixtures need `bash` — on Windows the two need not be the same family. Both now run a script, and a real fixture, and require it to exit cleanly, so an unusable shell skips instead of failing. The fixture is named from `CARGO_MANIFEST_DIR` rather than `fs::canonicalize`, whose `\?\` prefix usage cannot open: probing with one made the guard fail on the shape of the path rather than on the shell, skipping every mount test on a machine where they run. Shell resolution goes through `USAGE_SHELL_<SHELL>`, the variable `shell_program_override` already reads, so one setting covers the whole suite. Every invocation is built by one `script_command`, so the guard and the tests it guards cannot drift apart — not in which program they run, nor in how. pwsh is where they would have: it needs `-File` to take a script path, and `-NoProfile -NonInteractive` so a developer's profile cannot add output to a run whose stdout is compared against a single token. The zsh pty test's `timeout` wrapper goes through the shell too, since spawning it directly walks into the same trap as `bash` — Windows keeps an unrelated `timeout.exe`, a sleep rather than a watchdog, in the system directory. Three places hand a Windows path to something that parses it, and each needed a different answer. Paths interpolated into shell script bodies are normalized to `/`, because `\` is an escape character there and `C:\Users\Jam` was arriving as `C:UsersJam`. `$PATH` entries go through `cygpath` instead: `$PATH` is colon-separated, so `C:/Users/...` splits into `C` and `/Users/...` and neither resolves, and only the shell in use knows whether the answer is `/c/...` or `/cygdrive/c/...`. And `sdk_compile.rs` put the SDK directory inside a Python string literal, where `C:\Users\RUNNER~1\...` fails to parse at all because `\U` opens a unicode escape — that one now travels as argv, clear of any escaping rules. `test_empty_defaults_example` now runs through `usage bash` like the nine other tests in its file. It was the only one relying on the shebang, and Windows cannot execute a `.sh` at all — os error 193, before the shebang is read. `shell_override.rs` loses `#![cfg(unix)]`. Its substitute program is `$CARGO` rather than `/bin/echo`, which has no Windows counterpart: cargo sets the variable for the processes it spawns, and handed a path it does not recognize it repeats it back, which is what makes the substitution observable. The three fallback tests compare against a baseline run with nothing overridden instead of asserting the script's output, since whether the default shell works is a property of the machine, while "same as unset" is what those tests are about. Tests only; no library or CLI source changes. Measured on a windows-latest runner: with `USAGE_SHELL_BASH` naming a real bash and `core.autocrlf` off, the whole suite passes with nothing skipped. Without the override the shell-dependent tests skip rather than fail. Linux is unchanged.

greptile-apps Bot reviewed Aug 2, 2026

View reviewed changes

Comment thread cli/tests/shell_completions_integration.rs Outdated

JamBalaya56562 force-pushed the windows-test-suite branch from 6787fac to 8818c3b Compare August 2, 2026 04:36

greptile-apps Bot previously approved these changes Aug 2, 2026

View reviewed changes

JamBalaya56562 marked this pull request as draft August 2, 2026 04:45

coderabbitai Bot reviewed Aug 2, 2026

View reviewed changes

Comment thread cli/tests/shell_completions_integration.rs

JamBalaya56562 force-pushed the windows-test-suite branch from 8818c3b to bb9c130 Compare August 2, 2026 05:16

This was referenced Aug 2, 2026

refactor(lib)!: mark ParseOutput and UsageErr non_exhaustive #772

Open

ci: EXPERIMENT — what a windows-latest job needs (reference, do not merge) JamBalaya56562/usage#1

Open

JamBalaya56562 force-pushed the windows-test-suite branch from bb9c130 to d1a3d02 Compare August 2, 2026 05:57

JamBalaya56562 force-pushed the windows-test-suite branch from d1a3d02 to 4fe17a5 Compare August 2, 2026 08:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

test(windows): make the suite runnable on Windows - #771

test(windows): make the suite runnable on Windows#771
JamBalaya56562 wants to merge 1 commit into
jdx:mainfrom
JamBalaya56562:windows-test-suite

JamBalaya56562 commented Aug 2, 2026 •

edited

Loading

Uh oh!

coderabbitai Bot commented Aug 2, 2026 •

edited

Loading

Walkthrough

Changes

Uh oh!

greptile-apps Bot commented Aug 2, 2026 •

edited

Loading

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

JamBalaya56562 commented Aug 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What was wrong

bash on Windows is not a POSIX bash

The skip guards were checking the wrong thing

What changed

Verified

Follow-up: the Windows CI job

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Aug 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Changes

Uh oh!

greptile-apps Bot commented Aug 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

Uh oh!

coderabbitai Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

JamBalaya56562 commented Aug 2, 2026 •

edited

Loading

`bash` on Windows is not a POSIX bash

coderabbitai Bot commented Aug 2, 2026 •

edited

Loading

greptile-apps Bot commented Aug 2, 2026 •

edited

Loading