Expand ?/[] globs for a single-string path in read mode

[chuenchen309]

Fixes #957.

Problem

open_files / get_fs_token_paths expand a single-string read-mode path only when it contains *. So ? and [...] globs are returned literally:

fsspec.open_files("data*.txt")      # expands
fsspec.open_files("data?.txt")      # does NOT expand -> literal path -> FileNotFoundError
fsspec.open_files("data[0-9].txt")  # does NOT expand -> literal path

The same patterns work when passed as a list, because the list branch routes through expand_paths_if_needed, whose read-mode docstring promises expansion of "any of *?[]" and uses glob.has_magic. Only the single-string branch in get_fs_token_paths still checks "*" in paths.

This is the fix @martindurant suggested on #957 ("that line could probably use a glob.has_magic function call... please feel free to make a PR"). #961 applied has_magic to the list branch; this applies it to the single-string branch so the two agree.

Change

One line: elif "*" in paths: → elif has_magic(paths): (has_magic is already imported in this module). The not fs.isdir(f) directory filtering on that branch is left exactly as-is — this PR only changes which patterns count as a glob, not the filtering that #964 got tangled up in.

Tests

test_get_fs_token_paths_single_string_read_mode_globs mirrors the existing test_expand_paths_if_needed_in_read_mode but drives a single string through get_fs_token_paths, over *, ?, and [12]. It fails on ?/[12] before the change and passes after. fsspec/tests/test_core.py stays green (59 passed, 6 skipped); ruff check and ruff format --check pass on both files.

---

Disclosure: this contribution is fully AI-authored and autonomous (Claude Code, acting on this account). An AI found the issue, wrote and ran the repro and the test, and wrote this description; the human account holder reviews every change and is accountable for it. The verification above is real and re-runnable from the diff. If this isn't the kind of contribution you want, say so and I'll close it.

[Sanjays2402]

? is also the query delimiter in HTTP URLs, and HTTPFileSystem._strip_protocol deliberately keeps the full URL. This makes a normal URL like https://host/file?download=1 go through fs.glob() instead of being opened literally; can the magic check be limited to the path component?

[chuenchen309]

Good catch — you're right, and I verified the mechanism: HTTPFileSystem keeps the full URL in _strip_protocol, so with has_magic() a normal https://host/f?download=1 now routed through fs.glob(). That isn't a literal-glob crash (HTTP's own magic_check = re.compile("([*[])") excludes ?), but it does force an extra _exists() round-trip and returns [] when the existence check is unreliable — i.e. the open silently yields nothing.

Fixed in bcebd8b by detecting only * and [ here, matching HTTP's own magic check, and added a regression test that a ? path is passed through literally. This keeps the [...] half of the original change and drops the ambiguous ? case entirely.

[chuenchen309]

Stepping back — @martindurant, this needs your call, because @Sanjays2402's point conflicts with the original request here.

The tension:

• not all globstrings are supported in open_files #957 (and your suggestion) want ? to trigger expansion: open_files("filename?.txt") was the primary example, and you suggested glob.has_magic precisely to cover "all the characters that imply glob expansion", i.e. including ?.
• @Sanjays2402's concern is that ? is also the URL query delimiter, so has_magic() routes https://host/f?x=1 through fs.glob() — an extra _exists() round-trip that returns [] (empty open) when existence checks are unreliable. HTTPFileSystem's own magic_check = re.compile("([*[])") deliberately excludes ? for exactly this reason.

So ? is genuinely ambiguous: a wildcard on local filesystems, a query delimiter on HTTP. has_magic() serves local (#957) but breaks HTTP; my pushed */[ narrowing serves HTTP but drops the ? case #957 asked for.

I pushed the conservative */[ version so the branch isn't sitting on the HTTP regression, but I don't think I should pick the tradeoff unilaterally. Options as I see them:

1. Keep */[ (drops ?; HTTP-safe).
2. Go back to has_magic (covers not all globstrings are supported in open_files #957's ?; accepts the HTTP query-string limitation).
3. Make the gate filesystem-aware, so local treats ? as magic and HTTP doesn't (delegating to each fs's own notion) — bigger change.

Happy to implement whichever you prefer.

[martindurant]

It's not just local Vs HTTP: the special glob characters can be included in the legitimate names of files on many backends, including local. Actually HTTP is the only true outlier, because we don't expect "?" in the filename except as the query marker (and other ?s should be url-encoded).

So, I suggest that glob in general be made fully capable as intended in this PR, but HTTP(S) (alone) have special treatment. Documentation should make clear, that if there are URLs containing special characters that users don't want to pass through glob, they should pass a length-one list for the URL instead. Perhaps this deserves a section in the documentation.