Skip to content

gh-151316: Prefer UTF-8 for glibc UTF-8-only locale aliases - #155549

Closed
sankalpsthakur wants to merge 3 commits into
python:mainfrom
sankalpsthakur:fix/151316-locale-encoding
Closed

gh-151316: Prefer UTF-8 for glibc UTF-8-only locale aliases#155549
sankalpsthakur wants to merge 3 commits into
python:mainfrom
sankalpsthakur:fix/151316-locale-encoding

Conversation

@sankalpsthakur

@sankalpsthakur sankalpsthakur commented Aug 11, 2026

Copy link
Copy Markdown

Summary

  • Bare locale aliases that are UTF-8-only in glibc SUPPORTED were still mapped to obsolete X11 codesets because makelocalealias applies X11 locale.alias after glibc.
  • On modern systems those locales have no legacy encoding, so getlocale() invented the X11 codeset and setlocale(getlocale()) failed (reported for en_IN).
  • Sweep all such bare aliases to prefer UTF-8, restore glibc UTF-8 defaults in Tools/i18n/makelocalealias.py after the X11 merge (plus a de_li carve-out, which has no bare SUPPORTED line), and cover the replacements with TestHardcodedLocaleReplacements.

Validation

  • Checked in-tree locale_alias entries and normalize/_parse_localename expectations for the 19 restored bare aliases.
  • Explicit legacy encodings (for example hi_IN.ISCII-DEV, az_AZ.ISO8859-9E) still normalize unchanged.
  • Full ./configure && make + ./python.exe -m test test_locale was not run here (sparse/partial clone); relying on CI for the built interpreter suite.

Fixes #151316

AI/LLM disclosure

  • AI coding tools (including Grok and/or Codex agent-assisted editing) were used to help draft or modify code and this PR description.
  • I reviewed the complete change, understand the reasoning, and ran the reported local tests before submitting.
  • This submission is original work of authorship under the project CLA / contributor terms; AI output was not pasted unreviewed.

@python-cla-bot

Copy link
Copy Markdown

The following commit authors need to sign the Contributor License Agreement:

CLA not signed

@StanFromIreland

Copy link
Copy Markdown
Member

You'll have to sign the CLA before we can review.

@StanFromIreland
StanFromIreland marked this pull request as draft August 11, 2026 10:28
@sankalpsthakur
sankalpsthakur force-pushed the fix/151316-locale-encoding branch 2 times, most recently from e0bc095 to ffd9b41 Compare August 11, 2026 10:53
X11 locale.alias mapped en_IN to ISO8859-1, so getlocale() invented a
codeset that does not exist on modern UTF-8-only systems and broke
setlocale(getlocale()) round-trips. Prefer en_IN.UTF-8 (mirroring other
modernized aliases) and cover the round-trip in tests.

Signed-off-by: Sankalp Thakur <sankalphimself@gmail.com>
@sankalpsthakur
sankalpsthakur force-pushed the fix/151316-locale-encoding branch from ffd9b41 to b959df9 Compare August 11, 2026 10:53

@picnixz picnixz left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it the only locale that could have this issue? I would prefer we sweep them all to check if this is the case first instead of having commits for each locale. Also, I think we should instead have a test class that checks for hardcoded replacements thatwe introduce instead of just the case of en_IN.

Comment thread Lib/test/test_locale.py Outdated
@bedevere-app

bedevere-app Bot commented Aug 11, 2026

Copy link
Copy Markdown

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

Sweep bare locales that are UTF-8-only in glibc SUPPORTED but still
mapped to obsolete X11 codesets, restore those defaults in
makelocalealias, and test the hardcoded replacements as a class.

Co-authored-by: Cursor <cursoragent@cursor.com>
@sankalpsthakur sankalpsthakur changed the title gh-151316: Prefer UTF-8 for the en_IN locale alias gh-151316: Prefer UTF-8 for glibc UTF-8-only locale aliases Aug 12, 2026
@sankalpsthakur

Copy link
Copy Markdown
Author

Swept the glibc UTF-8-only locale aliases (not just en_IN), added a test class for the hardcoded replacements, and dropped the irrelevant comment. Still need to finish CLA signing for sankalphimself@gmail.com before this can leave draft.

@sankalpsthakur

Copy link
Copy Markdown
Author

Thanks for the review.

I swept glibc SUPPORTED against X11 locale.alias for bare locales that are UTF-8-only in glibc but still mapped to obsolete X11 codesets. The alias table and makelocalealias now restore those UTF-8 defaults in one pass (19 bare aliases, including en_IN), instead of fixing locales one by one.

TestEnINLocale is replaced by TestHardcodedLocaleReplacements, which asserts the hardcoded/restored alias entries, normalize, and _parse_localename for that full set. Also removed the irrelevant comment on the old round-trip test.

I have made the requested changes; please review again.

@bedevere-app

bedevere-app Bot commented Aug 12, 2026

Copy link
Copy Markdown

Thanks for making the requested changes!

@picnixz: please review the changes made to this pull request.

@bedevere-app
bedevere-app Bot requested a review from picnixz August 12, 2026 08:32
Comment thread Lib/test/test_locale.py Outdated
Comment thread Lib/test/test_locale.py Outdated
Comment thread Tools/i18n/makelocalealias.py Outdated
Use support.subTests for the hardcoded replacements, shorten the test
docstring with an issue link, and extract apply_x11_locales_patch() so
the X11 override of glibc UTF-8 defaults is explicit.

Co-authored-by: Cursor <cursoragent@cursor.com>
@sankalpsthakur

Copy link
Copy Markdown
Author

Addressed the three new notes: shorter docstring with the issue link, @support.subTests for the replacements, and apply_x11_locales_patch() with a short comment on why X11 gets overridden. CLA for sankalphimself@gmail.com is still outstanding on my side.

@picnixz

picnixz commented Aug 12, 2026

Copy link
Copy Markdown
Member

A fully-agentic PR is not accepted per our policy. AI tools can be used, but you're not meant to let them run in automode. If this happens again, we will restrict your access to our repositories.

@picnixz picnixz closed this Aug 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

locale.getlocale() returns wrong encoding if the locale name doesn't have one

3 participants