gh-88886: Remove excessive encoding name normalization #137167
Conversation
There was a problem hiding this comment.
You need to update normalize_encodings doc.
@malemburg Which order should the two PRs be merged in, switching it to the C implementation and simplifying the implementation?
|
This mainly restores the status quo prior to bpo-37751 and updates the documentation. But I am planning more changes. I was not sure about replacing spaces with hyphens? Should we kept this here, on leave it to the search function? Should we convert spaces to underscores instead? Or should we remove any transformation? I plan also to change |
The codecs lookup function now performs only minimal normalization of the encoding name before passing it to the search functions: all ASCII letters are converted to lower case, spaces are replaced with hyphens. Excessive normalization broke third-party codecs providers, like python-iconv. Revert "bpo-37751: Fix codecs.lookup() normalization (pythonGH-15092)" This reverts commit 20f59fe.
2df6169 to
ae1cae2
Compare
| @@ -0,0 +1,3 @@ | |||
| The codecs lookup function now performs only minimal normalization of the | |||
There was a problem hiding this comment.
I'd write "The codecs lookup function now again performs only minimal normalization of the encoding name before passing it to the search functions: all ASCII letters are converted to lower case, spaces are replaced with hyphens. This restores the pre-Python 3.9 behavior.".
There was a problem hiding this comment.
Yes, it restores the pre-Python 3.9 behavior. I wonder if we can go further -- do not replace spaces, or replace them with underscores instead of hyphens, or replace also hyphens or underscores. We can handle this in normalize_encoding() or just add aliases. iconv on FreeBSD do not convert any non-letter character. iconv on Linux preserves alphanumerics and "_-.,:/", and removes all others.
Should it still convert to lower case? Most standard encoding names in the IANA registry are in upper case. iconv encoding names are all in upper case (on Linux and FreeBSD). But this is potentially a breaking change without much benefit.
There was a problem hiding this comment.
No, let's not change the behavior again. It's been like this for ages before the Python 3.9 change and we really only want to roll back that change.
The lower casing was added to make sure that encoding names are handled case insensitive. It's rather common to default to lower casing rather then upper casing to do this.
There was a problem hiding this comment.
The docs still need updating: https://docs.python.org/3.15/library/codecs.html#encodings.normalize_encoding
The codecs lookup function now performs only minimal normalization of the encoding name before passing it to the search functions: all ASCII letters are converted to lower case, spaces are replaced with hyphens.
Excessive normalization broke third-party codecs providers, like python-iconv.
Revert "bpo-37751: Fix codecs.lookup() normalization (GH-15092)"
This reverts commit 20f59fe.
📚 Documentation preview 📚: https://cpython-previews--137167.org.readthedocs.build/