As discussed. I'm not sure where this should ultimately go. There was a suggestion we should write a PEP. I'm making this github issue since other dev's will be able to edit it.

TLDR: How do I opt-out of this new behaviour?

A simple way is to set the following environment variable PYTHONINTMAXSTRDIGITS=0. From within Python, call sys.set_int_max_str_digits(0).

Why is considered a security bug?

Or, certain computing operations can take a long time, depending on the size of input data. Why is this specific issue considered a security bug?

It is quite common for Python code implementing network protocols and data serialization to do int(untrusted_string_or_bytes_value) on input to get a numeric value, without having limited the input length or to do log("processing thing id %s", unknowingly_huge_integer) or any similar concept to convert an int to a string without first checking its magnitude. (http, json, xmlrpc, logging, loading large values into integer via linear-time conversions such as hexadecimal stored in yaml, or anything computing larger values based on user controlled inputs… which then wind up attempting to output as decimal later on). All of these can suffer a CPU consuming DoS in the face of untrusted data.

The size of input data required to trigger long processing times is large but not larger than what could be allowed for common network software and services. e.g. a few tens of megabytes of input data could cause multiple seconds of processing time. For an operation that would normally complete on the order of milli or micro seconds, that can effectively be a DoS of the service.

Why was it required to be changed in a bugfix release?

Auditing all existing Python code for this problem, adding length guards, and maintaining that practice everywhere is not feasible nor is it what we deem the vast majority of our users want to do.

For the set of users and software that require the old behaviour of unlimited digits, it can be enabled by a global system environment variable (PYTHONINTMAXSTRDIGITS=0) or by calling sys.set_int_max_str_digits(0).

Why not instead of changing int(), add limited_int()?

As above, changing all existing code to use limited_int() would be a huge task and something we deem the vast majority of our users don't want to do. Also, some mitigation would needed for the int-to-str case.

Why choose 4300 as default limit?

It was chosen as a limit that's high enough to allow commonly used libraries to work correctly and low enough that even relatively weak CPUs will not be vulnerable to a DoS attack. It is fairly simple to increase the limit with a global environment setting (PYTHONINTMAXSTRDIGITS=400000), e.g. if you have a fast CPU.

Put another way, the limit was chosen to limit the global amount of pain caused. The team saw no way to satisfy all of the different users of Python, so a secure default was chosen and only less common code that actually wants to deal with huge integers in decimal needs to do something different.

That limit seems too low, why not something higher?

Any limit is likely going to break some code somewhere. It expected there is a "long tail" distribution in effect and so a limit of 10x the current would only allow slightly more code to work. It is expected that a vast majority of code will work fine with the default limit. For the code that isn't fine, it is better to let the limit be disabled or set to something that's appropriate for that usage. In that case, the limit is unlikely to be suitable as an out-of-the-box default.

Why is str(integer) being limited?

In the Python versions affected by the bugfix, int-to-str also takes O(n^2) where n is the number of digits (if converting to non-power-of-2 base, like base 10). That case is likely a bit harder to exploit but as in the log('%d' % some_number) example, is possible. Fixing the int-to-str operation is harder because there are quite a few more places it happens, e.g. % format, format() function, f-strings and more. For most of these operations, there is no keyword parameter that could be added.

Why global interpreter setting rather than something else?

First, a global interpreter setting was the least complex to implement and causes the least risk when back-porting to older Python versions. Second, for most cases, is expected that fine-grained control of the limit is not required. Either the default global limit is okay or a new global limit would be set. It is possible that new versions of Python, like 3.12 will have ways to set the limit at a finer level (e.g. thread local variable or context manager).

Why not keyword parameter of int?

Having a keyword that defaults to the "safe" or limited mode would be an option but there is no convenient keyword that could be used for the int-to-str case. So, the global interpreter setting is the simple approach.

Can’t we just fix the algorithms to be faster?

Implementing sub-quadratic time algorithms for decimal int-to-str and str-to-int is possible. However, it's not something practical to put into a bugfix release. Some work is being done for better algorithms in Python 3.12.

In general, the Python maintainers would prefer to keep source code from being too hard to maintain due to complex algorithms. For users who want fast operation on large integers, using something like GMP (via the gmpy2 module) would be preferred. The GMP library contains state-of-the-art algorithms and is optimized with CPU specific assembly code. Python is never going to complete with that for speed.

Can’t we just fix the software calling int() and formatting long integers?

Sanitation and validation of untrusted input data is always a good idea. However, calling int(untrusted_string_or_bytes_value) or print(f{"got: {unknowingly_huge_integer}") is very common. The amount of code that would need to be fixed is vast and that is unlikely to happen, at least on any reasonable time scale.

Why wasn't it discussed publicly before making releases?

The Python Security Response Team (PSRT) discussed the issue internally, including with the steering council, and determined that the risk of exploitation would increase if they were to disclose it without a fix, and that there was no reasonable mitigation available without a patch. As the patch was going to be fairly invasive (multiple files and user-visible APIs), it could not have been easily applied by builders, and so security releases were scheduled for a few days after disclosure. The initial change leaves the user in full control of the conversion limit, or else massive pressure would have been applied to every individual library developer to patch their own library for security. Future development can properly enable libraries to manage their own exceptions to the user preference, though we hope that libraries will largely respect their users' wishes.

Is the Decimal module affected by this change?

The Decimal type has not been changed to use the int_max_str_digits limit. Since the Decimal type stores numbers in base 10, it is less common to run into the quadratic time case. decimal_str to decimal.Decimal conversion are linear-time (in both directions, because of common bases). OTOH, int <-> Decimal conversions have the same O() behavior as int <-> str.

Why was there such a delay between the initial report and the bug fix?

The PSRT admits that the delay between report and fix is undesirable and is their own fault. The security team is made up of volunteers, their availability isn't always reliable, and there's nobody "in charge" to coordinate work. Process improvements are being discussed. However, they did agree that the potential for exploitation is high enough that they didn't want to disclose the issue without a fix available and ready for use.