FEA add temperature scaling to CalibratedClassifierCV
#31068
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
virchan
commented
Mar 25, 2025
There was a problem hiding this comment.
A follow-up to my comment on the Array API: I don't think we can support the Array API here, as scipy.optimize.minimize does not appear to support it.
If I missed anything, please let me know—I'd be happy to investigate further.
ogrisel
reviewed
Mar 25, 2025
…enting_temperature_scaling
…fier`. Updated constructor of `_TemperatureScaling` class. Updated `test_temperature_scaling` in `test_calibration.py`. Added `__sklearn_tags__` to `_TemperatureScaling` class.
…enting_temperature_scaling
…enting_temperature_scaling
…enting_temperature_scaling
…enting_temperature_scaling
…enting_temperature_scaling
…Updated doc-strings of temperature scaling in `calibration.py`. Updated formatting.
virchan
commented
Apr 10, 2025
…enting_temperature_scaling
ogrisel
reviewed
Apr 15, 2025
…enting_temperature_scaling
Update `minimize` in `_temperture_scaling` to `minimize.scalar`. Update `test_calibration.py` to check the optimised inverse temperature is between 0.1 and 10.
virchan
commented
Apr 26, 2025
…enting_temperature_scaling
…id `method` parameter.
…enting_temperature_scaling
virchan
commented
Jul 18, 2025
…eter is close to 1.0 when fitted on the training set of LogisticRegression.
virchan
commented
Jul 29, 2025
There was a problem hiding this comment.
I updated the user guide and the docstrings in calibration.py. I also modified the test to check that the temperature parameter is close to 1 when the temperature scaler is fitted on the training set of the LogisticRegression classifier.
There are still some comments that need to be addressed, and I'll work on them later.
…ibratedClassifier.predict_proba` only.
…ature_scaling' into issues/28574_implementing_temperature_scaling # Conflicts: # sklearn/calibration.py
…tedClassifier.predict_proba` only.
virchan
commented
Jul 30, 2025
There was a problem hiding this comment.
I've refactored the part for checking reponse_method_name:
if len(classes) == 2 and predictions.shape[-1] == 1:
response_method_name = _check_response_method(
clf,
["decision_function", "predict_proba"],
).__name__
if response_method_name == "predict_proba":
predictions = np.hstack([1 - predictions, predictions])I think this only needs to be applied in two places: : _fit_calibrator and _CalibratedClassifier.predict_proba. But please let me know if there's a better way to handle this.
I've also moved _temperature_scaling inside _TemperatureScaling.fit.
CI has passed, so it's ready for review!
Comment on lines
+1085
to
+1119
| def _temperature_scaling(predictions, labels, sample_weight=None): | ||
| """Calibrate the temperature of temperature scaling. | ||
|
|
||
| Parameters | ||
| ---------- | ||
| predictions : ndarray of shape (n_samples,) or (n_samples, n_classes) | ||
| The output of `decision_function` or `predict_proba`. If the input | ||
| appears to be probabilities (i.e., values between 0 and 1 that sum to 1 | ||
| across classes), it will be converted to logits using `np.log(p + eps)`. | ||
|
|
||
| Binary decision function outputs (1D) will be converted to two-class | ||
| logits of the form (-x, x). For shapes of the form (n_samples, 1), the | ||
| same process applies. | ||
|
|
||
| labels : ndarray of shape (n_samples,) | ||
| True labels for the samples. | ||
|
|
||
| sample_weight : array-like of shape (n_samples,), default=None | ||
| Sample weights. If None, then samples are equally weighted. | ||
|
|
||
| Returns | ||
| ------- | ||
| beta : float | ||
| The optimised inverse temperature parameter for probability calibration, | ||
| with a value in the range (0, infinity). | ||
|
|
||
| References | ||
| ---------- | ||
| On Calibration of Modern Neural Networks, | ||
| C. Guo, G. Pleiss, Y. Sun, & K. Q. Weinberger, ICML 2017. | ||
| """ | ||
| check_consistent_length(predictions, labels) | ||
| logits = _convert_to_logits( | ||
| predictions | ||
| ) # guarantees np.float64 or np.float32 |
There was a problem hiding this comment.
Suggested change
| def _temperature_scaling(predictions, labels, sample_weight=None): | |
| """Calibrate the temperature of temperature scaling. | |
| Parameters | |
| ---------- | |
| predictions : ndarray of shape (n_samples,) or (n_samples, n_classes) | |
| The output of `decision_function` or `predict_proba`. If the input | |
| appears to be probabilities (i.e., values between 0 and 1 that sum to 1 | |
| across classes), it will be converted to logits using `np.log(p + eps)`. | |
| Binary decision function outputs (1D) will be converted to two-class | |
| logits of the form (-x, x). For shapes of the form (n_samples, 1), the | |
| same process applies. | |
| labels : ndarray of shape (n_samples,) | |
| True labels for the samples. | |
| sample_weight : array-like of shape (n_samples,), default=None | |
| Sample weights. If None, then samples are equally weighted. | |
| Returns | |
| ------- | |
| beta : float | |
| The optimised inverse temperature parameter for probability calibration, | |
| with a value in the range (0, infinity). | |
| References | |
| ---------- | |
| On Calibration of Modern Neural Networks, | |
| C. Guo, G. Pleiss, Y. Sun, & K. Q. Weinberger, ICML 2017. | |
| """ | |
| check_consistent_length(predictions, labels) | |
| logits = _convert_to_logits( | |
| predictions | |
| ) # guarantees np.float64 or np.float32 | |
| X, y = indexable(X, y) # Is this really needed? | |
| predictions, labels = X, y | |
| check_consistent_length(predictions, labels) | |
| logits = _convert_to_logits( | |
| predictions | |
| ) # guarantees np.float64 or np.float32 |
and so on.
So remove the function _temperature_scaling altogether and integrate it in fit.
It will end with
self.beta = np.exp(log_beta_minimizer.x)
return self
Comment on lines
+724
to
+730
| if len(classes) == 2 and predictions.shape[-1] == 1: | ||
| response_method_name = _check_response_method( | ||
| clf, | ||
| ["decision_function", "predict_proba"], | ||
| ).__name__ | ||
| if response_method_name == "predict_proba": | ||
| predictions = np.hstack([1 - predictions, predictions]) |
There was a problem hiding this comment.
Why not put it inside _TemperatureScaling.fit?
Comment on lines
+822
to
+828
| if n_classes == 2 and predictions.shape[-1] == 1: | ||
| response_method_name = _check_response_method( | ||
| self.estimator, | ||
| ["decision_function", "predict_proba"], | ||
| ).__name__ | ||
| if response_method_name == "predict_proba": | ||
| predictions = np.hstack([1 - predictions, predictions]) |
There was a problem hiding this comment.
Why not put it into _TemperatureScaling.predict?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Reference Issues/PRs
Closes #28574
What does this implement/fix? Explain your changes.
This PR adds temperature scaling to scikit-learn's
CalibratedClassifierCV:Temperature scaling can be enabled by setting
method = "temperature"inCalibratedClassifierCV:This method supports both binary and multi-class classification.
Any other comments?
Cc @adrinjalali, @lorentzenchr in advance.