FEA add temperature scaling to `CalibratedClassifierCV` #31068

virchan · 2025-03-25T07:57:42Z

Reference Issues/PRs

What does this implement/fix? Explain your changes.

This PR adds temperature scaling to scikit-learn's CalibratedClassifierCV:

Temperature scaling can be enabled by setting method = "temperature" in CalibratedClassifierCV:

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.calibration import CalibratedClassifierCV
from sklearn.svm import LinearSVC

X, y = make_classification(random_state=42)

X_train, X_calib, y_train, y_calib = train_test_split(X, y, random_state=42)

clf = LinearSVC(random_state=42)
clf.fit(X_train, y_train)
cal_clf = CalibratedClassifierCV(clf, method="temperature").fit(X_train, y_train)

This method supports both binary and multi-class classification.

Any other comments?

Cc @adrinjalali, @lorentzenchr in advance.

github-actions · 2025-03-25T07:58:58Z

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: ad2a786. Link to the linter CI: here}

virchan

A follow-up to my comment on the Array API: I don't think we can support the Array API here, as scipy.optimize.minimize does not appear to support it.

If I missed anything, please let me know—I'd be happy to investigate further.

sklearn/calibration.py

sklearn/tests/test_calibration.py

ogrisel

Thanks for the PR. Here is a first pass of feedback:

sklearn/calibration.py

sklearn/tests/test_calibration.py

sklearn/calibration.py

…enting_temperature_scaling

…fier`. Updated constructor of `_TemperatureScaling` class. Updated `test_temperature_scaling` in `test_calibration.py`. Added `__sklearn_tags__` to `_TemperatureScaling` class.

…enting_temperature_scaling

sklearn/calibration.py

…enting_temperature_scaling

…Updated doc-strings of temperature scaling in `calibration.py`. Updated formatting.

virchan

I'm still working on addressing the feedback, but I also wanted to share some findings related to it and provide an update.

sklearn/calibration.py

sklearn/tests/test_calibration.py

…enting_temperature_scaling

lorentzenchr

I few computational things seem off.

sklearn/calibration.py

…enting_temperature_scaling

Update `minimize` in `_temperture_scaling` to `minimize.scalar`. Update `test_calibration.py` to check the optimised inverse temperature is between 0.1 and 10.

virchan

There are some CI failures—I'll fix those shortly.

Also considering adding a verbose parameter to CalibratedClassifierCV to optionally display convergence info when optimising the inverse temperature beta.

sklearn/calibration.py

…enting_temperature_scaling

…id `method` parameter.

…ce in all `ensemble` cases.

…gestions.

…ction.

…enting_temperature_scaling

virchan

CI passed!

sklearn/calibration.py

OmarManzoor

LGTM. Thank you @virchan

sklearn/calibration.py

sklearn/tests/test_calibration.py

doc/modules/calibration.rst

…edback.

…enting_temperature_scaling

virchan

The CI fails when checking that the ROC AUCs are equal up to 7 decimal places. I'll fix it later.

…enting_temperature_scaling

virchan

CI passed!

lorentzenchr

Close to the finish line.

doc/modules/calibration.rst

sklearn/calibration.py

lorentzenchr · 2025-07-28T08:45:54Z

sklearn/calibration.py

+        l = halfmulti_loss.loss(y_true=labels, raw_prediction=raw_prediction)
+
+        return l.sum()


Suggested change

l = halfmulti_loss.loss(y_true=labels, raw_prediction=raw_prediction)

return l.sum()

return halfmulti_loss(y_true=labels, raw_prediction=raw_prediction)

The call method returns the (weighted) average loss.

lorentzenchr · 2025-07-28T08:57:21Z

sklearn/calibration.py

+    return logits
+
+
+def _temperature_scaling(predictions, labels, sample_weight=None):


This is only called in _TemperatureScaling.fit. So why not place the code there instead of this separate function?

lorentzenchr · 2025-07-28T09:02:49Z

sklearn/tests/test_calibration.py

@@ -359,7 +377,7 @@ def test_calibration_prefit(csr_container):
                )


-@pytest.mark.parametrize("method", ["sigmoid", "isotonic"])
+@pytest.mark.parametrize("method", ["sigmoid", "isotonic", "temperature"])


Suggested change

@pytest.mark.parametrize("method", ["sigmoid", "isotonic", "temperature"])

@pytest.mark.parametrize(

["method", "calibrator"],

[

("sigmoid", _SigmoidCalibration()),

("isotonic", IsotonicRegression(out_of_bounds="clip")),

("temperature", _TemperatureScaling())

]

)

and then remove the if-else inside.

sklearn/tests/test_calibration.py

…eter is close to 1.0 when fitted on the training set of LogisticRegression.

virchan

I updated the user guide and the docstrings in calibration.py. I also modified the test to check that the temperature parameter is close to 1 when the temperature scaler is fitted on the training set of the LogisticRegression classifier.

There are still some comments that need to be addressed, and I'll work on them later.

virchan · 2025-07-29T08:04:16Z

sklearn/tests/test_calibration.py

+        y_scores_train = clf.predict_proba(X_train)
+        ts = _TemperatureScaling().fit(y_scores_train, y_train)
+        assert_allclose(ts.beta_, 1.0, atol=1.1e-7, rtol=0)


For logistic regression, we could even check that temperature is 1 if fitted on the training set.

Let me know if atol or rtol needs to be adjusted here.

FEA add temperature scaling to CalibratedClassifierCV

604e0da

added changelog

257fd03

virchan added the New Feature label Mar 25, 2025

virchan commented Mar 25, 2025

View reviewed changes

sklearn/calibration.py Outdated Show resolved Hide resolved

sklearn/tests/test_calibration.py Outdated Show resolved Hide resolved

virchan added the module:calibration label Mar 25, 2025

updated docstring.

eb6dd4a

virchan marked this pull request as ready for review March 25, 2025 10:55

ogrisel reviewed Mar 25, 2025

View reviewed changes

virchan added 4 commits March 27, 2025 18:14

Merge remote-tracking branch 'upstream/main' into issues/28574_implem…

73d0335

…enting_temperature_scaling

Updated docstrings of CalibratedClassifierCV and `_CalibratedClassi…

6d6963f

…fier`. Updated constructor of `_TemperatureScaling` class. Updated `test_temperature_scaling` in `test_calibration.py`. Added `__sklearn_tags__` to `_TemperatureScaling` class.

Fix typo in _TemperatureScaling's fit method.

c8ffc1b

Merge remote-tracking branch 'upstream/main' into issues/28574_implem…

5a09af3

…enting_temperature_scaling

lorentzenchr reviewed Mar 31, 2025

View reviewed changes

sklearn/calibration.py Outdated Show resolved Hide resolved

sklearn/calibration.py Show resolved Hide resolved

sklearn/calibration.py Outdated Show resolved Hide resolved

sklearn/calibration.py Show resolved Hide resolved

sklearn/calibration.py Outdated Show resolved Hide resolved

virchan added 5 commits March 31, 2025 15:11

Merge remote-tracking branch 'upstream/main' into issues/28574_implem…

a1be098

…enting_temperature_scaling

Merge remote-tracking branch 'upstream/main' into issues/28574_implem…

d715e6e

…enting_temperature_scaling

Merge remote-tracking branch 'upstream/main' into issues/28574_implem…

4ce1452

…enting_temperature_scaling

Merge remote-tracking branch 'upstream/main' into issues/28574_implem…

b039a5a

…enting_temperature_scaling

Updated test cases for temperature scaling in test_calibration.py. …

93c7972

…Updated doc-strings of temperature scaling in `calibration.py`. Updated formatting.

virchan commented Apr 10, 2025

View reviewed changes

sklearn/calibration.py Outdated Show resolved Hide resolved

sklearn/calibration.py Outdated Show resolved Hide resolved

sklearn/tests/test_calibration.py Outdated Show resolved Hide resolved

virchan added 2 commits April 14, 2025 15:52

Merge remote-tracking branch 'upstream/main' into issues/28574_implem…

7d50ea7

…enting_temperature_scaling

Fix failing test_float32_predict_proba.

dfcaa39

lorentzenchr reviewed Apr 15, 2025

View reviewed changes

sklearn/calibration.py Outdated Show resolved Hide resolved

sklearn/calibration.py Outdated Show resolved Hide resolved

sklearn/calibration.py Outdated Show resolved Hide resolved

ogrisel reviewed Apr 15, 2025

View reviewed changes

sklearn/calibration.py Outdated Show resolved Hide resolved

sklearn/calibration.py Outdated Show resolved Hide resolved

virchan added 2 commits April 25, 2025 22:16

Merge remote-tracking branch 'upstream/main' into issues/28574_implem…

24c1266

…enting_temperature_scaling

Update HalfMultinomialLoss docstring.

ad8dea5

Update `minimize` in `_temperture_scaling` to `minimize.scalar`. Update `test_calibration.py` to check the optimised inverse temperature is between 0.1 and 10.

virchan commented Apr 26, 2025

View reviewed changes

sklearn/calibration.py Outdated Show resolved Hide resolved

sklearn/calibration.py Outdated Show resolved Hide resolved

virchan added 3 commits April 30, 2025 19:36

Update _TemperatureScaling tags.

9f1626c

Merge remote-tracking branch 'upstream/main' into issues/28574_implem…

b4f1aad

…enting_temperature_scaling

Add test_calibration_method in test_calibration.py to check inval…

1a9e307

…id `method` parameter.

virchan added 8 commits June 18, 2025 10:39

update test_temperature_scaling function to check accuracy invarian…

7f39877

…ce in all `ensemble` cases.

update CalibratedClassifierCV docstrings by applying reviewer's sug…

b08a228

…gestions.

add check_is_fitted to the test_temperature_scaling function.

060f240

add get_tags to the test_temperature_scaling_input_validation fun…

283f4a4

…ction.

Merge remote-tracking branch 'upstream/main' into issues/28574_implem…

cbc0af6

…enting_temperature_scaling

update the test_temperature_scaling function to fix CI failure.

c0caeb6

update test_calibration.py to trigger CI re-run.

3cc0161

update test_calibration.py to trigger CI re-run.

b0584d6

virchan commented Jun 18, 2025

View reviewed changes

sklearn/calibration.py Outdated Show resolved Hide resolved

OmarManzoor approved these changes Jun 19, 2025

View reviewed changes

sklearn/calibration.py Outdated Show resolved Hide resolved

lorentzenchr reviewed Jul 17, 2025

View reviewed changes

virchan added 5 commits July 17, 2025 20:26

update calibration.rst by applying reviewer's feedback.

f968257

update _TemperatureScaling.predict method by applying reviewer's fe…

55a330d

…edback.

update test_temperature_scaling by applying code-review suggestions.

b23b256

Merge remote-tracking branch 'upstream/main' into issues/28574_implem…

9a3eda7

…enting_temperature_scaling

update test_temperature_scaling.

40c5601

virchan commented Jul 18, 2025

View reviewed changes

virchan added 3 commits July 18, 2025 14:54

removed error-raisings in private functions

662414e

Merge remote-tracking branch 'upstream/main' into issues/28574_implem…

51a9e26

…enting_temperature_scaling

updated ROC-AUC checking in the test_temperature_Scaling function

903930c

virchan commented Jul 18, 2025

View reviewed changes

lorentzenchr reviewed Jul 28, 2025

View reviewed changes

virchan added 7 commits July 28, 2025 17:58

Update calibration.rst by applying review suggestions.

eba8821

Update calibration.py docstrings by applying review suggestions.

41ac95d

Update _convert_to_logits by tightening eps.

9a7b787

Update test_temperature_scaling to check that the temperature param…

7f68ac2

…eter is close to 1.0 when fitted on the training set of LogisticRegression.

Merge branch 'main' into issues/28574_implementing_temperature_scaling

a3734c6

Update calibration.py

b48a4c8

Update _temperature_scaling docstrings.

ad2a786

virchan commented Jul 29, 2025

View reviewed changes

		l = halfmulti_loss.loss(y_true=labels, raw_prediction=raw_prediction)

		return l.sum()

		return logits


		def _temperature_scaling(predictions, labels, sample_weight=None):

-@pytest.mark.parametrize("method", ["sigmoid", "isotonic", "temperature"])
+@pytest.mark.parametrize(
+    ["method", "calibrator"],
+    [
+        ("sigmoid", _SigmoidCalibration()),
+        ("isotonic", IsotonicRegression(out_of_bounds="clip")),
+        ("temperature", _TemperatureScaling())
+    ]
+)

Uh oh!

FEA add temperature scaling to CalibratedClassifierCV #31068

Are you sure you want to change the base?

FEA add temperature scaling to CalibratedClassifierCV #31068

Conversation

virchan commented Mar 25, 2025

Reference Issues/PRs

What does this implement/fix? Explain your changes.

Any other comments?

Uh oh!

github-actions bot commented Mar 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✔️ Linting Passed

Uh oh!

virchan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ogrisel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

virchan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

lorentzenchr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

virchan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

virchan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

OmarManzoor left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

virchan left a comment

Choose a reason for hiding this comment

Uh oh!

virchan left a comment

Choose a reason for hiding this comment

Uh oh!

lorentzenchr left a comment

FEA add temperature scaling to `CalibratedClassifierCV` #31068

FEA add temperature scaling to `CalibratedClassifierCV` #31068

github-actions bot commented Mar 25, 2025 •

edited

Loading

lorentzenchr Jul 28, 2025 •

edited

Loading