[MNT] Regression on `forecaster.clone()`

[CloseChoice]

Describe the maintenance issue
With the change in this PR, the clone plugins were changed effectively from None (at least in the case I discovered, see script below for reproducibility) to [_PretrainedCloner] see here.

This causes issues with skbase HEAD, but is fixed in skbase main, specifically by this commit 4a79942. As can be seen in that commit, a test was properly added in skbase, nethertheless the dependency of sktime main to skbase main is not clearly stated and rather implicit. If one needs to act upon that is another question or if the implicit knowledge is enough.

Here is the script for reproduction I used:

import numpy as np

import sktime
from sktime.datasets import load_airline
from sktime.forecasting.exp_smoothing import ExponentialSmoothing
from sktime.forecasting.model_evaluation import evaluate
from sktime.performance_metrics.forecasting import MeanAbsolutePercentageError
from sktime.split import ExpandingWindowSplitter
import pandas as pd

y = load_airline()
fh = np.arange(1, 25)

ets = ExponentialSmoothing(trend="add", seasonal="mul", sp=12)
cv = ExpandingWindowSplitter(fh=fh, initial_window=24, step_length=12)

results = evaluate(
    forecaster=ets,
    cv=cv,
    y=y,
    scoring=MeanAbsolutePercentageError(),
    return_data=True,
)

print(results[["test_MeanAbsolutePercentageError", "cutoff"]])

# On main, ExponentialSmoothing fits fail silently (cutoff=NaT),
# so y_pred at split 5 are NA instead of pd.Series.
# This causes plot_series (and any downstream use) to crash.
y_test = results.loc[5, "y_test"]
y_pred = results.loc[5, "y_pred"]


assert isinstance(y_test, pd.Series), f"BUG: y_test should be a pd.Series, got {type(y_test)}: {y_test!r}"
assert isinstance(y_pred, pd.Series), f"BUG: y_pred should be a pd.Series, got {type(y_pred)}: {y_pred!r}"

sktime main + skbase 0.13.1 result in a silent error that cause y_pred to be NA, while with skbase main we get

ipdb> y_test
Period
1956-01    284.0
1956-02    277.0
1956-03    317.0
1956-04    313.0
1956-05    318.0
1956-06    374.0
1956-07    413.0
1956-08    405.0
1956-09    355.0
1956-10    306.0
1956-11    271.0
1956-12    306.0
1957-01    315.0
1957-02    301.0
1957-03    356.0
1957-04    348.0
1957-05    355.0
1957-06    422.0
1957-07    465.0
1957-08    467.0
1957-09    404.0
1957-10    347.0
1957-11    305.0
1957-12    336.0
Freq: M, Name: Number of airline passengers, dtype: float64

Some more details on the incompatibility: there is an error thrown and caught when executing the script, specifically on forecaster.clone(), the error is something like list does not have attribute clone since we append a list to a list resulting in [_PretrainedCloner, [list of other cloners]]. The error is caught and default values assigned subsequently.

cc @SimonBlanke @fkiraly

[fkiraly]

can you please report your exact versions, @CloseChoice? Especially of sktime and scikit-base (python package names), you can use the show_versions tool in sktime.

[CloseChoice]

I need to correct myself. So the error does not happen with these versions:

>>> from sktime import show_versions; show_versions()

System:
    python: 3.13.0 (main, Oct 16 2024, 03:23:02) [Clang 18.1.8 ]
executable: /home/tobias/programming/talks/vortrag_lidl/forecasting-course/.venv/bin/python
   machine: Linux-6.17.0-19-generic-x86_64-with-glibc2.39

Python dependencies:
          pip: None
       sktime: 0.40.1
      sklearn: 1.6.1
       skbase: 0.13.1
        numpy: 2.2.6
        scipy: 1.17.1
       pandas: 2.2.3
   matplotlib: 3.10.8
       joblib: 1.4.2
huggingface-hub: None
        numba: 0.64.0
     pmdarima: None
pytorch-forecasting: None
   skforecast: None
        skpro: 2.12.0
statsforecast: None
  statsmodels: 0.14.6
 transformers: None
      tsfresh: None
      tslearn: None
        torch: None
   tensorflow: None

but happens with

>>> from sktime import show_versions; show_versions()

System:
    python: 3.13.0 (main, Oct 16 2024, 03:23:02) [Clang 18.1.8 ]
executable: /home/tobias/programming/talks/vortrag_lidl/forecasting-course/.venv/bin/python
   machine: Linux-6.17.0-19-generic-x86_64-with-glibc2.39

Python dependencies:
          pip: None
       sktime: 0.40.1
      sklearn: 1.6.1
       skbase: 0.13.0
        numpy: 2.2.6
        scipy: 1.17.1
       pandas: 2.2.3
   matplotlib: 3.10.8
       joblib: 1.4.2
huggingface-hub: None
        numba: 0.64.0
     pmdarima: None
pytorch-forecasting: None
   skforecast: None
        skpro: 2.12.0
statsforecast: None
  statsmodels: 0.14.6
 transformers: None
      tsfresh: None
      tslearn: None
        torch: None
   tensorflow: None

My problem was that first installed sktime with a previous version of skbase, then switched to the latest sktime and the skbase version did not upgrade, I would suggest to update this range: https://github.com/sktime/sktime/blob/main/pyproject.toml#L59.

[fkiraly]

ok, makes sense. This means that sktime 0.40.1 should basically had a lower bound to the newest scikit-base 0.13.1.
Which will be added from sktime 1.0 onwards (soon to be released).

It is unpleasant that we did not manage to build in downwards compatibility somehow, but I suppose it is to be expected for a major API addition. We could have delayed it to 1.0, but I would say, the 1.0 release will remedy the issue as people migrate naturally to higher scikit-base versions.

[fkiraly]

Also, phew. (not as bad as it originally looked like)

[SimonBlanke]

This means that sktime 0.40.1 should basically had a lower bound to the newest scikit-base 0.13.1.
Agreed.

From what I can tell the fix works as expected. How should we proceed with this issue?

[CloseChoice]

I suggest simply merging sktime/sktime#9736 and closing this issue.

ok, makes sense. This means that sktime 0.40.1 should basically had a lower bound to the newest scikit-base 0.13.1. Which will be added from sktime 1.0 onwards (soon to be released).

It is unpleasant that we did not manage to build in downwards compatibility somehow, but I suppose it is to be expected for a major API addition. We could have delayed it to 1.0, but I would say, the 1.0 release will remedy the issue as people migrate naturally to higher scikit-base versions.

Well, just the combination of skbase and sktime broke for a specific version combination (which I assume is rare). No released versions were broken, so I won't think this isn't a big thing. I also don't recognise what's the value of keeping skbase backwards compatible to sktime over multiple versions, but I don't have the full picture here.