[ENH] Support custom "deep_equals" for testing specific objects types

[felipeangelimvieira]

I'm currently developing a library (Prophetverse), which provides an sktime interface to run the timeseries models. The code is heavily based on numpyro/jax, and as I was using check_estimatorfrom sktime.utils.estimator_check, I had some issues with deep_equals functionality in skbase. (suggestion related to #323 )

The estimator has some attributes which are dictionaries with string keys and jax.numpy.ndarray values. This creates some problems with the current implementation of deep_equals, which is not robust to arbritrary attribute types. The np.any(_coerce_list(x != y)) throws error ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all().

Example of code to reproduce the error (needs jax dependency installed):

from sktime.forecasting.naive import NaiveForecaster as _NaiveForecaster
import jax.numpy as jnp

class NaiveForecaster(_NaiveForecaster):

    def _fit(self, y, X, fh):
        self.some_dict_test_ = {
            "a" : jnp.array([1,2,3]).reshape((-1,1))
        }
        return super()._fit(y, X, fh)

from sktime.utils.estimator_checks import check_estimator

check_estimator(NaiveForecaster, raise_exceptions=True)

Describe the solution you'd like
As the sktime interface starts to be used by third-party libraries, it would be interesting to be able to define custom equality verifications. I think it isn't possible to handle all those cases in skbase - some libraries/objects are specific to those third-party implementations.

Describe alternatives you've considered

One option would be adding a pluggy hook to allow third-party packages to define custom hook implementations to handle specific use-cases.

[fkiraly]

Agreed, that is a good idea!

Recently we have designed this to be possible on the framework level, i.e., for framework libraries, through the plugin system.

However, it is not convenient (or possible?) for third-party libraries compliant with a given framework.

It seems to be difficult to work out where the hooks exactly would have to go, but it does appear feasible, perhaps using pluggy.

I'll have a look if I see a direct route here!

[Aymuos22]

Hi @fkiraly
I’d like to work on this issue.

I’ve reviewed the discussion and the related references (#323, #477), and I’m familiar with the deep_equals limitations regarding JAX arrays. I can help implement a plugin-based or hook-based solution to allow custom equality handling for third-party estimator attributes.

Could you please assign this issue to me?

Thanks!