[Bug]: get_metrics fails for ts_forecasting models due to inconsistent input sample sizes (1 vs 30) #1366
Description
Description
When attempting to retrieve metrics for a time series forecasting model using get_metrics(), an error occurs due to inconsistent input sample sizes between the model's predictions and validation data.
Expected Behavior
Calling get_metrics() should return a dictionary of metrics e.g., {'rmse': 0.0} for the trained pipeline.
Current Behavior
The get_metrics call raises an exception with the message:
2025-02-10 13:35:38,815 - MetricsObjective - Objective evaluation error for graph {'depth': 2, 'length': 2, 'nodes': [adareg, ets]} on metric rmse: Metric can not be evaluated because of: Found input variables with inconsistent numbers of samples: [1, 30]
Traceback (most recent call last):
File "/home/hex/FEDOT/test-cache.py", line 380, in <module>
print(auto_model.get_metrics(validation_blocks=1))
File "/home/hex/FEDOT/fedot/api/main.py", line 475, in get_metrics
metrics = obj_eval.evaluate(self.current_pipeline).values
File "/home/hex/FEDOT/fedot/core/optimisers/objective/data_objective_eval.py", line 90, in evaluate
self._log.warning(f'Invalid fitness after objective evaluation. '
File "/home/hex/.local/lib/python3.8/site-packages/golem/core/log.py", line 172, in warning
raise_if_test(msg, **kwargs)
File "/home/hex/.local/lib/python3.8/site-packages/golem/core/log.py", line 213, in raise_if_test
raise Exception(msg)
Exception: Invalid fitness after objective evaluation. Skipping the graph: (/n_ets_{'error': 'add', 'trend': None, 'seasonal': 'mul', 'damped_trend': False, 'seasonal_periods': 24};)/n_adareg_{'learning_rate': 0.029655915966156916, 'loss': 'square'}
Possible Solution
- Data Alignment: The validation data and predictions might not be aligned in time steps.
- Pipeline Configuration: The
etsoradaregnodes may mishandle forecasting horizons. - Validation Blocks: The
validation_blocks=1parameter might incorrectly slice the time series data.
Steps to Reproduce
Run the test file test-cache.py in the #1334 PR.
Context [OPTIONAL]
Working on prediction cache for ts_forecasting pipelines in #1334