release: v0.0.12 (#141)

Author: AzulGarza

docs/changelogs/index.md

@@ -3,4 +3,6 @@
 Welcome to the TimeCopilot Changelog. Here, you will find a comprehensive list of all the changes, updates, and improvements made to the TimeCopilot project. This section is designed to keep you informed about the latest features, bug fixes, and enhancements as we continue to develop and refine the TimeCopilot experience. Stay tuned for regular updates and feel free to explore the details of each release below.
 
 
+- [v0.0.12](v0.0.12.md)
+- [v0.0.11](v0.0.11.md)
 - [v0.0.10](v0.0.10.md)

docs/changelogs/v0.0.12.md

@@ -0,0 +1,144 @@
+### Features
+
+* **Query Method**: Added a `query` method to the forecaster for flexible, programmatic access to model capabilities. See [#134](https://github.com/AzulGarza/timecopilot/pull/134).
+    ```python
+    from timecopilot import TimeCopilot
+
+    tc = TimeCopilot(llm="openai:gpt-4o")
+    tc.forecast(
+        df="https://timecopilot.s3.amazonaws.com/public/data/air_passengers.csv", 
+        h=12,
+    )
+    result = tc.query("What is the best model for monthly data?")
+    print(result.output)
+    ```
+
+* **Async TimeCopilot Agent**: Introduced the `AsyncTimeCopilot` class for asynchronous forecasting and querying. See [#135](https://github.com/AzulGarza/timecopilot/pull/135) and [#138](https://github.com/AzulGarza/timecopilot/pull/138).
+    ```python
+    import asyncio
+    from timecopilot import AsyncTimeCopilot
+
+    async def main():
+        tc = AsyncTimeCopilot(llm="openai:gpt-4o")
+        await tc.forecast(
+            df="https://timecopilot.s3.amazonaws.com/public/data/air_passengers.csv",
+            h=12
+        )
+        answer = await tc.query("Which model performed best?")
+        print(answer.output)
+
+    asyncio.run(main())
+    ```
+
+* **Fallback Model Support**: The `TimeCopilotForecaster` now supports a fallback model, which is used if the primary model fails. See [#123](https://github.com/AzulGarza/timecopilot/pull/123).
+    ```python
+    from timecopilot.forecaster import TimeCopilotForecaster
+    from timecopilot.models.foundational.timesfm import TimesFM
+    from timecopilot.models.benchmarks.stats import SeasonalNaive
+
+    forecaster = TimeCopilotForecaster(
+        models=[TimesFM()],
+        fallback_model=SeasonalNaive()
+    )
+    ```
+
+* **TimesFM 2.0 Support**: Added support for TimesFM 2.0, enabling the use of the latest version of Google's TimesFM model. See [#128](https://github.com/AzulGarza/timecopilot/pull/128).
+    ```python
+    from timecopilot.models.foundational.timesfm import TimesFM
+
+    model = TimesFM(
+        # default value
+        repo_id="google/timesfm-2.0-500m-pytorch",
+    )
+    ```
+
+* **TabPFN Foundation Model**: Added the [TabPFN](https://github.com/PriorLabs/TabPFN) time series foundation model. See [#113](https://github.com/AzulGarza/timecopilot/pull/113).
+    ```python
+    import pandas as pd
+    from timecopilot.models.foundational.tabpfn import TabPFN
+
+    df = pd.read_csv("https://timecopilot.s3.amazonaws.com/public/data/algeria_exports.csv", parse_dates=["ds"])
+    model = TabPFN()
+    fcst = model.forecast(df, h=12)
+    print(fcst)
+    ```
+
+* **Median Ensemble**: Introduced a new Median Ensemble model that combines predictions from multiple models to improve forecast accuracy. See [#144](https://github.com/AzulGarza/timecopilot/pull/144).
+    ```python
+    import pandas as pd
+    from timecopilot.models.benchmarks import SeasonalNaive
+    from timecopilot.models.ensembles.median import MedianEnsemble
+    from timecopilot.models.foundational.chronos import Chronos
+
+
+    df = pd.read_csv(
+        "https://timecopilot.s3.amazonaws.com/public/data/air_passengers.csv",
+        parse_dates=["ds"],
+    )
+
+    models = [
+        Chronos(
+            repo_id="amazon/chronos-t5-tiny",
+            alias="Chronos-T5",
+        ),
+        Chronos(
+            repo_id="amazon/chronos-bolt-tiny",
+            alias="Chronos-Bolt",
+        ),
+        SeasonalNaive(),
+    ]
+    median_ensemble = MedianEnsemble(models=models)
+    fcst_df = median_ensemble.forecast(
+        df=df,
+        h=12,
+    )
+    print(fcst_df)
+    ```
+
+* **GIFTEval Module**: Added the [GIFTEval](https://github.com/SalesforceAIResearch/gift-eval/) module for advanced evaluation of forecasting models. See [#140](https://github.com/AzulGarza/timecopilot/pull/140).
+    ```python
+    import pandas as pd
+    from timecopilot.gift_eval.eval import GIFTEval, QUANTILE_LEVELS
+    from timecopilot.gift_eval.gluonts_predictor import GluonTSPredictor
+    from timecopilot.models.benchmarks import SeasonalNaive
+
+    storage_path = ".pytest_cache/gift_eval"
+    GIFTEval.download_data(storage_path)
+
+    gifteval = GIFTEval(
+        dataset_name="m4_weekly",
+        term="short",
+        output_path="./seasonal_naive",
+        storage_path=storage_path,
+    )
+    predictor = GluonTSPredictor(
+        forecaster=SeasonalNaive(),
+        h=gifteval.dataset.prediction_length,
+        freq=gifteval.dataset.freq,
+        quantiles=QUANTILE_LEVELS,
+        batch_size=512,
+    )
+    gifteval.evaluate_predictor(
+        predictor,
+        batch_size=512,
+    )
+    eval_df = pd.read_csv("./seasonal_naive/all_results.csv")
+    print(eval_df)
+    ```
+
+### Fixes
+
+* **Model Compatibility**: Added support for the Moirai and TimeGPT models. See [#115](https://github.com/AzulGarza/timecopilot/pull/115), [#117](https://github.com/AzulGarza/timecopilot/pull/117).
+* **GluonTS Forecaster**: Improved frequency handling and now uses the median for forecasts. See [#124](https://github.com/AzulGarza/timecopilot/pull/124), [#127](https://github.com/AzulGarza/timecopilot/pull/127).
+* **TimesFM Quantile Names**: TimesFM now returns correct quantile names. See [#131](https://github.com/AzulGarza/timecopilot/pull/131).
+* **Removed Lag Llama**: The Lag Llama model has been removed. See [#116](https://github.com/AzulGarza/timecopilot/pull/116).
+* **DataFrame Handling**: Fixed DataFrame copying to avoid index side effects. See [#120](https://github.com/AzulGarza/timecopilot/pull/120).
+
+### Docs
+
+* **Foundation Model Documentation**: Added comprehensive documentation for foundation models, including paper citations and repository links. See [#118](https://github.com/AzulGarza/timecopilot/pull/118).
+* **Unique Alias Validation**: Added validation to prevent column conflicts in `TimeCopilotForecaster`. See [#122](https://github.com/AzulGarza/timecopilot/pull/122).
+
+---
+
+**Full Changelog**: https://github.com/AzulGarza/timecopilot/compare/v0.0.11...v0.0.12

mkdocs.yml

@@ -32,6 +32,7 @@ nav:
       - api/gift_eval/gift_eval.md
   - Changelogs:
       - changelogs/index.md
+      - changelogs/v0.0.12.md
       - changelogs/v0.0.11.md
       - changelogs/v0.0.10.md
 

pyproject.toml

@@ -61,6 +61,7 @@ docs = [
     "mkdocs>=1.6.1",
     "mkdocs-material>=9.6.14",
     "mkdocstrings[python]>=0.29.1",
+    "mktestdocs>=0.2.5",
     "modal>=1.0.4",
     "ruff>=0.12.1",
 ]

tests/gift_eval/conftest.py

@@ -2,19 +2,20 @@
 
 import pandas as pd
 import pytest
-from huggingface_hub import snapshot_download
+
+from timecopilot.gift_eval.eval import GIFTEval
 
 
 @pytest.fixture(scope="session")
-def cache_dir() -> Path:
-    cache_dir = Path(".pytest_cache") / "gift_eval"
-    cache_dir.mkdir(parents=True, exist_ok=True)
-    return cache_dir
+def cache_path() -> Path:
+    cache_path = Path(".pytest_cache") / "gift_eval"
+    cache_path.mkdir(parents=True, exist_ok=True)
+    return cache_path
 
 
 @pytest.fixture(scope="session")
-def all_results_df(cache_dir: Path) -> pd.DataFrame:
-    all_results_file = cache_dir / "seasonal_naive_all_results.csv"
+def all_results_df(cache_path: Path) -> pd.DataFrame:
+    all_results_file = cache_path / "seasonal_naive_all_results.csv"
     if not all_results_file.exists():
         all_results_df = pd.read_csv(
             "https://huggingface.co/spaces/Salesforce/GIFT-Eval/raw/main/results/seasonal_naive/all_results.csv"
@@ -24,15 +25,6 @@ def all_results_df(cache_dir: Path) -> pd.DataFrame:
 
 
 @pytest.fixture(scope="session")
-def gift_eval_dir(cache_dir: Path) -> Path:
-    snapshot_download(
-        repo_id="Salesforce/GiftEval",
-        repo_type="dataset",
-        local_dir=cache_dir,
-    )
-    return cache_dir
-
-
-@pytest.fixture(autouse=True)
-def gift_eval_env(monkeypatch, gift_eval_dir):
-    monkeypatch.setenv("GIFT_EVAL", gift_eval_dir)
+def storage_path(cache_path: Path) -> Path:
+    GIFTEval.download_data(cache_path)
+    return cache_path

tests/gift_eval/test_evaluation.py

@@ -47,12 +47,14 @@ def test_evaluation(
     dataset_name: str,
     term: str,
     all_results_df: pd.DataFrame,
+    storage_path: Path,
 ):
     with tempfile.TemporaryDirectory() as temp_dir:
         gifteval = GIFTEval(
             dataset_name=dataset_name,
             term=term,
-            output_dir=temp_dir,
+            output_path=temp_dir,
+            storage_path=storage_path,
         )
         predictor = GluonTSPredictor(
             forecaster=SeasonalNaive(

timecopilot/__init__.py

@@ -1,4 +1,4 @@
-from .agent import TimeCopilot
+from .agent import AsyncTimeCopilot, TimeCopilot
 from .forecaster import TimeCopilotForecaster
 
-__all__ = ["TimeCopilot", "TimeCopilotForecaster"]
+__all__ = ["AsyncTimeCopilot", "TimeCopilot", "TimeCopilotForecaster"]

timecopilot/gift_eval/data.py

@@ -107,18 +107,25 @@ def __call__(
 
 
 class Dataset:
+    def _storage_path_from_env_var(self, env_var: str) -> Path:
+        load_dotenv()
+        env_var_value = os.getenv(env_var)
+        if env_var_value is None:
+            raise ValueError(f"Environment variable {env_var} is not set")
+        return Path(env_var_value)
+
     def __init__(
         self,
         name: str,
         term: Term | str = Term.SHORT,
         to_univariate: bool = False,
+        storage_path: Path | str | None = None,
         storage_env_var: str = "GIFT_EVAL",
     ):
-        load_dotenv()
-        env_var = os.getenv(storage_env_var)
-        if env_var is None:
-            raise ValueError(f"Environment variable {storage_env_var} is not set")
-        storage_path = Path(env_var)
+        if storage_path is None:
+            storage_path = self._storage_path_from_env_var(storage_env_var)
+        else:
+            storage_path = Path(storage_path)
         self.hf_dataset = datasets.load_from_disk(str(storage_path / name)).with_format(
             "numpy"
         )

timecopilot/gift_eval/eval.py

@@ -19,6 +19,7 @@
 from gluonts.model import evaluate_model
 from gluonts.model.predictor import RepresentablePredictor
 from gluonts.time_feature import get_seasonality
+from huggingface_hub import snapshot_download
 
 from .data import Dataset
 from .gluonts_predictor import GluonTSPredictor
@@ -54,11 +55,26 @@ class GIFTEval:
     desired.
     """
 
+    @staticmethod
+    def download_data(storage_path: Path | str | None = None):
+        """
+        Download the GIFTEval dataset from Hugging Face.
+
+        Args:
+            storage_path (Path | str | None): Path to store the dataset.
+        """
+        snapshot_download(
+            repo_id="Salesforce/GiftEval",
+            repo_type="dataset",
+            local_dir=storage_path,
+        )
+
     def __init__(
         self,
         dataset_name: str,
         term: str,
-        output_dir: str | Path | None,
+        output_path: Path | str | None = None,
+        storage_path: Path | str | None = None,
     ):
         # fmt: off
         """
@@ -67,19 +83,25 @@ def __init__(
         Args:
             dataset_name (str): Name of the dataset to evaluate on.
             term (str): Evaluation term (e.g., 'medium', 'long').
-            output_dir (str | Path | None): Directory to save results CSV, or
+            output_path (str | Path | None): Directory to save results CSV, or
                 None to skip saving.
+            storage_path (Path | str | None): Path where the dataset is stored.
 
         Example:
             ```python
+            import pandas as pd
             from timecopilot.gift_eval.eval import GIFTEval, QUANTILE_LEVELS
             from timecopilot.gift_eval.gluonts_predictor import GluonTSPredictor
             from timecopilot.models.benchmarks import SeasonalNaive
 
+            storage_path = "./gift_eval_data"
+            GIFTEval.download_data(storage_path)
+
             gifteval = GIFTEval(
-                dataset_name=dataset_name,
-                term=term,
-                output_dir="./my-dir",
+                dataset_name="m4_weekly",
+                term="short",
+                output_path="./seasonal_naive",
+                storage_path=storage_path,
             )
             predictor = GluonTSPredictor(
                 forecaster=SeasonalNaive(),
@@ -92,7 +114,7 @@ def __init__(
                 predictor,
                 batch_size=512,
             )
-            eval_df = pd.read_csv("./my-dir/all_results.csv")
+            eval_df = pd.read_csv("./seasonal_naive/all_results.csv")
             ```
 
         Raises:
@@ -131,17 +153,19 @@ def __init__(
                 name=dataset_name,
                 term=term,
                 to_univariate=False,
+                storage_path=storage_path,
             ).target_dim
             != 1
         )
         self.dataset = Dataset(
             name=dataset_name,
             term=term,
             to_univariate=to_univariate,
+            storage_path=storage_path,
         )
         self.dataset_name = dataset_name
         self.seasonality = get_seasonality(self.dataset.freq)
-        self.output_dir = output_dir
+        self.output_path = output_path
 
     def evaluate_predictor(
         self,
@@ -220,8 +244,8 @@ def evaluate_predictor(
                 "num_variates",
             ],
         )
-        if self.output_dir is not None:
-            csv_file_path = Path(self.output_dir) / "all_results.csv"
+        if self.output_path is not None:
+            csv_file_path = Path(self.output_path) / "all_results.csv"
             csv_file_path.parent.mkdir(parents=True, exist_ok=True)
             results_df.to_csv(csv_file_path, index=False)