really slow CI due to install packages

[chrisiacovella]

Micromamba seems to be taking a really long time to run the install packages step.

Here are two examples of successful, but really slow install steps .

Dataviz, took 3 hours to install on Ubuntu:

https://github.com/choderalab/drugforge/actions/runs/22459598605/job/65067697532

Workflows, took 2 hours to install on Ubuntu:
https://github.com/choderalab/drugforge/actions/runs/22459598695/job/65067493154?pr=132 two hours for this to install.

In both cases, Install packages is calling micromamba several times in sequence, to install the environment files for the all the sub packages it depends on, e.g. in the dataviz:

`
- name: Install packages
run: |
micromamba install -n dataviz --file devtools/conda-envs/${{ matrix.os }}/data.yaml
micromamba install -n dataviz --file devtools/conda-envs/${{ matrix.os }}/docking.yaml
micromamba install -n dataviz --file devtools/conda-envs/${{ matrix.os }}/simulation.yaml
micromamba install -n dataviz --file devtools/conda-envs/${{ matrix.os }}/modeling.yaml
micromamba install -n dataviz --file devtools/conda-envs/${{ matrix.os }}/spectrum.yaml

    python -m pip install -e ./drugforge-data --no-deps
    python -m pip install -e ./drugforge-modeling --no-deps
    python -m pip install -e ./drugforge-cli --no-deps
    python -m pip install -e ./drugforge-dataviz --no-deps
    python -m pip install -e ./drugforge-docking --no-deps
    python -m pip install -e ./drugforge-simulation --no-deps
    python -m pip install -e ./drugforge-spectrum --no-deps
    micromamba list

`

Interestingly, the CLI only took half an hour and called all of the various yaml files

Obviously, this was not happening previously, so it could be a GitHub issue (there are a bunch of warnings that seem to have shown up in some of the tests about caching services unavailable, but those repos ran efficiently).

Is there a better/more efficient way to streamline this?

Could this issue be with some of the pins not being consistent across the various .yaml files making it harder to solve each time?

[mikemhenry]

I will try and see what changed, since like you said it used to be much faster. One thing that looks odd is:

   - python[version=">=3.10,<3.12"]

for the specs, even though this is a python=3.11 test so I think we need to set that in the CI on those install commands since not having to solve for more than one python version should help.

It seems like it wants to upgrade and downgrade things at each step as well which slows things down.

I will see if there is an obvious change, but an easy fix would be to setup the conda env in one shot.

[chrisiacovella]

A few things I've done:

• standardized the pins, which is even more important now that we are using pydantic v2 (as there are minimum versions for gufe and other openfe toolkits).
  -- This still didn't speed things up, but something needed to be done anyway
• I had to add in the official rapidsai channel to allow us to install newer versions of dask-cuda (we need newer versions of dark for the newer versions of openfe tools. coda-forge dask-cuda is out of date). I think this may have been causing some issues as there was no way to solve, but because pins were not as clear, it was trying with a ton of different (and possibly incompatible) versions of the openfe tools
• I moved the additional calls to update the micromamba environment (with additional yawls) out of the "install packages" step and added them as "create-args" in the "setup conda environment" step. It then tries to solve these simultaneous and will allow better caching.
  -- I had to specify which yaml files in the matrix setup, since Mac OS and ubuntu need different yaml files. It seems to be working much better now:

matrix:
os: [ubuntu-latest, macos-latest]
python-version: ["3.11"]
include:
- os: ubuntu-latest
yaml_file_args:
--file devtools/conda-envs/ubuntu-latest/data.yaml
--file devtools/conda-envs/ubuntu-latest/docking.yaml
--file devtools/conda-envs/ubuntu-latest/modeling.yaml
--file devtools/conda-envs/ubuntu-latest/alchemy.yaml
--file devtools/conda-envs/ubuntu-latest/simulation.yaml
--file devtools/conda-envs/ubuntu-latest/dataviz.yaml
--file devtools/conda-envs/ubuntu-latest/spectrum.yaml
--file devtools/conda-envs/ubuntu-latest/ml.yaml
- os: macOS-latest
yaml_file_args:
--file devtools/conda-envs/macOS-latest/data.yaml
--file devtools/conda-envs/macOS-latest/docking.yaml
--file devtools/conda-envs/macOS-latest/modeling.yaml
--file devtools/conda-envs/macOS-latest/alchemy.yaml
--file devtools/conda-envs/macOS-latest/simulation.yaml
--file devtools/conda-envs/macOS-latest/dataviz.yaml
--file devtools/conda-envs/macOS-latest/spectrum.yaml

...

    cache-downloads-key: downloads-${{ steps.date.outputs.date }}
    create-args: >-
      python==${{ matrix.python-version }}
      ${{ matrix.yaml_file_args }}