[CI] Improvements to conditional GPU PR tests #10859
+302
−11
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,223 @@ | ||
| name: Fast GPU Tests on PR | ||
|
|
||
| on: | ||
| pull_request: | ||
| branches: main | ||
| paths: | ||
| - "src/diffusers/models/modeling_utils.py" | ||
| - "src/diffusers/models/model_loading_utils.py" | ||
| - "src/diffusers/pipelines/pipeline_utils.py" | ||
| - "src/diffusers/pipeline_loading_utils.py" | ||
| - "src/diffusers/loaders/lora_base.py" | ||
| - "src/diffusers/loaders/lora_pipeline.py" | ||
| - "src/diffusers/loaders/peft.py" | ||
| - "src/diffusers/loaders/ip_adapter.py" | ||
| workflow_dispatch: | ||
| env: | ||
| DIFFUSERS_IS_CI: yes | ||
|
Member
There was a problem hiding this comment. Do we want to run any slow tests here? If so, we should set
Collaborator
Author
There was a problem hiding this comment. Since it's running on a PR there is a high likelihood of multiple pushes occurring on a branch. This would end up triggering slow tests many times. Plus having to wait for them to finish before merge could be a bit overkill. Fast GPU tests with tiny versions of models should be able to detect serious breaking functionality. |
||
| OMP_NUM_THREADS: 8 | ||
| MKL_NUM_THREADS: 8 | ||
| HF_HUB_ENABLE_HF_TRANSFER: 1 | ||
| PYTEST_TIMEOUT: 600 | ||
| PIPELINE_USAGE_CUTOFF: 1000000000 # set high cutoff so that only always-test pipelines run | ||
|
|
||
| jobs: | ||
| setup_torch_cuda_pipeline_matrix: | ||
| name: Setup Torch Pipelines CUDA Slow Tests Matrix | ||
| runs-on: | ||
| group: aws-general-8-plus | ||
| container: | ||
| image: diffusers/diffusers-pytorch-cpu | ||
| outputs: | ||
| pipeline_test_matrix: ${{ steps.fetch_pipeline_matrix.outputs.pipeline_test_matrix }} | ||
| steps: | ||
| - name: Checkout diffusers | ||
| uses: actions/checkout@v3 | ||
| with: | ||
| fetch-depth: 2 | ||
| - name: Install dependencies | ||
| run: | | ||
| python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH" | ||
| python -m uv pip install -e [quality,test] | ||
| - name: Environment | ||
| run: | | ||
| python utils/print_env.py | ||
| - name: Fetch Pipeline Matrix | ||
| id: fetch_pipeline_matrix | ||
| run: | | ||
| matrix=$(python utils/fetch_torch_cuda_pipeline_test_matrix.py) | ||
| echo $matrix | ||
| echo "pipeline_test_matrix=$matrix" >> $GITHUB_OUTPUT | ||
| - name: Pipeline Tests Artifacts | ||
| if: ${{ always() }} | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: test-pipelines.json | ||
| path: reports | ||
|
|
||
| torch_pipelines_cuda_tests: | ||
| name: Torch Pipelines CUDA Tests | ||
| needs: setup_torch_cuda_pipeline_matrix | ||
| strategy: | ||
| fail-fast: false | ||
| max-parallel: 8 | ||
| matrix: | ||
| module: ${{ fromJson(needs.setup_torch_cuda_pipeline_matrix.outputs.pipeline_test_matrix) }} | ||
| runs-on: | ||
| group: aws-g4dn-2xlarge | ||
| container: | ||
| image: diffusers/diffusers-pytorch-cuda | ||
| options: --shm-size "16gb" --ipc host --gpus 0 | ||
| steps: | ||
| - name: Checkout diffusers | ||
| uses: actions/checkout@v3 | ||
| with: | ||
| fetch-depth: 2 | ||
| - name: NVIDIA-SMI | ||
| run: | | ||
| nvidia-smi | ||
| - name: Install dependencies | ||
| run: | | ||
| python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH" | ||
| python -m uv pip install -e [quality,test] | ||
| pip uninstall accelerate -y && python -m uv pip install -U accelerate@git+https://github.com/huggingface/accelerate.git | ||
|
DN6 marked this conversation as resolved.
|
||
| - name: Environment | ||
| run: | | ||
| python utils/print_env.py | ||
| - name: Extract tests | ||
| run: | | ||
| pattern=$(python utils/extract_tests_from_mixin.py --type pipeline) | ||
| echo "TEST_PATTERN=$pattern" >> $GITHUB_ENV | ||
| - name: PyTorch CUDA checkpoint tests on Ubuntu | ||
| env: | ||
| HF_TOKEN: ${{ secrets.DIFFUSERS_HF_HUB_READ_TOKEN }} | ||
| # https://pytorch.org/docs/stable/notes/randomness.html#avoiding-nondeterministic-algorithms | ||
| CUBLAS_WORKSPACE_CONFIG: :16:8 | ||
| run: | | ||
| python -m pytest -n 1 --max-worker-restart=0 --dist=loadfile \ | ||
| -s -v -k "not Flax and not Onnx" \ | ||
| --make-reports=tests_pipeline_${{ matrix.module }}_cuda \ | ||
| tests/pipelines/${{ matrix.module }} | ||
| - name: Failure short reports | ||
| if: ${{ failure() }} | ||
| run: | | ||
| cat reports/tests_pipeline_${{ matrix.module }}_cuda_stats.txt | ||
| cat reports/tests_pipeline_${{ matrix.module }}_cuda_failures_short.txt | ||
| - name: Test suite reports artifacts | ||
| if: ${{ always() }} | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: pipeline_${{ matrix.module }}_test_reports | ||
| path: reports | ||
|
|
||
| torch_cuda_tests: | ||
| name: Torch CUDA Tests | ||
| runs-on: | ||
| group: aws-g4dn-2xlarge | ||
| container: | ||
| image: diffusers/diffusers-pytorch-cuda | ||
| options: --shm-size "16gb" --ipc host --gpus 0 | ||
| defaults: | ||
| run: | ||
| shell: bash | ||
| strategy: | ||
| fail-fast: false | ||
| max-parallel: 2 | ||
| matrix: | ||
| module: [models, schedulers, lora, others] | ||
| steps: | ||
| - name: Checkout diffusers | ||
| uses: actions/checkout@v3 | ||
| with: | ||
| fetch-depth: 2 | ||
|
|
||
| - name: Install dependencies | ||
| run: | | ||
| python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH" | ||
| python -m uv pip install -e [quality,test] | ||
| python -m uv pip install peft@git+https://github.com/huggingface/peft.git | ||
| pip uninstall accelerate -y && python -m uv pip install -U accelerate@git+https://github.com/huggingface/accelerate.git | ||
|
DN6 marked this conversation as resolved.
|
||
| pip uninstall transformers -y && python -m uv pip install -U transformers@git+https://github.com/huggingface/transformers.git --no-deps | ||
|
|
||
| - name: Environment | ||
| run: | | ||
| python utils/print_env.py | ||
|
|
||
| - name: Extract tests | ||
| run: | | ||
| pattern=$(python utils/extract_tests_from_mixin.py --type ${{ matrix.module }}) | ||
| echo "TEST_PATTERN=$pattern" >> $GITHUB_ENV | ||
|
|
||
| - name: Run PyTorch CUDA tests | ||
| env: | ||
| HF_TOKEN: ${{ secrets.DIFFUSERS_HF_HUB_READ_TOKEN }} | ||
| # https://pytorch.org/docs/stable/notes/randomness.html#avoiding-nondeterministic-algorithms | ||
| CUBLAS_WORKSPACE_CONFIG: :16:8 | ||
| run: | | ||
| python -m pytest -n 1 --max-worker-restart=0 --dist=loadfile \ | ||
| -s -v -k "not Flax and not Onnx and $TEST_PATTERN" \ | ||
|
Member
There was a problem hiding this comment. I am getting the following for test_StableDiffusionMixin_component or test_attention_slicing_forward_pass or test_callback_cfg or test_callback_inputs or test_cfg or test_components_function or test_cpu_offload_forward_pass_twice or test_dict_tuple_outputs_equivalent or test_encode_prompt_works_in_isolation or test_float16_inference or test_group_offloading_inference or test_inference_batch_consistent or test_inference_batch_single_identical or test_layerwise_casting_inference or test_loading_with_incorrect_variants_raises_error or test_loading_with_variants or test_model_cpu_offload_forward_pass or test_num_images_per_prompt or test_pipeline_call_signature or test_save_load_dduf or test_save_load_float16 or test_save_load_local or test_save_load_optional_components or test_sequential_cpu_offload_forward_pass or test_sequential_offload_forward_pass_twice or test_serialization_with_variants or test_to_device or test_to_dtype or test_xformers_attention_forwardGenerator_passIs the "or" joining prefix expected? Is it needed for
Collaborator
Author
There was a problem hiding this comment. Correct. We only want to run the tests in |
||
| --make-reports=tests_torch_cuda_${{ matrix.module }} \ | ||
| tests/${{ matrix.module }} | ||
|
|
||
| - name: Failure short reports | ||
| if: ${{ failure() }} | ||
| run: | | ||
| cat reports/tests_torch_cuda_${{ matrix.module }}_stats.txt | ||
| cat reports/tests_torch_cuda_${{ matrix.module }}_failures_short.txt | ||
|
|
||
| - name: Test suite reports artifacts | ||
| if: ${{ always() }} | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: torch_cuda_test_reports_${{ matrix.module }} | ||
| path: reports | ||
|
|
||
| run_examples_tests: | ||
| name: Examples PyTorch CUDA tests on Ubuntu | ||
|
|
||
| runs-on: | ||
| group: aws-g4dn-2xlarge | ||
|
|
||
| container: | ||
| image: diffusers/diffusers-pytorch-cuda | ||
| options: --gpus 0 --shm-size "16gb" --ipc host | ||
| steps: | ||
| - name: Checkout diffusers | ||
| uses: actions/checkout@v3 | ||
| with: | ||
| fetch-depth: 2 | ||
|
|
||
| - name: NVIDIA-SMI | ||
| run: | | ||
| nvidia-smi | ||
| - name: Install dependencies | ||
| run: | | ||
| python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH" | ||
| python -m uv pip install -e [quality,test,training] | ||
|
|
||
| - name: Environment | ||
| run: | | ||
| python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH" | ||
| python utils/print_env.py | ||
|
|
||
| - name: Run example tests on GPU | ||
| env: | ||
| HF_TOKEN: ${{ secrets.DIFFUSERS_HF_HUB_READ_TOKEN }} | ||
| run: | | ||
| python -m venv /opt/venv && export PATH="/opt/venv/bin:$PATH" | ||
| python -m uv pip install timm | ||
| python -m pytest -n 1 --max-worker-restart=0 --dist=loadfile -s -v --make-reports=examples_torch_cuda examples/ | ||
|
|
||
| - name: Failure short reports | ||
| if: ${{ failure() }} | ||
| run: | | ||
| cat reports/examples_torch_cuda_stats.txt | ||
| cat reports/examples_torch_cuda_failures_short.txt | ||
|
|
||
| - name: Test suite reports artifacts | ||
| if: ${{ always() }} | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: examples_test_reports | ||
| path: reports | ||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| import argparse | ||
| import inspect | ||
| import sys | ||
| from pathlib import Path | ||
| from typing import List, Type | ||
|
|
||
|
|
||
| root_dir = Path(__file__).parent.parent.absolute() | ||
| sys.path.insert(0, str(root_dir)) | ||
|
|
||
| parser = argparse.ArgumentParser() | ||
| parser.add_argument("--type", type=str, default=None) | ||
| args = parser.parse_args() | ||
|
|
||
|
|
||
| def get_test_methods_from_class(cls: Type) -> List[str]: | ||
| """ | ||
| Get all test method names from a given class. | ||
| Only returns methods that start with 'test_'. | ||
| """ | ||
| test_methods = [] | ||
| for name, obj in inspect.getmembers(cls): | ||
| if name.startswith("test_") and inspect.isfunction(obj): | ||
| test_methods.append(name) | ||
| return sorted(test_methods) | ||
|
|
||
|
|
||
| def generate_pytest_pattern(test_methods: List[str]) -> str: | ||
| """Generate pytest pattern string for the -k flag.""" | ||
| return " or ".join(test_methods) | ||
|
|
||
|
|
||
| def generate_pattern_for_mixin(mixin_class: Type) -> str: | ||
| """ | ||
| Generate pytest pattern for a specific mixin class. | ||
| """ | ||
| if mixin_cls is None: | ||
| return "" | ||
| test_methods = get_test_methods_from_class(mixin_class) | ||
| return generate_pytest_pattern(test_methods) | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| mixin_cls = None | ||
| if args.type == "pipeline": | ||
| from tests.pipelines.test_pipelines_common import PipelineTesterMixin | ||
|
|
||
| mixin_cls = PipelineTesterMixin | ||
|
|
||
| elif args.type == "models": | ||
| from tests.models.test_modeling_common import ModelTesterMixin | ||
|
|
||
| mixin_cls = ModelTesterMixin | ||
|
|
||
| elif args.type == "lora": | ||
|
Member
There was a problem hiding this comment. "others" and "schedulers" aren't covered here.
Collaborator
Author
There was a problem hiding this comment. Others and Schedulers are quite fast to run and don't involve downloading any large models. In this case it would print an empty string and the full test suite would run for those modules. |
||
| from tests.lora.utils import PeftLoraLoaderMixinTests | ||
|
|
||
| mixin_cls = PeftLoraLoaderMixinTests | ||
|
|
||
| pattern = generate_pattern_for_mixin(mixin_cls) | ||
| print(pattern) | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.