MAINT: Add dependencies on MKL and DPC runtimes used by oneDAL to nightly jobs#3010
MAINT: Add dependencies on MKL and DPC runtimes used by oneDAL to nightly jobs#3010david-cortes-intel wants to merge 2 commits intouxlfoundation:mainfrom
Conversation
david-cortes-intel
requested review from
Vika-F,
ahuber21,
ethanglaser,
homksei,
icfaust,
maria-Petrova,
syakov-intel and
yuejiaointel
as code owners
david-cortes-intel
changed the title
Codecov Report✅ All modified and coverable lines are covered by tests.
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
|
CI failure is in KMeans and happens sporadically in other jobs too. |
ethanglaser
left a comment
ethanglaser
left a comment
There was a problem hiding this comment.
Can all of this be bypassed by adding bash .ci/env/apt.sh mkl to the dpcpp installation step? In https://github.com/uxlfoundation/oneDAL/blob/main/.github/workflows/nightly-build.yml those two lines are sufficient to set up the environment necessary for oneDAL build. It's possible this oneDAL nightly build logic would need to be reconfigured as part of the library split, in which case we should update here to align, but I don't think we should be installing conda oneAPI deps on the sklearnex side.
|
@ethanglaser I cannot find any file named like that in this repository. Are you suggesting to add a new file? |
Right, it comes from the oneDAL repo (https://github.com/uxlfoundation/oneDAL/blob/main/.ci/env/apt.sh) and is used here to replicate the environment used to produce the oneDAL build. It's a bit confusing but that nightly-build.yml in oneDAL github actions also zips up the oneDAL env scripts for reuse here |
Looks like that file currently doesn't have entries for MKL DPC runtimes. |
|
Comment above was wrong, they do have them as transitive dependencies, but they use different names in APT than in pip and conda. |
|
@ethanglaser I'm not sure it'd be a good idea to reuse that script. It install development dependencies in addition to runtime ones, which could potentially mask dependency problems. |
|
@ethanglaser Also, how would that work for windows? The script uses APT to pull packages. |
We are building sklearnex as part of this scope - why are development dependencies not necessary? |
Because sklearnex doesn't depend directly on packages like |
There is similar scope for windows: https://github.com/uxlfoundation/scikit-learn-intelex/blob/main/.github/workflows/ci.yml#L201-L248, not sure on the specifics |
| python -m venv venv | ||
| source venv/bin/activate | ||
| pip install -r dependencies-dev | ||
| # Note: oneDAL packages should have dependencies on these, but |
this will make dependency tracking and upgrading harder making for a more often broken CI i agree with @ethanglaser |
|
everything in the github nightly should use pip hence the name. if it cannot it replicates infrastructure code from oneDAL for ease of maintenance and will be within one day of changes in oneDAL master making changes in oneDAL easier to see in sklearnex in a public manner |
3 participants
Description
OneDAL now has runtime dependencies on MKL and compiler DPC runtimes when using the DPC modules.
If installing a oneDAL package from pip/conda/apt/etc., these should be automatically pulled by the same package manager, but if using a from-source build directly, those dependencies won't necessarily be available in the system at the time of building / testing sklearnex.
This PR adds explicit dependencies on DPC components to the nightly jobs which use from-source builds of oneDAL, and updates the documentation to reflect these dependencies.
Checklist:
Completeness and readability
Testing