enhance PythonPackage and PythonBundle to verify Python package names and versions with pip list with sanity_check_pip_list parameter#4049
Conversation
boegel
changed the title
sanity_pip_list parameter to verify Python package names and versions with 'pip list'sanity_pip_list parameter to verify Python package names and versions with pip list
|
hm, ERROR: Installation of Python-bundle-PyPI-2025.04-GCCcore-14.2.0.eb failed: The following Python packages were likely specified with a wrong name because they are missing from the 'pip list' output:
semantic-version (close matches: ['semantic_version']) -> wrong
importlib-metadata (close matches: ['importlib_metadata']) -> wrong
backports.entry-points-selectable (close matches: ['backports.entry_points_selectable']) -> wrong
importlib-resources (close matches: ['importlib_resources']) -> wrong
py-expression-eval (close matches: ['py_expression_eval']) -> wrong
backports.functools-lru-cache (close matches: ['backports.functools_lru_cache']) -> wrong
Babel (close matches: ['babel']) -> correct
tomli-w (close matches: ['tomli_w', 'tomli', 'toml']) -> wrong
rapidfuzz (close matches: ['RapidFuzz']) -> correctwe might have to query pypi to get the correct name: $ python -c 'import requests; print(requests.get("https://pypi.org/pypi/semantic_version/json").json()["info"]["name"])'
semantic-version@boegel what do you think? |
|
i may have a better idea. pypa normalizes the names as follows (see https://packaging.python.org/en/latest/specifications/name-normalization/#name-normalization) import re
def normalize(name):
return re.sub(r"[-_.]+", "-", name).lower()this produces a canonical name for all the allowed variations. the idea is to let EB normalize the names on the fly and write the normalized extension names in the module file. the upside is that there is guaranteed to be only 1 extension name in the modules, making lookup (e.g. with the downside is that some extension names will look a bit different from before and from what's shown on pypi (no dots, no underscores). in case some sites care about that, we could make writing the normalized extension names in the module file optional, e.g. with a build option. thoughts? |
|
new plan after short discussion with @boegel:
EDIT: after thinking about this more, i'm not sure if we should be querying pypi.org for the display name of every package, because project authors can change this, so it's not guaranteed to stay the same. in any case, that will be for another PR. |
smoors
changed the title
sanity_pip_list parameter to verify Python package names and versions with pip listpip list in sanity_pip_check
smoors
changed the title
pip list in sanity_pip_checkpip list as part of sanity_pip_check
|
Test report by @smoors Overview of tested easyconfigs (in order)
Build succeeded for 4 out of 5 (total: 2 mins 35 secs) (5 easyconfigs in total) |
|
Test report by @smoors Overview of tested easyconfigs (in order)
Build succeeded for 1 out of 1 (total: 18 secs) (1 easyconfigs in total) |
|
Test report by @smoors Overview of tested easyconfigs (in order)
Build succeeded for 1 out of 1 (total: 1 min 57 secs) (1 easyconfigs in total) |
smoors
changed the title
pip list as part of sanity_pip_checkpip list with sanity_pip_check
smoors
changed the title
pip list with sanity_pip_checkpip list with sanity_check_pip_list
smoors
changed the title
pip list with sanity_check_pip_listpip list with sanity_check_pip_list parameter
|
@boegelbot please test @ jsc-zen3 |
|
@boegel: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de PR test command '
Test results coming soon (I hope)... Details- notification for comment with ID 4200667368 processed Message to humans: this is just bookkeeping information for me, |
|
Test report by @boegelbot Overview of tested easyconfigs (in order)
Build succeeded for 3 out of 4 (total: 2 hours 8 mins 34 secs) (4 easyconfigs in total) |
This is expected, it's being fixed in: |
|
@boegelbot please test @ jsc-zen3 |
|
@boegel: Request for testing this PR well received on jsczen3l1.int.jsc-zen3.fz-juelich.de PR test command '
Test results coming soon (I hope)... Details- notification for comment with ID 4201753494 processed Message to humans: this is just bookkeeping information for me, |
|
Test report by @boegelbot Overview of tested easyconfigs (in order)
Build succeeded for 1 out of 1 (total: 4 mins 44 secs) (1 easyconfigs in total) |
|
Test report by @boegel Overview of tested easyconfigs (in order)
Build succeeded for 3 out of 4 (total: 23 mins 36 secs) (4 easyconfigs in total) |
|
@smoors Another actual problem uncovered: That shows that this all works as designed, so good! |
boegel
changed the title
pip list with sanity_check_pip_list parameterPythonPackage and PythonBundle to verify Python package names and versions with pip list with sanity_check_pip_list parameter
3 participants
cfr. conf call discussion https://github.com/easybuilders/easybuild/wiki/Conference-call-notes-20260114
pip listname and version.pip listif there is no matching name.implemented for
Python,PythonPackage,PythonBundleeasyblocks (+ derivatives).UPDATE 2026-03-22:
pip listsanity check runs only if--upload-test-reportis enabled. you can override this by setting easyconfig parametersanity_check_pip_listtoTrueorFalsepip listcheck for0.0.0versions is still done by default as before this PR, but is now done inrun_pip_list()instead ofrun_pip_check().