Don't install pypi packages to ~/.local

[danielhollas]

Based on #1423

This drops the image size by 300Mb, since /opt/home.tar contained unwanted packages installed in ~/.local, whereas they should have been installed to /opt/conda. Using uv instead of pip should also speed up the build.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 72.31%. Comparing base (c37b545) to head (88b9bd8).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1337      +/-   ##
==========================================
- Coverage   72.32%   72.31%   -0.02%     
==========================================
  Files         112      112              
  Lines        7245     7245              
==========================================
- Hits         5240     5239       -1     
- Misses       2005     2006       +1     

Flag	Coverage Δ
python-3.11	?
python-3.12	72.31% <ø> (?)
python-3.9	72.34% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[danielhollas]

There is something wrong with the aiidalab_qe_vibroscopy package. uv refuses to install it, erroring about RECORD file. I can install it via pip, but then mamba list is broken, with the same issue.
I think the web of packages here has gotten really out of hand...

EDIT: aiidalab/aiidalab-qe-vibroscopy#147

[danielhollas]

Blocked on aiidalab/aiidalab-qe-vibroscopy#147

[mikibonacci]

Hi @danielhollas , now should be ok. Can you see if it works by changing here the two github values of the first two packages (vibroscopy and muon), to be https://github.com/aiidalab/aiidalab-qe-vibroscopy@v1.2.1 and https://github.com/aiidalab/aiidalab-qe-muon@v1.0.1?
Thanks

[danielhollas]

Thanks @mikibonacci. We're now running into another issue

6.735   × No solution found when resolving dependencies:
6.735   ╰─▶ Because only aiida-vibroscopy==1.2.0 is available and
6.735       aiida-vibroscopy==1.2.0 depends on spglib<2.5, we can conclude that all
6.735       versions of aiida-vibroscopy depend on spglib<2.5.
6.735       And because aiidalab-qe-vibroscopy==1.2.0 depends on aiida-vibroscopy
6.735       and only aiidalab-qe-vibroscopy==1.2.0 is available, we can conclude
6.735       that all versions of aiidalab-qe-vibroscopy depend on spglib<2.5.
6.735       And because you require spglib==2.5.0 and aiidalab-qe-vibroscopy, we can
6.735       conclude that your requirements are unsatisfiable.

Do you know why the spglib lib is pinned to v2.5.0 in the Dockerfile? It's doesn't seem to be compatible with aiida-vibroscopy.

[mikibonacci]

Hi @danielhollas, I don't know. This is however strange: in the aiidalab/qe:v25.04.1, we have aiida-vibroscopy==1.2.0 and spglib==2.5.0, without issues. I wonder why now the issue is there. Maybe the order of packages installation changed?

Anyway, pinging @superstar54 and @edan-bainglass to see if they knows the reason of fixing the spglib version in the Dockerfile.

[danielhollas]

I wonder why now the issue is there.

That's because in this PR, I am installing all the packages together in a single pip install invocation so the resolver rightfully complains that the dependencies cannot be resolved.

In the previous version might work, but the python environment is not consistent (you can verify that by running pip check.

[danielhollas]

Do you know why the spglib lib is pinned to v2.5.0 in the Dockerfile?

The spglib pin has been added in #1282, but there's not comment about it in the PR.

[danielhollas]

@edan-bainglass FYI the discussion here gives a small taste for the dependency issues I briefly mentioned on Friday.

[edan-bainglass]

@danielhollas well, at least your PR is passing the build phase - can't say the same for the rest of the current PRs 🥲 I'm trying a few things on your PR to check the error in integration tests. Feel free to force push over these commits later once the PR is ready to merge.

Update

@danielhollas what I'm trying to do here is to disable the AiiDAlab error handler during integration tests, so that we can see the error in the screenshot if one arises. The mechanism I tried didn't take for some reason. In any case, I find it a bit inconvenient to have to wait on the build process just to test integration - unclear why this was designed this way. In #1422, I decouple these tests. My intention was to iterate on the integration tests to uncover the bug. However, I see that they pass, so the failure is coming from your PR 🤔 Peripherally, it would be great to understand how one can run selenium tests locally 🙂

Final update

Okay, looking at the screenshot error, the issue is related to an incompatible plugin w.r.t recent changes (#1389). Pinging @AndresOrtegaGuerrero @mikibonacci @superstar54 as a reminder to update all plugins.

[AndresOrtegaGuerrero]

I would be careful with this one , we install in .local , in case you are working in the server and there is a restart then when the server back the user has to install things manually. Will this PR avoid this ?

[danielhollas]

Peripherally, it would be great to understand how one can run selenium tests locally 🙂

I am pretty sure you if you build the qeapp image locally, then you should be able to run the Selenium tests in the same way as in CI. @yakutovicha definitely has done that with AWB so might have more insight.

Feel free to drive this PR forward and merge it, I will not have capacity to debug this further as QeApp is not really my area. I ran into issues with ARM build, which was failing due to euphonic package build so I needed to still install aiidalab-vibroscopy via pip. But that can be improved in a subsequent PR.

I would be careful with this one , we install in .local , in case you are working in the server and there is a restart then when the server back the user has to install things manually. Will this PR avoid this ?

I don't think I understand this. The packages are installed in the image itself in the last step in Dockerfile, so they should always be available? But maybe I am missing some context here. But in any case, things like this need to be documented in the Dockerfile.

@edan-bainglass were you able to manually test the QeApp image from this PR?

[danielhollas]

@AndresOrtegaGuerrero to clarify, this change is about the QeApp image, which is used to power the demo server, and has everything (QeApp and all plugins) pre-installed. Perhaps what you meant is when QeApp is installed manually from the AppStore? In that case I agree and the dependencies need to be installed in ~/.local so that they persist across restarts. This PR doesn't change that.

[edan-bainglass]

Peripherally, it would be great to understand how one can run selenium tests locally 🙂

I am pretty sure you if you build the qeapp image locally, then you should be able to run the Selenium tests in the same way as in CI. @yakutovicha definitely has done that with AWB so might have more insight.

Cool! I'll give it a try.

Feel free to drive this PR forward and merge it, I will not have capacity to debug this further as QeApp is not really my area. I ran into issues with ARM build, which was failing due to euphonic package build so I needed to still install aiidalab-vibroscopy via pip. But that can be improved in a subsequent PR.

Since the current state of this PR already reduces the size sufficiently to pass the build test, maybe we merge it already as is. We can continue to iterate in future PRs. For now, I've opened #1423 to handle the plugin compatibility issue, which should resolve the failed test here. I've rebased this PR on top of it. Let's see.

Update

Okay, #1423 indeed resolves one issue. The other is handled by #1417 (merged). This PR is now rebased on top and passes all tests. @superstar54 great if you can review this, as you put significant work on the image 🙏

[danielhollas]

This has been fixed

Suggested change

	# NOTE: uv refuses the install aiidalab-qe-vibroscopy due to invalid RECORD file,
	# pyproject.toml is likely invalid as it contains direct git dependency.

[edan-bainglass]

@danielhollas #1464 and #1465 fix the two issues that are failing your tests here. Both quick and easy. Great if you can review 🙏

As for this one, anything holding it back?

[danielhollas]

As for this one, anything holding it back?

I had a quick look and don't see any blockers here. You might want to re-verify that the changes here drop the image size as advertised in PR description.

[edan-bainglass]

@danielhollas thanks for the reviews 🙏 Hope you don't mind... I've rebased this PR on main to see if tests pass 🤞 For context, I'm interested in this one mostly due to the lifting of the spglib pin, which I guess is a driveby change here 😅

[danielhollas]

Not at all, please go ahead with any changes needed to get this merged. Yes, I did try to do some cleanup here, there are a lot of strange things in the Dockerfile currently.

[danielhollas]

@edan-bainglass @mikibonacci please verify that these repositories are in sync.

[edan-bainglass]

Not sure what you mean by in sync. As far as I can see, after moving vibroscopy under the AiiDAlab team, development has only continued there. Correct me if I'm wrong there @mikibonacci.

However, the opposite appears to be true for Muons 🤔

[danielhollas]

However, the opposite appears to be true for Muons 🤔

I guess this is what I meant 😉 When I looked originally, it looked like the active development was done on private repo's and not those that are under aiidalab org.

[edan-bainglass]

@mikibonacci can you clarify? 🙏