BUG: env vars & variant configs not set in (staging) build scripts?

[h-vetinari]

The docs for "Default environment variables set during the build process" say

rattler-build/docs/build_script.md

Lines 430 to 432 in f0ddcc9

	`PKG_VERSION`
	: The version of the package currently under construction.

but in a v1 translation I'm doing (including staging builds, if that's relevant), previously working code

for /f "tokens=1 delims=." %%a in ("%PKG_VERSION%") do (
  set "MAJOR_VERSION=%%a"
)

now evaluates as

for /F "tokens=1 delims=." %a in ("") do (set "MAJOR_VERSION=%a" ) 

in rattler-build - note the empty ("").

Not sure how many other default env vars might be affected.

Same effect on unix;

MAJOR_VERSION=$(echo ${PKG_VERSION} | cut -f1 -d".")

now evaluates as

 │ │ ++ echo
 │ │ ++ cut -f1 -d.
 │ │ + MAJOR_VERSION=

[h-vetinari]

Sigh,

TARGET:
- x86_64-conda-linux-gnu

in the variant configs (e.g. .ci_support/linux_64_.yaml) does also not get populated as an env var in the build scripts, apparently.

This code (after having overridden MAJOR_VERSION manually)

ln -sf ${PREFIX}/bin/clang-${MAJOR_VERSION} ${PREFIX}/bin/${TARGET}-clang
ln -sf ${PREFIX}/bin/clang-${MAJOR_VERSION} ${PREFIX}/bin/${TARGET}-clang++
ln -sf ${PREFIX}/bin/clang-${MAJOR_VERSION} ${PREFIX}/bin/${TARGET}-clang-cpp

now produces logs like

 │ │ + ln -sf $PREFIX/bin/clang-22 $PREFIX/bin/-clang
 │ │ + ln -sf $PREFIX/bin/clang-22 $PREFIX/bin/-clang++
 │ │ + ln -sf $PREFIX/bin/clang-22 $PREFIX/bin/-clang-cpp

I suspect that this must have something to do with the mechanisms around staging: / inherit:, otherwise this would break far too many recipes AFAICT.

[lucascolley]

reproducer for the PKG_VERSION problem:

#!/bin/bash
set -x
echo ${PKG_VERSION}

context:
  name: array-api-extra
  version: "0.10.0"
  python_min: '3.11'

source:
  url: https://pypi.org/packages/source/${{ name[0] }}/${{ name }}/array_api_extra-${{ version }}.tar.gz
  sha256: 0c5db2fb058e883f9b4b0c74f59c1e8c52c27bda1b183632645a551c1822dc0e

build:
  number: 0

# THIS WORKS
# package:
#   name: ${{ name|lower }}
#   version: ${{ version }}

# THIS DOESN'T WORK
recipe:
  name: ${{ name|lower }}
  version: ${{ version }}

outputs:
  - staging:
      name: foobar-stage
    build:
      script:
        file: build

  - package:
      name: foobar
    inherit: foobar-stage

[lucascolley]

in the variant configs (e.g. .ci_support/linux_64_.yaml) does also not get populated as an env var in the build scripts, apparently.

@Hofer-Julian @wolfv is this by design?

[lucascolley]

oh, so PKG_VERSION was deliberately removed from the staging context by @wolfv in #2449

[h-vetinari]

That PR says

This change fixes environment variable handling in staging cache contexts by removing misleading PKG_* variables that would otherwise inherit from the parent output's identity.

I disagree with this. PKG_VERSION makes complete sense even during the staging build. Most multi-output recipes have a single global version, but even if individual outputs set their own version, PKG_VERSION for the staging build is well-defined by taking the global

recipe:
  name: clangdev
  version: ${{ version }}

If you want to be super-technical about "staging caches don't produce packages", then you could rename it to RECIPE_VERSION, but of course that's even worse. Also, none of the handling w.r.t. PKG_* variables was mentioned anywhere in CEP 41.

There's certainly a discussion to be had about how to handle these things w.r.t. hashes, inheritance etc., but the presented work-around (to another user who's build was broken by #2449)

You can use

script:
  env:
    MY_VERSION_VARIABLE: ${{ version }}  # or what you define in context
  file: build

And the env var should be set accordingly.

is a non-solution. Please don't unilaterally break patterns without a CEP (and against your own docs I might add).

[wolfv]

Yeah IMO PKG_VERSION is ambigous and shouldn't be made available in the staging cache context. I think the workaround posted (forwarding the precise version via env is fine). There is not really a concept of "global" version in the recipe execution (we only know about the materialized outputs at that point), and the staging cache doesn't have a version field.

[h-vetinari]

There is not really a concept of "global" version in the recipe execution

Huh? Isn't that exactly what

recipe:
  name: clangdev
  version: ${{ version }}

is?

Yeah IMO PKG_VERSION is ambigous and shouldn't be made available in the staging cache context. I think the workaround posted (forwarding the precise version via

I can agree that PKG_HASH is pointless, but PKG_VERSION (the value from recipe.version) and PKG_NAME (the name: of the staging ouptut) are clearly non-ambiguous and need to be available.

What do you gain by removing these useful variables that will make it harder to translate and maintain recipes? I don't see any of the ambiguity you claim, and without that, this is just needlessly hostile IMO.

That goes even more so for the variant configs, whose variables shouldn't also have to be re-piped through script.env.

[lucascolley]

PKG_VERSION (the value from recipe.version) and PKG_NAME (the name: of the staging ouptut) are clearly non-ambiguous and need to be available.

recipe.version can be omitted, though: https://github.com/lucascolley/cpython/blob/pixi-flags/Tools/pixi-packages/recipe.yaml.

[h-vetinari]

recipe.version can be omitted, though

So then we make that mandatory? People who don't intend to use it can still put '0.0.0' or whatever there.

The reason that I'm pushing back here is that this change is taking something that's both working and sensible in the vast majority of cases, and forces awful hacks to restore that functionality, for no tangible benefit.

And that goes tenfold for the values from the variant configs, which I will absolutely refuse to pass through script.env. They're right there in the variant configs, just pass them through to the staging build. If you want, leave out those with multiple values (e.g. if python: ['3.11', '3.12', '3.13']), i.e. only forward keys whose values are of length one.

As a meta-comment: I understand the benefits of "move fast and fix things later where necessary", and I support the vast majority of innovations and changes that rattler & pixi have brought (though, unfortunately, you'll basically only hear about the disagreements...), in no small part by being able to move quickly. But the flipside of that is that where you broke too much too quickly, you have to be able to backpedal occasionally, to keep that approach viable for evolving an entire ecosystem. This is such a case.