Multiple build with boa

[ffirmanff]

First off, hats off to @wolfv and other mamba/boa project members. This is a huge improvements in developer quality of life vs conda build.

I have a requirement to create multiple C++ builds in our pipeline using boa (e.g. build vs debug, static vs dynamic).
With conda build, I can specify the variants in the conda_build_config.yaml file like such:

buildtype:
  - debug
  - release

I can reference the buildtype as a variable in meta.yaml file directly like so:

build:
  string: 'build{{ buildtype }}'
  number: 0
  script_env:
    - CMAKE_BUILD_TYPE={{ buildtype }}

Conda build will take the vars and update the recipe accordingly.

However, if I were to do the same in boa's recipe.yaml, I find that the variables from the conda_build_config.yaml are not accessible, and really only useable as version references for packages that matched the variable name.

Peeking through the codes, seems boa parses out the recipe.yaml alone (without using any values from the variants), and replaces non-matching jinja2 variables with blank. Only later is the variants from conda_build_config.yaml used to apply the versions against matching packages.

1. Can someone please help explain the reasoning for changing the logic this extensively? I had read the reasoning for simplifying the recipe format, but this seems a bit drastic.
2. What is the recommendation for how I can create separate debug and release build of my packages?

[wolfv]

Hi @ffirmanff thanks for the kind words!

Indeed, I have an open PR to tackle this here: #261

I haven't yet gotten around to finish it -- if you have cycles to put into this that would be awesome.

You can do it today with a simple script that would change the YAML file (it's very easy to manipulate because it's proper YAML :)

One reason for changing some of the logic is that conda-build is evaluating the recipes recursively which makes it really slow. However, I think we could make this jinja-variant logic work without having to go down real recursions.

[ffirmanff]

Happy to contribute @wolfv !
I was trying to figure out where to make the changes, but got lost, so perhaps you can help clarify things a bit.
From what I understand:

1. The recipe.yaml is parsed and flattened here: https://github.com/mamba-org/boa/blob/main/boa/core/run_build.py#L492
2. But the variants from conda_build_config.yaml only gets applied here: https://github.com/mamba-org/boa/blob/main/boa/core/run_build.py#L500

What is the best way to do this? (apology if I have the terms wrong)

• I can't apply the build variants in step 1, because if there are multiple variants then I will need to keep multiple recipe variants, and carry them into step 2.
• Alternatively, do not (jinja) parse the recipe.yaml in step 1, so that we keep all the jinja2 references. But in step 2 when the build variant permutations are being generated, a copy of the 'recipe.yaml' is parsed using the variant permutation variables first, before applying the variant permutation.

[wolfv]

Awesome, yes, I think those were exactly the problems that I was hitting.

I think I was attempting to keep the un-evaluated jinja in a adapted string, and then apply it with the variant evaluation.

But it feels like we need another "loop" where we first check the recipe for all used jinja variables. The easiest would be to use "regular" jinja to insert those in the "text" form of the recipe (instead of carrying them with us in that modified string.

I think we can add a step where we try to find the jinja variables, then add them to the context and render the recipe as normal, execute the build, and repeat for other jinja options.

[wolfv]

^ That would be different from what I've attempted in #261 but maybe more straightforward to implement.

[ffirmanff]

Sorry haven't had a chance to look into this, been a bit tied up at work, but will hopefully come back next week.