Building large packages with the default settings for spack can exhaust /tmp

[marcpaterno]

While trying to build a large package (gcc 13) on scisoftbuild02, my build failed because it filled /tmp. Also, having filled /tmp, things go wrong for other users.

This is a known feature of the default configuration for spack. It would be great if we could change this (perhaps in our fork of the spack repository?). If we can not, or should not, change this, we should have our introduction to setting up spack explain what to do to put the files generated by spack into a better place.

[knoepfel]

@marcpaterno, to override the default settings in Spack, the following can be added to user configuration using spack config edit config:

config:
  ...
  # The build stage can be purged with `spack clean --stage` and
  # `spack clean -a`, so it is important that the specified directory uniquely
  # identifies Spack staging to avoid accidentally wiping out non-Spack work.
  build_stage:
    - $spack/var/spack/stage            # <- favor location within spack installation
    - $tempdir/$user/spack-stage
    - $user_cache_path/stage

This assumes that spack is installed on a disk that can support a multi-GB build.

We should document this somewhere.