Tracking usecases for getting intermediate build artifacts from build-dir

[ranger-ross]

Currently, the Cargo build-dir's internals are considered internal to Cargo.
Assuming a specific file structure is considered relying on Cargo's implementation details are not subject to Cargo's stability guarantees.

However there are usecases where tools need to use intermediate build artifacts from rustc, but not all files in build-dir have a stable way to access them so tools are forced to assume on the build-dir's file layout to work.
This is not ideal as when Cargo change's the build-dir internals it leads to breakage for tools (e.g. #15010)

This issue is serves a place to collect the various usecases to hopes that we could provide stable ways interact with build-dir artfiacts.

Known Usecases

• Code coverage (e.g. https://github.com/taiki-e/cargo-llvm-cov)
• Rustc Bootstrap
• Make it possible to run binaries produced by cargo directly.  #3670
• Cargo: produce deterministic filenames for build --test and test --no-run #1924
• Support exporting informations to execute cargo test artifact separately #17026
• Custom test runners
  • ykjit/yk
• Feature detection from tests (trybuild, see cargo build-dir-new-layout breaks feature detection dtolnay/trybuild#336)

[weihanglo]

In case that get buried in a closed PR.

@ltratt commented in #17272 (comment):

This has affected https://github.com/ykjit/yk because we rely on knowing where a cdylib is placed in one crate so we can pick it up in another (to run tests). Previously we'd end up with target/<profile>/libykcapi.so, which would always reference the most recent build of that shared librarie. With this PR it's now tucked away in something like build/ykcapi/0804c17cacc23d63/out/libykcapi.so where it's not clear we could easily work out (other than sorting by time on directories perhaps?) what the latest build is.

[I'm not saying that how we used this was pretty; and perhaps not even the least worst way of doing it! I'm very open to suggestions for how to improve things.]

[weihanglo]

Not yet checked but "libc CI red after the 07/30 nightly cargo layout change":
https://rust-lang.zulipchat.com/#narrow/channel/122651-general/topic/libc.20CI.20red.20after.20the.2007.2F30.20nightly.20cargo.20layout.20change/with/613934956

[illicitonion]

See dtolnay/trybuild#336 - trybuild currently uses the directory layout to find the features enabled in a crate so that it can reproduce them for the purposes of integration testing cargo-built binaries.