Tracking Issue for `-Zrustdoc-mergeable-info`

[weihanglo]

Summary

RFC: rust-lang/rfcs#3662 (rustdoc focused)
rust-lang/rust issue: rust-lang/rust#130676
Implementation: #16167, #16309
Documentation: https://doc.rust-lang.org/nightly/cargo/reference/unstable.html#rustdoc-mergeable-info

Leverage rustdoc's mergeable crate info, so that cargo doc can merge cross-crate information (like the search index, source files index) from separate output directories, run rustdoc in parallel, and avoid (sometimes) unexpected documentation merge.

Unresolved Issues

• cargo clean support
• It would be a breaking change, as cargo doc already exposes the legacy rustdoc --merge=shared behavior (rustdoc artifacts are additive) it is now additive
• Should rustdoc --merge=finalize be modeled in Cargo's "unit of execution" model (a.k.a. build graph)
  • Currently in Cargo, a unit in the unit graph is tied to a Command execution, for example rustc or rustdoc. rustdoc --merge=finialize is apparently a "unit of execution". However, the current Unit struct is designed for compiler invocation, which have features, rustflags and Cargo target information. If we don't model this invocation in unit of work model, a special rebuild detection mechanism may need to be implemented. It might be worth a refactor if we want to support both this and Support -Zno-link #9019, and perhaps doctest execution in the future.
• Race condition between the build holding the build-dir/target-dir locks and the merge step re-acquiring those locks, feat: support for rustdoc mergeable cross-crate info #16309 (comment) (would be resolved if merge was in the unit graph)
• Targets are merged serially, feat: support for rustdoc mergeable cross-crate info #16309 (comment) (would be resolved if merge was in the unit graph)
• Should all RUSTDOCFLAGS apply to rustdoc --merge=finalize call?
  • Some probably only need to pass to the --merge=finalize invocation, like --extend-css?
• rustdoc ICE when building different workspace member in the same workspace
  • feat: support for rustdoc mergeable cross-crate info #16309 (comment)
  • Maybe be a rustdoc bug. Putting it here for awareness, since it blocks the stabilization on Cargo side.
  • Fixed by rustdoc: fix bugs with search aliases and merging rust#149700

Future Extensions

• Ability to turn off cargo doc invocations from being additive
  • If Cargo provides a flag, say --config doc.merge=on|off, how to deal with that user switching between doc.merge=off and doc.merge=on? Should the pre-existing HTML file be kept or removed?
• crate-info JSON file name collision. Same as below
  • rustdoc-json: multiple package versions clobber their output paths rust#142370
  • cargo rustdoc --output-format=json sometimes doesn't rebuild for different versions of same crate name. #16291

About tracking issues

Tracking issues are used to record the overall progress of implementation.
They are also used as hubs connecting to other relevant issues, e.g., bugs or open design questions.
A tracking issue is however not meant for large scale discussion, questions, or bug reports about a feature.
Instead, open a dedicated issue for the specific matter and add the relevant feature gate label.

[ranger-ross]

In the current implementation, -Zrustdoc-mergeable-info only supports the new build-dir layout.

However, its possible to run it using the old layout and get a broken target.

cargo +nightly -Zrustdoc-mergeable-info doc
...

tree target
target
├── debug
│   ├── build
│   │   ├── foo-9c20e08689afe5f0
│   │   │   ├── invoked.timestamp
│   │   │   ├── out
│   │   │   ├── output
│   │   │   ├── root-output
│   │   │   └── stderr
│   │   └── foo-d86ccb5386f50cd9
│   │       └── deps
│   │           └── foo.json
....
            ^^^ We get both new and old layouts mixed together

I think -Zrustdoc-mergeable-info should implicitly enable -Zbuild-dir-new-layout in the same way that -Zfine-grain-locking does.

cargo/src/cargo/core/features.rs

Lines 1529 to 1534 in a793cd1

	fn implicitly_enable_features_if_needed(&mut self) {
	if self.fine_grain_locking && !self.build_dir_new_layout {
	debug!("-Zbuild-dir-new-layout implicitly enabled by -Zfine-grain-locking");
	self.build_dir_new_layout = true;
	}
	}