redesign stage 0 std

[onur-ozkan]

Summary

This PR changes how bootstrap builds the stage 1 compiler by switching to precompiled stage 0 standard library instead of building the in-tree one. The goal was to update bootstrap to use the beta standard library at stage 0 rather than compiling it from source (see the motivation at rust-lang/compiler-team#619).

Previously, to build a stage 1 compiler bootstrap followed this path:

download stage0 compiler -> build in-tree std -> compile stage1 compiler with in-tree std

With this PR, the new path is:

download stage0 compiler -> compile stage1 compiler with precompiled stage0 std

This also means that cfg(bootstrap)/cfg(not(bootstrap)) is no longer needed for library development.

Building "library"

Since stage0 std is no longer in-tree x build/test/check library --stage 0 is now no-op. The minimum supported stage to build std is now 1. For the same reason, default stage values in the library profile is no longer 0.

Because building the in-tree library now requires a stage1 compiler, I highly recommend library developers to enable download-rustc to speed up compilation time.

---

If you encounter a bug or unexpected results please open a topic in the #t-infra/bootstrap Zulip channel or create a bootstrap issue.

Blocked on #122709

[rustbot]

This PR modifies src/bootstrap/src/core/config.

If appropriate, please update CONFIG_CHANGE_HISTORY in src/bootstrap/src/utils/change_tracker.rs.

[onur-ozkan]

@rustbot ready

r? bootstrap
cc @rust-lang/libs

[onur-ozkan]

@rustbot author (currently stage 2 std is copied from stage 1 and this behaviour should change with the beta std change)

[Kobzol]

I tried download-rustc = true again. When I do x build (from scratch, without pre-existing build directory), it now builds a stage1 libstd, but it still builds a stage2 sysroot, which is a bit confusing to me, I would expect that it uses the downloaded rustc as stage1, builds stage1 libstd and assembles a stage1 sysroot. But I think that this behavior is pre-existing.

However, when I actually tried to use the stage2 sysroot, it fails with this:

   Compiling hello-world v0.1.0 (/projects/personal/rust/rust/crates/hello-world)
error[E0464]: multiple candidates for `dylib` dependency `std` found
  |
  = note: candidate #1: /projects/personal/rust/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-11d409dc53867953.so
  = note: candidate #2: /projects/personal/rust/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-168b2af3cdbd9a8b.so

error[E0464]: multiple candidates for `dylib` dependency `std` found
 --> src/main.rs:3:5
  |
3 | use std::path::PathBuf;
  |     ^^^
  |
  = note: candidate #1: /projects/personal/rust/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-11d409dc53867953.so
  = note: candidate #2: /projects/personal/rust/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-168b2af3cdbd9a8b.so

It seems like multiple standard libraries are being copied to the stage2 sysroot.

[onur-ozkan]

I tried download-rustc = true again. When I do x build (from scratch, without pre-existing build directory), it now builds a stage1 libstd, but it still builds a stage2 sysroot, which is a bit confusing to me, I would expect that it uses the downloaded rustc as stage1, builds stage1 libstd and assembles a stage1 sysroot. But I think that this behavior is pre-existing.

However, when I actually tried to use the stage2 sysroot, it fails with this:

   Compiling hello-world v0.1.0 (/projects/personal/rust/rust/crates/hello-world)
error[E0464]: multiple candidates for `dylib` dependency `std` found
  |
  = note: candidate #1: /projects/personal/rust/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-11d409dc53867953.so
  = note: candidate #2: /projects/personal/rust/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-168b2af3cdbd9a8b.so

error[E0464]: multiple candidates for `dylib` dependency `std` found
 --> src/main.rs:3:5
  |
3 | use std::path::PathBuf;
  |     ^^^
  |
  = note: candidate #1: /projects/personal/rust/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-11d409dc53867953.so
  = note: candidate #2: /projects/personal/rust/rust/build/x86_64-unknown-linux-gnu/stage2/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-168b2af3cdbd9a8b.so

It seems like multiple standard libraries are being copied to the stage2 sysroot.

Thanks for testing it! Can you try again? It should be working now.

[Kobzol]

Thanks! It indeed seems to work now, and changes in stdlib are reflected in the stage2 toolchain. Given the amount of if builder.download_rustc() checks in bootstrap, I expect that there will be some other rough edges, but the basic use-case of "download rustc when you only want to modify the library" seems to work now 👍

One thing I have noticed is that if I modify stdlib and then try to rebuild a crate using the stage2 toolchain, cargo doesn't notice that the stdlib has been invalidated, and I have to remove the target directory in order to force recompilation and see the new stdlib code in effect. I'm not sure if this happened before - perhaps not, because "naive" modifications of the stdlib also rebuilt the compiler before? Anyway, it's not a blocking concern, just something that we might want to write down somewhere, as it can be a bit surprising behavior.

[onur-ozkan]

One thing I have noticed is that if I modify stdlib and then try to rebuild a crate using the stage2 toolchain, cargo doesn't notice that the stdlib has been invalidated, and I have to remove the target directory in order to force recompilation and see the new stdlib code in effect. I'm not sure if this happened before - perhaps not, because "naive" modifications of the stdlib also rebuilt the compiler before? Anyway, it's not a blocking concern, just something that we might want to write down somewhere, as it can be a bit surprising behavior.

That happens on master even without CI-rustc enabled.

[jieyouxu]

Thanks for working on this! The changes themselves look good to me. I have further (manual) testing I would like to do on Monday to exercise some more flows, especially related to dist flows, build cache invalidations and test flows.

(Purely illustrative .oO)

[bors]

☔ The latest upstream changes (presumably #136974) made this pull request unmergeable. Please resolve the merge conflicts.

[onur-ozkan]

Rebased.