More principled tempdir usage by and between rustc, rustdoc and compiletest

[scottmcm]

"Specifically, it's somewhere here:

rust/compiler/rustc_codegen_ssa/src/back/link.rs

Lines 102 to 107 in cbfdf0b

	if outputs.outputs.should_link() {
	let tmpdir = TempFileBuilder::new()
	.prefix("rustc")
	.tempdir()
	.unwrap_or_else(|error| sess.dcx().emit_fatal(errors::CreateTempDir { error }));
	let path = MaybeTempDir::new(tmpdir, sess.opts.cg.save_temps);

"but yeah, in any case, we probably should override that, to point it under build/test/$test_suite_name/$test_suite_revisioned_compare_moded/__temp/
"(E.g. by setting TMP_DIR/TEMP/TMP or whatever the env vars were called)
"Can you open an E-needs-investigation issue?" ~ @jieyouxu https://discord.com/channels/273534239310479360/957720175619215380/1349912066772963469

Spotted in #138157 (comment), the bors job failed with

--- stderr -------------------------------
error: couldn't create a temp dir: Access is denied. (os error 5) at path "C:\\Users\\RUNNER~1\\AppData\\Local\\Temp\\rustcfzdxGW"

error: aborting due to 1 previous error

It would be nice if those stayed in the build/target directory, which is more likely to have a defender exclusion (and thus more likely to not have that access error). It would also be helpful to stay on the Dev Drive on Windows 11 if people have that set up, rather than use C: which typically has the most extra filesystem access costs (from filters and such).

And it'd just be nice not to have a bazillion of these left over after running tests a bunch :)

[jieyouxu]

Yeah, this behavior is definitely a bit strange and sub-optimal. Addressing this properly would need fixes from two directions:

1. The compiler itself needs to be modified to consistently use temp dirs that can be controlled by the user (through flags and whatever).
2. compiletest should also specify temp dirs to be somewhere under the build directory, and not random places e.g. under an entirely different drive,

[scottmcm]

Which reminds me I should go clean this out again. I have pages and pages of these:

(All of them empty, at least of the ones I randomly checked.)

EDIT: I checked with find, apparently 10390 empty rust* directories in temp.

[jieyouxu]

@scottmcm that might not be rustc, that might be rustdoc:

rust/src/librustdoc/doctest.rs

Lines 133 to 135 in addae07

	fn get_doctest_dir() -> io::Result<TempDir> {
	TempFileBuilder::new().prefix("rustdoctest").tempdir()
	}

[jieyouxu]

I'm going to tag this with rustdoc too, because while we also should control the tempdir env vars explicitly from compiletest, rustdoc may need to be fixed itself like rustc (I haven't looked deeply, lmw if this is some compiletest-only issue).

[GuillaumeGomez]

The rustdoctest* issue should be fixed with #140706.

[bridiver]

1. The compiler itself needs to be modified to consistently use temp dirs that can be controlled by the user (through flags and whatever).

I really think it's problematic that the compiler uses temp dirs at all. Why would any build artifacts not go in the build output dir? Even if you can adjust the actual directory used for temp dir, it's still potentially a problem if it creates randomly named artifacts that don't get cleaned up. I would think you would only use temp files and create them somewhere in the output directory. I have never run into this problem with other compilers.

To be clear this is the kind of issue we run into rust-lang/cargo#4350 and this happens even when specifying a target dir