fix(env): cargo:rerun-if-env-changed doesn't work with env configuration #14058

heisen-li · 2024-06-13T11:41:40Z

What does this PR try to resolve?

Fixes #10358
Fixes #8693

Additional information

Due to my stupidity, I screwed up the previous PR. I didn't know what to do with it, so I created a new PR.

r?@epage

src/cargo/core/compiler/fingerprint/mod.rs

src/cargo/core/compiler/build_context/target_info.rs

src/cargo/core/compiler/fingerprint/mod.rs

epage · 2024-06-26T13:35:22Z

btw I believe this would also fix #8693.

This would likely need to revert #12482.

src/cargo/core/compiler/fingerprint/mod.rs

tests/testsuite/build_script_env.rs

src/cargo/core/compiler/fingerprint/mod.rs

tests/testsuite/build_script_env.rs

src/cargo/core/compiler/fingerprint/mod.rs

tests/testsuite/build_script_env.rs

epage · 2024-07-22T20:55:59Z

tests/testsuite/build_script_env.rs

+    p.change_file(
+        ".cargo/config.toml",
+        r#"
+            [env]
+            OUT_DIR = "somedir"
+        "#,
+    );


The intention of this test is for detecting changes made by cargo, and not the user. Us manually setting one runs counter to that

I know this is for #8693 and planning to revert #12482. However Cargo doesn't really respect env vars that are set by Cargo internally. Why this should recompile?
This fixes a confusion with the other new confusion introduced.

I'm not too sure what the concern is.

Setting env.OUR_DIR has no effect on build script execution.

Oh, that. The intent of my comment was to say that we should not test this with [env] but making a change that causes the cargo-provided env to change.

weihanglo · 2024-07-23T18:06:24Z

tests/testsuite/build_script_env.rs

+            "build.rs",
+            r#"
+fn main() {
+    println!("cargo:rerun-if-env-changed=foo");


double colons syntax is preferrable

Suggested change

println!("cargo:rerun-if-env-changed=foo");

println!("cargo::rerun-if-env-changed=foo");

weihanglo · 2024-07-23T18:19:06Z

tests/testsuite/build_script_env.rs

-                    }
-                }
-            "#,
+fn main() {


This formatting change shouldn't have happened, or should merge into the first commit adding the test.

weihanglo · 2024-07-23T18:19:43Z

tests/testsuite/build_script_env.rs

-
+[COMPILING] foo v0.1.0 ([ROOT]/foo)
+[ERROR] failed to run custom build command for `foo v0.1.0 ([ROOT]/foo)`
+...


We should better check if our assertion actually causes the error message. Perhaps add an assertion message assert!(..., "rerun due to FOO changed") and see if the message contain it.

weihanglo · 2024-07-23T18:21:44Z

tests/testsuite/build_script_env.rs

    let p = project()
        .file("Cargo.toml", &basic_manifest("foo", "0.1.0"))
        .file("src/main.rs", "fn main() {}")
        .file(
            ".cargo/config.toml",
            r#"
                [env]
-                FOO = "good"
+                Foo = "good"


Could you merge this into the commit that the test first appeared?

weihanglo · 2024-07-23T18:42:20Z

tests/testsuite/build_script_env.rs

+    p.change_file(
+        ".cargo/config.toml",
+        r#"
+            [env]
+            OUT_DIR = "somedir"
+        "#,
+    );


I know this is for #8693 and planning to revert #12482. However Cargo doesn't really respect env vars that are set by Cargo internally. Why this should recompile?
This fixes a confusion with the other new confusion introduced.

weihanglo · 2024-07-23T19:24:24Z

src/cargo/core/compiler/fingerprint/mod.rs

+        } else {
+            envs.get(key).and_then(|v| v.to_owned())
+        }
+        .or(env::var_os(key))


Lazy evaluation

Suggested change

.or(env::var_os(key))

.or_else(|| env::var_os(key))

weihanglo · 2024-07-23T19:31:37Z

src/cargo/core/compiler/fingerprint/mod.rs

+            envs: &BTreeMap<String, Option<OsString>>,
+        ) -> Option<OsString> {
+            let upper_case_key: String = key.to_uppercase();
+            for (k, v) in envs {


This conversion seems to be in a deeply nested loop. M rerun_if_env_changed * N [env] keys on Windows. We should probably find a way to do it once and for all.

This doesn't really seem to be a performance issue for general usage though.

A small fix would be to check the map with the key as-is, and then only check for the case-insensitive if that fails. 99% of the time the case will match. EnvConfig::get_env_os does something similar.

weihanglo · 2024-07-23T19:35:33Z

src/cargo/core/compiler/build_context/target_info.rs

@@ -588,6 +590,10 @@ impl TargetInfo {
            .iter()
            .any(|sup| sup.as_str() == split.as_str())
    }
+
+    pub fn get_target_envs(&self) -> &BTreeMap<String, Option<OsString>> {
+        return self.crate_type_process.get_envs();


Accessing crate_type_process seems like a hacky shortcut to me. The process is for learning target info from rustc, but fingerprint is for actual rustc compilation. One extension point for putting it here is for supporting #10273, though I believe that will need other rounds of env var system overhual in Cargo.

I would agree, the chance of this having unintended consequences seems kinda high since it isn't obvious when looking at TargetInfo that this would also influence build-script environment fingerprinting.

Would it be possible to rewrite apply_env_config so that there is a separate function that yields a map of resolved config values? Then the code in the fingerprint module could call that and collect the results into a map that it could then pass into the closure.

(I also wonder if it ends up creating a new map, if it could pre-normalize it on Windows to avoid the inner case-check loop.)

ehuss · 2024-07-24T23:01:13Z

src/cargo/core/compiler/fingerprint/mod.rs

-    // TODO: This is allowed at this moment. Should figure out if it makes
-    // sense if permitting to read env from the config system.


Instead of removing this, can we have a comment explaining why the allow can't be removed? It's not obvious to me that there is some fundamental problem that is preventing this.

I presume the problem is that the closure cannot access the GlobalContext, and thus is unable to access the env snapshot, and thus must access the raw environment directly? It seems like there are still options we can consider, such as wrapping the environment in Arc or something like that?

such as wrapping the environment in Arc

Agree Arc is the way to resolve this.

ehuss · 2024-07-24T23:06:37Z

src/cargo/core/compiler/fingerprint/mod.rs

+            envs: &BTreeMap<String, Option<OsString>>,
+        ) -> Option<OsString> {
+            let upper_case_key: String = key.to_uppercase();
+            for (k, v) in envs {


A small fix would be to check the map with the key as-is, and then only check for the case-insensitive if that fails. 99% of the time the case will match. EnvConfig::get_env_os does something similar.

ehuss · 2024-07-24T23:08:20Z

I'm not clear, how does this fix #8693?

MabezDev · 2024-09-11T18:51:47Z

@heisen-li Do you plan on finishing this up? It would be great to see this land.

heisen-li · 2024-09-12T07:32:44Z

@heisen-li Do you plan on finishing this up? It would be great to see this land.

I may not have time for this for a while due to changes in my job and the difficulty of modifying this PR which may take more time for me.If you or anyone else is interested feel free to finish and close this.If I have more time I would be more than happy to continue this work.Sorry about that.

bors · 2024-10-08T21:47:18Z

☔ The latest upstream changes (presumably #14137) made this pull request unmergeable. Please resolve the merge conflicts.

weihanglo · 2024-11-01T23:34:32Z

No worries @heisen-li. Your help here is appreciated. I'll close this in favor of #14756. Thanks you!

rustbot assigned epage Jun 13, 2024

rustbot added A-cfg-expr Area: Platform cfg expressions A-rebuild-detection Area: rebuild detection and fingerprinting S-waiting-on-review Status: Awaiting review from the assignee but also interested parties. labels Jun 13, 2024

heisen-li changed the title ~~Config env changefix(env): cargo:rerun-if-env-changed doesn't work with env configuration~~ fix(env): cargo:rerun-if-env-changed doesn't work with env configuration Jun 13, 2024

epage reviewed Jun 13, 2024

View reviewed changes

src/cargo/core/compiler/fingerprint/mod.rs Outdated Show resolved Hide resolved

epage reviewed Jun 13, 2024

View reviewed changes

src/cargo/core/compiler/fingerprint/mod.rs Outdated Show resolved Hide resolved

epage reviewed Jun 13, 2024

View reviewed changes

src/cargo/core/compiler/build_context/target_info.rs Outdated Show resolved Hide resolved

epage reviewed Jun 13, 2024

View reviewed changes

src/cargo/core/compiler/fingerprint/mod.rs Show resolved Hide resolved

heisen-li force-pushed the config_env_change branch from 0a264a9 to 877da76 Compare June 14, 2024 09:01

epage reviewed Jun 14, 2024

View reviewed changes

src/cargo/core/compiler/fingerprint/mod.rs Outdated Show resolved Hide resolved

heisen-li force-pushed the config_env_change branch from 513422b to cbe3643 Compare June 25, 2024 07:49

epage reviewed Jun 26, 2024

View reviewed changes

src/cargo/core/compiler/fingerprint/mod.rs Outdated Show resolved Hide resolved

epage reviewed Jun 26, 2024

View reviewed changes

tests/testsuite/build_script_env.rs Outdated Show resolved Hide resolved

heisen-li force-pushed the config_env_change branch 4 times, most recently from 08bbb58 to d19af49 Compare July 1, 2024 09:16

epage reviewed Jul 5, 2024

View reviewed changes

src/cargo/core/compiler/fingerprint/mod.rs Outdated Show resolved Hide resolved

epage reviewed Jul 5, 2024

View reviewed changes

tests/testsuite/build_script_env.rs Show resolved Hide resolved

epage reviewed Jul 5, 2024

View reviewed changes

src/cargo/core/compiler/fingerprint/mod.rs Show resolved Hide resolved

epage added the T-cargo Team: Cargo label Jul 5, 2024

epage reviewed Jul 5, 2024

View reviewed changes

tests/testsuite/build_script_env.rs Show resolved Hide resolved

test: add Add rerun-if-env-changed test example

e0f523a

heisen-li force-pushed the config_env_change branch from d19af49 to 83eaae3 Compare July 18, 2024 09:08

rustbot added the A-documenting-cargo-itself Area: Cargo's documentation label Jul 18, 2024

heisen-li added 2 commits July 18, 2024 17:25

fix(env):Get environment variables from the latest environment config

f91dc78

revert previous documentation (PR: 12482)

fd2e75e

heisen-li force-pushed the config_env_change branch from 83eaae3 to fd2e75e Compare July 18, 2024 09:25

epage reviewed Jul 22, 2024

View reviewed changes

weihanglo reviewed Jul 23, 2024

View reviewed changes

ehuss reviewed Jul 24, 2024

View reviewed changes

weihanglo mentioned this pull request Aug 4, 2024

Build scripts don't rerun if [env] in config.toml is changed #14350

Closed

MabezDev mentioned this pull request Aug 23, 2024

RFC: We need some sort of configuration system esp-rs/esp-hal#1111

Closed

MabezDev mentioned this pull request Sep 13, 2024

esp-config esp-rs/esp-hal#2156

Merged

6 tasks

linyihai mentioned this pull request Oct 31, 2024

fix: env table config can't trigger rebuild with rerun-if-env-changed. #14756

Open

weihanglo closed this Nov 1, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(env): cargo:rerun-if-env-changed doesn't work with env configuration #14058

fix(env): cargo:rerun-if-env-changed doesn't work with env configuration #14058

heisen-li commented Jun 13, 2024 •

edited

Loading

epage commented Jun 26, 2024

epage Jul 22, 2024

weihanglo Jul 23, 2024

epage Jul 23, 2024

weihanglo Jul 23, 2024

epage Jul 23, 2024

weihanglo Jul 23, 2024

weihanglo Jul 23, 2024

weihanglo Jul 23, 2024

weihanglo Jul 23, 2024

weihanglo Jul 23, 2024

weihanglo Jul 23, 2024

weihanglo Jul 23, 2024

ehuss Jul 24, 2024

weihanglo Jul 23, 2024

ehuss Jul 24, 2024 •

edited

Loading

ehuss Jul 24, 2024

weihanglo Jul 24, 2024

ehuss Jul 24, 2024

ehuss commented Jul 24, 2024

MabezDev commented Sep 11, 2024

heisen-li commented Sep 12, 2024

bors commented Oct 8, 2024

weihanglo commented Nov 1, 2024

	println!("cargo:rerun-if-env-changed=foo");
	println!("cargo::rerun-if-env-changed=foo");

		// TODO: This is allowed at this moment. Should figure out if it makes
		// sense if permitting to read env from the config system.

fix(env): cargo:rerun-if-env-changed doesn't work with env configuration #14058

fix(env): cargo:rerun-if-env-changed doesn't work with env configuration #14058

Conversation

heisen-li commented Jun 13, 2024 • edited Loading

What does this PR try to resolve?

Additional information

epage commented Jun 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ehuss Jul 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ehuss commented Jul 24, 2024

MabezDev commented Sep 11, 2024

heisen-li commented Sep 12, 2024

bors commented Oct 8, 2024

weihanglo commented Nov 1, 2024

heisen-li commented Jun 13, 2024 •

edited

Loading

ehuss Jul 24, 2024 •

edited

Loading