Ergonomics improvements for alternative backends

[josephlr]

As noted in many issues (#658, bevyengine/bevy#18047, https://users.rust-lang.org/t/getrandom-0-3-2-wasm-js-feature-issue/127584, n0-computer/iroh#3200), the use of a Rust --cfg flag can be unergonomic. See the current docs about this for context.

This is mostly due to how tooling (rustc, cargo, etc...) interact with RUSTFLAGS and the rustc command line. This is exacerbated by the fact that the wasm32-unknown-unknown target cannot have a default supported backend, as it is used in many environments (some of which do not have JavaScript). Leading to an unfortunate situation where, if someone wants to build any code which depends on getrandom targeting wasm32-unknown-unknown, they must:

• Modify Cargo.toml to include:

  [dependencies]
  getrandom = { version = "0.3", features = ["wasm_js"] }

• Modify .cargo/config.toml to include:

  [target.wasm32-unknown-unknown]
  rustflags = ['--cfg', 'getrandom_backend="wasm_js"']

This issue is to explore and evaluate approaches like #670 which seek to improve this story, either in 0.3 or in a future major release. It may turn out we end up keeping the current approach for a future major release, but we should explore all reasonable alternatives.

Requirements

A bunch of the ergonomics issues stem from the requirements for this crate. Some of these requirements might be debatable or less important, but ideally any solution would have the following properties (note that the current approach only has most of these):

• Without using any alternative backend, a user shouldn't have to do anything. All the supported backends should work, and the user should get good error messages if building on an unsupported platform.
• Overriding the default backend should be possible.
• You cannot declare/enable multiple alternative backends (for any given target).
• For security, it should be difficult/impossible for a crate deep in the dependency tree to override the default source of randomness.
• The approach should encourage "leaf" crates (i.e. binary crates) to select the backend, rather than core crates.
• wasm32-unknown-unknown should not be special-cased and should not call JavaScript by default (as the target makes it very clear that cannot be assumed).
• Any dependencies used by an alternative backend should not be present in the user's Cargo.lock if that backend is not being used.
• The approach should be similar or analogous to #[global_allocator], as we are defining a global source of randomness.
• We should not conflict with Tracking Issue for secure random data generation in std rust-lang/rust#130703, and ideally have a solution which works well even if that feature gets stabilized.

[josephlr]

From @newpavlov's comment: bevyengine/bevy#18047 (comment)

Make it so features = ["wasm_js"] activates getrandom_backend="wasm_js" by default

Personally, I am against it. The unfortunate reality is that people unconditionally enable such features in library crates for "convenience" despite strongly worded warnings against it. This breaks builds for users who target non-Web WASM. Even worse, just having wasm-bindgen in project's dependency tree can be problematic in such cases (we had one such user, unfortunately I can not find his comment right now).

I'm inclined to agree, I don't really want to special case wasm_js more than we already have. However, if it turns out this is a huge ecosystem-wide issue, we could consider it without it technically being a breaking change.

[alice-i-cecile]

Good analysis of the problem and requirements. I'm very sympathetic: you're really not handed the right set of tools to solve this directly.

Would it be viable to force users to declare a runtime RNG source using a static or the like? In Bevy, we had a similar set of requirements (only ever set by users, not libraries) for our global error handler and that was the flavor of solution we used.

Obviously this is perf-critical though, so you'd need to be careful with locking data structures.

[newpavlov]

Would it be viable to force users to declare a runtime RNG source using a static or the like?

This does not satisfy the fifth requirement since it allows for a random crate in the middle of your dependency tree to change the entropy source used by your whole application (assuming I understood your idea correctly).

[ChrisDenton]

Would having a new wasm32-unknown-web target help here? That way there's a target that explicitly has web APIs.

[alice-i-cecile]

This does not satisfy the fifth requirement since it allows for a random crate in the middle of your dependency tree to change the entropy source used by your whole application (assuming I understood your idea correctly).

Using something like a OnceLock, you could panic when it's set more than once. That would prevent this sort of attack, assuming you forced every user to explicitly declare an RNG source in their binary to use rand.

[Bluefinger]

Would having a new wasm32-unknown-web target help here? That way there's a target that explicitly has web APIs.

It would greatly benefit specifically for the web case, but not so much beyond that. Being able to define a entropy source backend for something like no_std would still have the ergonomic hit, which is something also trying to be resolved here.

[epage]

It would greatly benefit specifically for the web case, but not so much beyond that. Being able to define a entropy source backend for something like no_std would still have the ergonomic hit, which is something also trying to be resolved here.

I think there are two questions

• What should be done generally
• What can unblock a lot of people to reduce the pressure on resolving this

For the second, that depends a lot on the people being impacted by the change

[josephlr]

Would having a new wasm32-unknown-web target help here? That way there's a target that explicitly has web APIs.

Really what we need is Rust to have a wasm32-unknown-js target which just says: "Hey this is a wasm target which has JavaScript" (i.e. js-sys is valid to use on this target). We can check at runtime to see if we are on Node vs the Web or if a given api is supported. The main issue is that if we are just compiling for wasm32-unknown-unknown there's no way to check if JS exists, you just fail to build with inscrutable compiler errors.

Adding this target would help ecosystem-wide, as many low-level crates could now just implement the desired bindings to the necessary JS APIs without any flags/features/etc... when targeting wasm-unknown-js. @Bluefinger is correct that this only helps with web stuff though.

As to @epage's point, I think the vast majority of people having problems here are having problems due to wasm32-unknown-unknown being an unsupported target. Other folks targeting unsupported targets like x86_64-unknown-none or armv7a-none-eabi already have to build libcore/liballoc so they are much more used to dealing with RUSTFLAGS (and the requirement to set particular flags is less unexpected for folks working on those sorts of projects).

I would be fine with a stopgap solution which mostly only helped for wasm32-unknown-unknown users, be that adding a new target to rustc or some other JS-specific solution.

[bushrat011899]

Other folks targeting unsupported targets like x86_64-unknown-none or armv7a-none-eabi already have to build libcore/liballoc so they are much more used to dealing with RUSTFLAGS (and the requirement to set particular flags is less unexpected for folks working on those sorts of projects).

Just to clarify, there's actually a lot of really well-supported no_std targets now. You can compile and use x86_64-unknown-none without ever using nightly or touching RUSTFLAGS, and both core and alloc are provided pre-compiled just as they are on (almost) all targets. But I do agree that if you're in no_std you're typically already a more experienced Rust user and more likely to be ok with managing RUSTFLAGS.

I would be fine with a stopgap solution which mostly only helped for wasm32-unknown-unknown users, be that adding a new target to rustc or some other JS-specific solution.

I would also support this, but I see some frustrating combinatorics here when you consider wasm32v1-none and a likely wasm32v2-none in the future. These would also need js variants under this plan, so I could imagine this being pushed back against on that alone. Additionally, I think the reason this isn't done already is the compiler can't really enforce it.

But, I do think we could potentially argue for a target_feature = "js", similar to how atomics is enabled on Wasm today. This would be something users would be able to enable themselves, can be detected in cfg(...) for conditional dependencies and conditional compilation, and is more in line with existing Wasm workflows (where you may already be enabling the atomics feature for improved performance).

This is also forwards compatible with a potential future "web-only" target, since it would just always have target_feature = "js" enabled.

A more general solution I would prefer to see for no_std, Wasm on web, etc. would be for the current custom backend to be stabilized (maybe improved to include u32 and u64 methods for better performance?) and made the last-resort instead of a compilation error. This defers failure to select a backend to a linking error, and would allow users or library crates to just add a dependency which provides the appropriate custom implementation. This moves the problem of how to handle Wasm and web out from the getrandom crate itself and into the community, where they can experiment with tradeoffs.

[newpavlov]

One potential solution which may improve ergonomics for Web WASM is to use an extern function defined in wasm-bindgen. We could introduce a crate feature to getrandom which would enable a backend based on the function. The intention for this feature to be enabled ONLY by wasm-bindgen, i.e. it would have a getrandom feature and would define the extern function. Since wasm-bindgen is effectively a mandatory dependency for Web WASM, for most users getrandom would work automatically out of box (assuming the getrandom feature is enabled by default). Unfortunately, the wasm-bindgen developers haven't agreed to that (see this comment left by @daxpedda).

Overall, as I wrote elsewhere, we discussed the current design quite thoroughly before the v0.3 release and I am not keen on changing it drastically in getrandom v0.3. We may be open to an exception for Web WASM (e.g. like one described above), but it should not be a simple rehash of the js or custom crate feature which we had in v0.2. If you want to improve ergonomics in this area properly, I believe you should argue for improvements in the toolchain and std/core.

Additionally, here my answers to some comments from #670:

Checking it into git in my_project/.cargo/config.toml is not a good option because it will override ~/.cargo/config.toml, which would contain the very crucial -Zthreads=0 and -Zshare-generics=y for people using nightly.

While I agree that this situation is not ideal, I believe it's a toolchain problem and should be resolved by proposals like EXTRA_RUSTFLAGS. For now, IIRC you can overwrite project-local rustfalgs with the RUSTFLAGS environment variable. Finally, there is a nuclear option of using a patched version of getrandom tailored for Web WASM.

Do you need someone with expertise here to fix the implementation?

We need a proper design which satisfies the requirements listed in the OP. For now, I haven't seen anything new which we haven't discussed previously.

[bushrat011899]

Do you need someone with expertise here to fix the implementation?

We need a proper design which satisfies the requirements listed in the OP. For now, I haven't seen anything new which we haven't discussed previously.

I agree, let's make sure we're working to the requirements listed above so we're productively moving towards a solution.
Looking at #672 as a possible solution, here's my evaluation of it against these requirements.

Requirements

• Without using any alternative backend, a user shouldn't have to do anything. All the supported backends should work, and the user should get good error messages if building on an unsupported platform.

Met, the default behavior is identical to 0.3.3.

• Overriding the default backend should be possible.

Met, the current mechanism of selecting a custom backend is unchanged.

• You cannot declare/enable multiple alternative backends (for any given target).

Met, this will cause a linking error for the exact same reason the current custom backend does.

• For security, it should be difficult/impossible for a crate deep in the dependency tree to override the default source of randomness.

Met, a library can only provide a source of randomness if getrandom does not provide a default. If a malicious dependency declares a custom backend that conflicts with one defined by the user (or otherwise included in their dependencies), a linking error will be thrown highlighting the attack.

• The approach should encourage "leaf" crates (i.e. binary crates) to select the backend, rather than core crates.

Subjectively, I'd say met. While library crates can provide a backend, it can only be done in the case when getrandom has no backend available. Additionally, default features make providing a fallback backend an error, so library crates must actively choose to participate in the fallback system, increasing the likelihood they are participating correctly.

• wasm32-unknown-unknown should not be special-cased and should not call JavaScript by default (as the target makes it very clear that cannot be assumed).

Met, this PR is entirely agnostic to the target.

• Any dependencies used by an alternative backend should not be present in the user's Cargo.lock if that backend is not being used.

Met in so far as it's the responsibility of the backend provider to do so. This PR makes no changes to dependencies in getrandom.

• The approach should be similar or analogous to #[global_allocator], as we are defining a global source of randomness.

Currently it is exactly the existing custom infrastructure. The PR could be expanded upon to make that API cleaner, providing a macro similar to set_impl! from critical-section or a proc-macro like #[panic_handler]. But I believe this should be a separate item as it's more about improvements to custom than it is to allowing a fallback backend.

• We should not conflict with Tracking Issue for secure random data generation in std rust-lang/rust#130703, and ideally have a solution which works well even if that feature gets stabilized.

Met. If std provides an RNG implementation then the fallback becomes even more niche without any changes required by getrandom, users, or libraries.

If there are other requirements that haven't been laid out in the OP, or there's a disagreement about the PR's suitability, please let me know! This probably could be in the PR itself, but I think this discussion is important for the issue here.

[newpavlov]

If a malicious dependency declares a custom backend that conflicts with one defined by the user (or otherwise included in their dependencies), a linking error will be thrown highlighting the attack.

Relying on cryptic linking errors is not great. But I guess we can count it as "met".

While library crates can provide a backend, it can only be done in the case when getrandom has no backend available. Additionally, default features make providing a fallback backend an error, so library crates must actively choose to participate in the fallback system, increasing the likelihood they are participating correctly.

I expect that in practice with your proposal we will have library crates providing fallback backend for Web WASM to make it more "convenient" for users and "to reduce number of papercuts". And you would get a mess of cryptic linking errors if two or more such crates will be in your dependency tree. It's less of a concern with the current opt-in custom backend, since adding the extern function is not enough and users still have to enabled the cfg option.

So I would say the requirement is not met.

You could say it's the libraries fault for using the feature incorrectly, but it still would be a mess enabled by it.

[Bluefinger]

I expect that in practice with your proposal we will have library crates providing fallback backend for Web WASM to make it more "convenient" for users and "to reduce number of papercuts".

This is already happening with what we have at the moment 1, so I'm not sure why this is an argument against the proposal. If things are made easier/clearer from getrandom side on how to handle Web WASM, then library crates would feel less inclined to provide their own backend. Currently, it is easier to vendor a Web WASM getrandom backend than to use RUSTFLAGS and --cfg if you are a transient dependency. So by your own requirements, the current approach of getrandom is not fit for purpose either. If folks are deciding to side-step the issue, then you can't really blame them for taking the easier route. You have to meet your users where they are at.

[benfrankel]

Linking the issue for RUSTFLAGS merging: rust-lang/cargo#5376.

[benfrankel]

I've been struggling with this issue while working on bevy_new_2d, which is a template for Bevy games.

Basic requirements:

1. CI/CD should pass.
2. Local web builds should compile (e.g. cargo build --profile web --no-default-features --features web --target wasm32-unknown-unknown)

In order to satisfy the second requirement, we have to enable the wasm_js feature on getrandom (trivial) and add --cfg 'getrandom_backend="wasm_js"' to RUSTFLAGS. This can't be done via Cargo.toml, so it has to go in either a .cargo/config.toml file or the actual $RUSTFLAGS environment variable. As there's no way to inject environment variables into the user's cargo build command, we're left with two options:

1. Include a .cargo/config.toml file in the template.
2. Teach users to work around this issue themselves.

Including a .cargo/config.toml file in the template is problematic because:

1. It overwrites any rustflags the user has set in their ~/.cargo/config.toml.
2. If the user modifies .cargo/config.toml for personal configuration (which is common in Bevy projects), it leads to a situation where their local .cargo/config.toml used for local builds perpetually shadows their pushed .cargo/config.toml used by CI/CD. If the user then wants to edit the CI/CD version, they have to stash their local version, make the edits and commit, and then pop the stash.

Alternatively, we could include both a .cargo/config.toml and a .cargo/config_ci.toml that gets activated during CI/CD, but we would have to prepend a hash of that file to the Rust cache key to correctly invalidate the cache when it changes.

This all amounts to a major introduction of complexity into a template that's supposed to be friendly for new users. Similarly, it's unreasonable to try to ease new users into understanding the nuances of this getrandom issue + explain how rustflags priority works before they can even compile their own code.

---

Considering the tradeoffs, our best way forward seems to be to [patch] to a fork of getrandom that allows enabling the wasm_js backend via feature flag. I'm hoping this will only be a temporary solution, but there's a game jam very soon and web builds need to compile today.

[dhardy]

@benfrankel is it viable to target wasm32-wasip2 instead?

[Bluefinger]

@dhardy What's the status of WASI on browsers? Is this something being provided by browsers themselves or will it be an additional runtime to import/shim? Given the example provided by @benfrankel, that is for compiling/running a game on the browser, not for a WASM+WASI runtime like Wasmer. So likely targetting a WASI environment is not a viable solution.

[dhardy]

Currently, it is easier to vendor a Web WASM getrandom backend than to use RUSTFLAGS and --cfg if you are a transient dependency. So by your own requirements, the current approach of getrandom is not fit for purpose either.

Given that all solutions for WASM-JS are hacky, it may even be worth promoting this approach; i.e. users add something like the following to the binary's Cargo.toml:

[patch.crates-io]
getrandom = { version = "0.3", package = "getrandom-wasm-js" }

This has some major caveats (must be added in the binary's Cargo.toml, may prevent publishing (untested), cannot cfg to support multiple targets), so clearly it cannot be the only solution here.

... or, I suppose we could publish a getrandom-js crate which behaves exactly like getrandom except that it assumes JS on wasm32.

[dhardy]

Regarding using patch.crates-io to target an alternative getrandom crate:

• Without using any alternative backend, a user shouldn't have to do anything. All the supported backends should work, and the user should get good error messages if building on an unsupported platform.
• Overriding the default backend should be possible.
• You cannot declare/enable multiple alternative backends (for any given target).

We effectively are specifying a different backend with the patch. We don't need to support another backend-specifier.

• For security, it should be difficult/impossible for a crate deep in the dependency tree to override the default source of randomness.
• The approach should encourage "leaf" crates (i.e. binary crates) to select the backend, rather than core crates.

patch can only be specified at the top-level (binary) which satisfies these points.

• wasm32-unknown-unknown should not be special-cased and should not call JavaScript by default (as the target makes it very clear that cannot be assumed).

getrandom-js would explicitly special-case JS. This is acceptable.

• Any dependencies used by an alternative backend should not be present in the user's Cargo.lock if that backend is not being used.

This is not the case for crates using getrandom-js via patch while building on a different platform. This is acceptable however, since users of getrandom (non-patch) are unaffected.

---

The major caveat is that this solution is not applicable to all users, hence we also need another mechanism to specify usage of the WASM-JS backend. Given that the major concern here appears to be an easy way for users to use getrandom on web targets, this approach (publishing getrandom-js and recommending usage of patch.crates-io.getrandom) is worth considering.

The major caveat is that this solution is not applicable to all users

I'm not sure if this is the case?

[bushrat011899]

While I think the [patch] approach is the only viable option in the immediate future (<1 month), I don't think it's a good solution to promote. For one thing, if we have two crates, one which causes compiler errors and confusion on the browser, and another which is API compatible and doesn't have those issues, why would the community opt for the worse experience? I think it's more likely users would just use the "alternative" crate instead. Especially considering that's what uuid has already done.

[dhardy]

For one thing, if we have two crates, one which causes compiler errors and confusion on the browser, and another which is API compatible and doesn't have those issues, why would the community opt for the worse experience?

Because crates like rand need getrandom without being able to assume anything about the target platform.

@bushrat011899 the question should then be why we don't support JS on wasm32-unknown-unknown by default. There are two parts to that answer:

• wasm32-unknown-unknown does not imply web (or that any JS APIs are available). How much of a problem this is in practice I don't know, but see this comment.
• Since Cargo.lock must be the same on all platforms, crate dependency resolution does not depend on the target. wasm-bindgen pulls in many dependencies, which then end up in the Cargo.lock on all platforms. See this comment (unfortunately we don't have a dedicated issue for this topic).

So we can't please everyone.

[bushrat011899]

@dhardy I personally believe #672 actually does give everyone what they want by simply allowing better interop with 3rd party backends. This would allow the "should wasm32-unknown-unknown mean Web?" question to be answered by the user instead of getrandom.

I think when posed with a difficult question that has no easy answer, the best path forward is the one that yields control to those with more information (the user).

[NthTensor]

There's a movement currently (I believe coming from the rust project itself) away from wasm32-unknown-unknown for web crates and towards the no-std wasm32v1-none target. I would not like to see this crate become a barrier to that effort.

[dhardy]

This is a long issue, so I just opened #675 and #674; please discuss there (or open another issue if you have further proposals).

I also closed #672, though I guess one could still argue in its favour.

@NthTensor the real problem here is what to do on wasm32-unknown-unknown. Well specified targets are not an issue.

@josephlr despite the name, this issue really concerns only wasm32-unknown-unknown. Rename?

[Bluefinger]

@dhardy This actually concerns more than just wasm32-unknown-unknown, but also no_std platforms and being able to more ergonomically define backends for them, as the --cfg problems affect them too (such as wasm32v1-none, and more).

[dhardy]

@Bluefinger most people have been talking about wasm32-u-u. Let's not assume that one solution is the best for all cases. The big problem I see here is that getrandom is hard to use on a common platform.

[Bluefinger]

@dhardy Fair, but any solution here also will have to apply to wasm32v1-none, which is the no_std version of wasm32-unknown-unknown, so there are overlapping concerns as the ecosystem is making progress to migrate to wasm32v1-none over wasm32-unknown-unknown.

[newpavlov]

@Bluefinger @NthTensor
From the getrandom's point of view, wasm32v1-none is handled absolutely the same way as wasm32-unknown-unknown. We are not tied to the std "implementation" in any way (well, except the std::error::Error trait implementation, but it's not relevant). In other words, in this discussion everything said about wasm32-unknown-unknown also applies to wasm32v1-none.

[NthTensor]

Sorry, my post was in response to the mention of wasm32-wasip2. Recommending people use a specific target is a non-starter, for a lot of reasons, among them that it may hamper movement in the ecosystem from one target to another.