RFC: Disable niche layout optimization on enum discriminants #3803

madsmtm · 2025-04-21T20:36:15Z

Allow #[repr(_, open)] on field-less (C-style) enums to allow the enum to contain unknown variants, which enables using the enum directly in FFI.

Enums with the open modifier cannot be matched on exhaustively without the use of a wildcard arm. This feature is similar but distinct from the #[non_exhaustive] attribute, since it works on an ABI level and applies in the defining module as well.

Based on @thomcc's initial work on a similar RFC. This kind of functionality has also been discussed in the past on the internals forum, more recently on Zulip and possibly many more times before that.

Pretty sure this is T-lang purview:
@rustbot label T-lang

Please try to create inline comments for discussions to keep this RFC manageable for me.

Rendered.

madsmtm · 2025-04-21T20:38:40Z

text/3803-enum-repr-no-niches.md

+## Syntax
+
+Current options and sub-options:
+1. Use `#[repr(_, something_here)]`.
+    Advantage: The feature semantically modifies the layout/representation.
+    1. `#[repr(_, no_niches)]`.
+        Advantage: Very precise in what it does.
+        Disadvantage: Uses "niche" which is a niche term.
+    2. `#[repr(_, non_exhaustive)]`.
+        Advantage: Clear that it implies `#[non_exhaustive]`.
+        Disadvantage: Might be confusing for users which kind of `non_exhaustive` they should use.
+    3. `#[repr(_, abi_stable)]`.
+    3. `#[repr(_, all_bits_valid)]`.
+    4. `#[repr(_, discriminant = no_niches)]` to mirror [RFC 3659](https://github.com/rust-lang/rfcs/pull/3659).
+2. Use `#[non_exhaustive(something_here)]`.
+    Advantage: Might be easier to explain the effect to users ("this works just like `#[non_exhaustive]`, except stronger").
+    Advantage: Might align better with future additions to `#[non_exhaustive]`, such as [`#[non_exhaustive(pub)]`](https://internals.rust-lang.org/t/pre-rfc-relaxed-non-exhaustive-structs/11977).
+    1. `#[non_exhaustive(no_niches)]`.
+    2. `#[non_exhaustive(abi)]`.
+    3. `#[non_exhaustive(repr)]`.
+    4. `#[non_exhaustive(layout)]`.
+3. New attribute `#[really_non_exhaustive]` or similar.
+
+We have a kind of decision tree here, where some unresolved questions depend on the syntax. The exact syntax does not need to be decided before accepting the RFC, though we should choose one of the main "branches".
+
+The RFC author himself is undecided on the syntax, but decided to go with `#[repr(_, no_niches)]` for the body of the RFC, to make the desired semantics clearer to the RFC reader.


Might be T-opsem purview in terms of "niche" vs. "ABI" vs. "non_exhaustive" terminology? See also rust-lang/unsafe-code-guidelines#122.

text/3803-enum-repr-no-niches.md

The `metal` crate is currently unsound regarding unknown/future enum variants, see gfx-rs/metal-rs#209 and rust-lang/rfcs#3803. `objc2-metal` fixes this by emitting C enums as a newtype + constants for each variant, but that prevents us from importing the variants/constants. So this commit converts to a pattern that works with that in preparation for the migration.

text/3803-enum-repr-no-niches.md

More straightforward framing.

madsmtm · 2025-05-05T14:15:47Z

Thanks for the comments all, and sorry for the delay. I believe I've incorporated your suggestions into the RFC text, but feel free to open a new comment if you disagree!

Add enum_repr_no_niches RFC

1b6f2e4

rustbot added the T-lang Relevant to the language team, which will review and decide on the RFC. label Apr 21, 2025

Add PR number

c7026ef

madsmtm commented Apr 21, 2025

View reviewed changes

Make syntax options render better

7220e08

kpreid reviewed Apr 22, 2025

View reviewed changes