Macro fragment fields

[joshtriplett]

Add a syntax and mechanism for macros to access "fields" of high-level fragment
specifiers that they've matched, to let macros use the Rust parser for
robustness and future compatibility, while still extracting pieces of the
matched syntax.

This RFC introduces the syntax ${fragname.field}, and a couple of fragment specifiers and their fields. The goal is to add more such fragment specifiers and fields, to allow more macros to leverage the Rust parser, but the purpose of this RFC is to introduce the concept and syntax.

Rendered

[nikomatsakis]

Oh, I like this. Cute idea.

[joshtriplett]

Nominating this (and related RFCs) for discussion, to decide whether we can process it asynchronously or whether we need a design meeting.

[matthieu-m]

I think an important discussion to be had will be whether it's okay for fields to "generate" tokens.

The RFC itself already proposes that fn.return_type materializes a () AST type node ex-nihilo, and the discussion on param notes that &self could simply be materialized as self: &Self thus fitting the larger pat: ty pattern.

Unless the plan is to drop fn.return_type from this RFC, hamstringing fn, I believe the lang/compiler teams should come to a consensus on the policy here:

• Is sticking to the source more important? In which case fn.return_type should be a ty?.
• Is simplification, if semantically equivalent, preferable?

---

Another discussion which may be necessary is pinning down exactly which types the fields should have.

Unless macro-rules are significantly complicated by allowing subtyping in the future, for now, types are final.

For example, using the pat: ty syntax for a function parameter may seem more favorable than adding an ad-hoc fragment type. Okay, &self is a bit weird, but it can be matched as self: &Self so all good?

The problem, though, is that suddenly:

• How do you attach attributes?
• How do you extend the parameter syntax to allow splat for variadic generics (eg. name...: T...), is splat shoehorned in the pat/ty?

Keen readers may notice that C-variadics ... would already be a problem, but please bear with me here. Finding examples is hard.

The conservative choice, it seems to me, would be to err on the side of introduce fragment types more often than not, even if in the meantime they end up being functionally equivalent to another fragment type (or a set thereof).

Note: editions may help here, but any change risks introducing breakage so... it may be best to think of editions as a last resort rather than as the default way.

[joshtriplett]

@matthieu-m So, I do in general think that we should allow fields to synthesize tokens. There are too many cases where it would be useful to present unified cases to macros to simplify writing macros.

And in particular, for return_type, I think it makes sense to synthesize (), to make it easier for macros to process the return type uniformly. For instance, consider a macro that wants to transform the return type from T to Option<T> for some reason; giving the macro () as the return type means it'll Just Work to write Option<${f.return_type}>.

But that said, your comment and the mention of things like ty? inspired some speculations about possible ways to handle conditionals. I've added those speculations into the future possibilities section.

Imagine, for instance, matching $t:adt, and accessing ${t.struct} inside of a $(...)?, which would "repeat zero times" (expanding to nothing) if t isn't a struct.

Please note that this remains firmly within the future possibilities section, and I'm very intentionally leaving it out of this RFC to avoid blocking on figuring out the right design for conditionally available fields. But it is a rather appealing design.

[joshtriplett]

@matthieu-m wrote:

Unless macro-rules are significantly complicated by allowing subtyping in the future, for now, types are final.

I don't think this is the case.

Today, you can write a macro that matches the same tokens several different ways. And I think we could, for instance, present param as pat: ty today, and later present it as a param type containing the same tokens. I don't think that would add any complexity or compatibility issues.

We can also add fields to existing fragment specifiers, without breaking compatibility.

There are compatibility considerations we have to take care with, and we may need to introduce new fragment specifiers in the future to handle those; for instance, if we make a field require and it later becomes optional, we might have to introduce a new fragment specifier with it optional.

But I don't think switching the type of a field (e.g. to a newly created fragment specifier) would break compatibility as long as it contains the same tokens.

[vincenzopalazzo]

LGMT otherwise

[jhpratt]

Glad to see someone materializing the idea I've had floating around for a while.

For the purpose of avoiding RFCs for future fields, I believe it would be best to explicitly grant T-lang the ability to decide this on their own volition.

[jhpratt]

What is the "type" of this field?

[joshtriplett]

If we add the macro fragment param, then each repetition will have type param; until then, each repetition looks like pat_param: ty. (Handwaving the ... case here.)

[joshtriplett]

For the purpose of avoiding RFCs for future fields, I believe it would be best to explicitly grant T-lang the ability to decide this on their own volition.

I added an unresolved question about whether we should develop a lighter-weight process/policy for approving these, and whether we should delegate them to another team (e.g. wg-macros).