Add (back) unsupported_calling_conventions lint to reject more invalid calling conventions

[RalfJung]

This adds back the unsupported_calling_conventions lint that was removed in #129935, in order to start the process of dealing with #137018. Specifically, we are going for the plan laid out here:

• thiscall, stdcall, fastcall, cdecl should only be accepted on x86-32
• vectorcall should only be accepted on x86-32 and x86-64

The difference to the status quo is that:

• We stop accepting stdcall, fastcall on targets that are windows && non-x86-32 (we already don't accept these on targets that are non-windows && non-x86-32)
• We stop accepting cdecl on targets that are non-x86-32
• (There is no difference for thiscall, this was already a hard error on non-x86-32)
• We stop accepting vectorcall on targets that are windows && non-x86-*

Vectorcall is an unstable ABI so we can just make this a hard error immediately. The others are stable, so we emit the unsupported_calling_conventions forward-compat lint. I set up the lint to show up in dependencies via cargo's future-compat report immediately, but we could also make it show up just for the local crate first if that is preferred.

[rustbot]

r? @lcnr

rustbot has assigned @lcnr.
They will have a look at your PR within the next two weeks and either review your PR or reassign to another reviewer.

Use r? to explicitly pick a reviewer

[rustbot]

These commits modify compiler targets.
(See the Target Tier Policy.)

[RalfJung]

@rust-lang/lang following positive vibes expressed here, I'd like to land this PR as a first step to implement the plan from that issue. Can we get this FCPd? :)

The one potential open question here is whether this should be a regular FCW first or whether it can immediately become an FCW that shows up in dependencies. I assume stdcall and fastcall are sufficiently rare that we can go for report-in-deps immediately (since they already get rejected on non-x86-32 non-Windows targets). "cdecl" might be more widely used though so it may be worth ramping up the lint more slowly.

[traviscross]

Searching across GitHub, there are 506 Rust file hits on extern "cdecl".

Regarding whether to fire the FCW in deps immediately, I lean toward it probably being OK. Given that this may show up in cfg-heavy code, I could imagine that many library authors may not notice until their downstreams report anyway. But if anyone on lang feels we should walk this more slowly, that's what I'd estimate we'd do, so I'm curious to hear what others think.

[RalfJung]

Oh that's the syntax for doing such a search... I tried and failed to get a similar result out of github search.^^

[traviscross]

Let's propose to do it.

@rfcbot fcp merge

[rfcbot]

Team member @traviscross has proposed to merge this. The next step is review by the rest of the tagged team members:

• @joshtriplett
• @nikomatsakis
• @scottmcm
• @tmandry
• @traviscross

No concerns currently listed.

Once a majority of reviewers approve (and at most 2 approvals are outstanding), this will enter its final comment period. If you spot a major issue that hasn't been raised at any point in this process, please speak up!

cc @rust-lang/lang-advisors: FCP proposed for lang, please feel free to register concerns.
See this document for info about what commands tagged team members can give me.

[tmandry]

@rfcbot reviewed

[joshtriplett]

@rfcbot reviewed

I think, in the case of extern "stdcall", we should consider proviing a suggestion for extern "system".

extern "fastcall" is uncommon enough that it's not that onerous if we tell people to either use cfg (despite the substantial duplication at the moment) or use a macro for this.

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[workingjubilee]

Noting using such searches as TC mentioned:

• extern "stdcall" has over ~2000 hits
• extern "cdecl" has ~522 hits (maybe some DB shards didn't respond to TC's query?)
• extern "fastcall" has ~480 hits
• extern "thiscall" has ~300 hits
• extern "vectorcall" has ~128 hits (unsurprising, it's still unstable)

...and many of these hits are just on our own repo but forked.

[workingjubilee]

Somewhat unscientifically skimming the searches, I notice:

• most users of extern "cdecl" seem to be
  • operating systems or bootloaders, primarily with cfg on x86
  • modding tech, either general solutions akin to Detours, or more specific ones targeted at one application, sometimes without cfg but with READMEs that contain explicit instructions to compile the code for i686-pc-windows-msvc
  • various forms of cross-language binding tools
• extern "fastcall" is similarly diverse
• extern "vectorcall" is much more biased towards cross-language bindings
• extern "stdcall" appears in a lot more things intended for on-Windows usage, including the previous categories (though OS-y stuff only in the sense of drivers) but also Windows applications and, importantly, a number of utility crates

So I agree that we should probably offer suggestions for migrating extern "stdcall" code, as it seems to be the one most likely to appear in an actual library case.

[RalfJung]

I added targeted help messages for "stdcall" and "cdecl". I made them also show up for the hard error since they should be helpful there too, e.g. when someone tries to use "stdcall" on x86_64-Linux.

[lcnr]

r? @workingjubilee maybe, feel free to reassign

[workingjubilee]

We already know this gating is leaky and allows ABIs to appear in places we do not want, c.f. #86232 and since we are reviving this lint, it might be more productive to move the linting into rustc_ast_lowering where we are guaranteed to catch such instances.

[RalfJung]

I'm not very familiar with that crate... I feel like moving the check in a better place is an orthogonal improvement to what this PR does.