update the manylinux wheel glibc version to 2.28 (#21650)

[benjyw]

While trying to land #21528 to was observed that the wheel building jobs were failing during git checkout with wonderful errors like /__e/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /__e/node20/bin/node). This appears to be the same symptoms as actions/checkout#1809 which pointed the the deprecation notice at
https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/. If this all sounds familiar that is because we mostly cleaned this up in #21133, but kept a fewer uses of the older actions for manylinux2014 compatibility. However, from the notice:

"To opt out of this and continue using Node16 while it is still available in the runner, you can choose to set
ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true
as an ‘env’ in their workflow or as an environment variable on your runner machine. This will only work until we upgrade the runner removing Node16 later in the spring. (emphasis added)"

From the wheel job failures during #21528 and my attempts to fiddle with all of ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION,
ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION,
ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION I think the removal time promised for the Spring has finally come, but I'm not that familiar with GitHub Actions and could be missing something.

As a consequence the version of both actions/upload-artifacts and actions/checkout is bumped to v4. To make this testable now and in the future, wheels are now build on ci config changes.

Why is bumping to manylinux_2_28 (the next oldest manylinux) not a big deal? The "2.28" refers to glibc
2.28 in 2018. That is older than Debian stable and if you need to use software that old you are presumably paying someone for an enterprise distribution.

NOTE: I think we should bump the manylinux version in main regardless, but if my read if the situation is correct we may need to backport this any active release line. Without this or a more complex change we would also be unable to release after December 5th per #21616.

ref #21195 #21616

[cburroughs]

I agree this is the only choice at reasonable LoE. I don't feel great about it mid release, but I don't think in practice anyone is actually using Pants with those super old glibc versions. Maybe merits a footnote in the release blog in the spirit of over-communicating?

[benjyw]

Yep. Since you've done the deep dive here, could you phrase that footnote?

[cburroughs]

"Due to deprecations by GitHub, the minimum supported glibc version for Pants wheels is now 2.28. Unless you are running Pants with in-repo plugins on a Linux distro from before 2018, this should have no effect."

[huonw]

I think this could applies to more than just the pants wheels/in-repo plugins, in that the new code could call some glibc 2.28 APIs, and thus fail to run with older versions. I don't know if this occurs in practice, though. (The code calling these APIs could either be direct Pants Rust code (since it's now being built-with/linked-against newer glibc), or a transitive dependency that's itself able to start using newer wheels).

Is it possible to get 2.23.0 out, as is, without this change? And then cherry pick for 2.23.1+ (it's a bit of a weird change to happen in the middle of a release cycle, but our hands are kinda forced...)

[benjyw]

Is it possible to get 2.23.0 out, as is, without this change? And then cherry pick for 2.23.1+ (it's a bit of a weird change to happen in the middle of a release cycle, but our hands are kinda forced...)

I don't think so. The wheel building shards just fail.

[benjyw]

Is it possible to get 2.23.0 out, as is, without this change? And then cherry pick for 2.23.1+ (it's a bit of a weird change to happen in the middle of a release cycle, but our hands are kinda forced...)

I don't think so. The wheel building shards just fail.

But I am open to ideas.

[huonw]

Ah okay, I see.

Digging a bit deeper, I suspect all of this might be caused by actions/runner-images@5618718 I.e. the runner images no longer has node16 (as we were told would happen), and apparently this the runner's binaries are used even when executing within a container?

So yeah, it seems our possible paths forward are:

1. land this as is
2. somehow convince the actions to execute with a version of node that we provide within the image somehow

Having worked through that thinking, I fully agree with @cburroughs: 1 does seem like the only reasonable option from an effort perspective. If someone is affected and cares, they need to get involved with maintaining Pants!

It's unlucky that this is all happening so close to a .0 stable release, but I guess we somewhat inflicted this on ourselves by doing the initial ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION workaround but not following up on getting rid of it.

[huonw]

I imagine we'll need to cherry-pick this to the 2.22.x branch too?

[benjyw]

Yeah, and 2.24.x...

[cburroughs]

There is probably some option where -- with much hand-waving -- we use docker environments and https://cibuildwheel.pypa.io/en/stable/, but I think that falls in the unreasonable LoE bucket.

I think this could applies to more than just the pants wheels/in-repo plugins, in that the new code could call some glibc 2.28 APIs, and thus fail to run with older versions. I don't know if this occurs in practice, though. (The code calling these APIs could either be direct Pants Rust code (since it's now being built-with/linked-against newer glibc),

So I have not done more than a toy examples with Rust, but thinking about this more we build the Pants Pex itself on ubuntu22, so absent some cross-compiling Rust magic I'd expect the minimum glibc version that works with would already be 2.35. If that is right, the only impacted workflows would be ones that use the Pants released wheels but not the Pants released Pex. I'm not sure there are any?

[huonw]

I think it's slightly more subtle than that, since it depends it on which wheels are selected: here's what's in 2.22.1's PEX-INFO:

{
    "PyYAML-6.0.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "8c0396211a76d27f152910bbbfdcabe20f4e03be0e4f54aca9c3bd79414bf3df",
    "ansicolors-1.1.8-py2.py3-none-any.whl": "18687c11ecd3f2ee80dffbcbfd248ea50309fa65a2df1d805c6bf1cec8b1f33d",
    "chevron-0.14.0-py3-none-any.whl": "d7316200cba6fc6d3935ba3e8f787b39e0a2d6bc64f091501285aba47a41845f",
    "fasteners-0.16.3-py2.py3-none-any.whl": "ccabc462ef0f9c6905cdf1c3c3709fca2f79db97310df0a71f29b1771bb48539",
    "ijson-3.2.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "0326508a88314ae80cddf17bb354e72bf415f75bd544635854e16b4018e3dd4a",
    "node_semver-0.9.0-py3-none-any.whl": "333d5510aeb120cfb3abdf9404720c9f23b7e4c7c78f11d761647da102c9c6fb",
    "packaging-21.3-py3-none-any.whl": "264b338d5e85b5a986fc45f766b53065dcaccd7cb35d204ff319bd80c503c5f6",
    "pantsbuild.pants-2.22.1-cp39-cp39-linux_x86_64.whl": "34c467e7f0f1124884b1e2ac263401a3088b674c66fd5eae147bc3f751e086ba",
    "pex-2.3.1-py2.py3-none-any.whl": "8d43c1ed16877f3d2ed39e5ef3e92aa8885d4427cd7d9ce62683c922a5c0cd07",
    "psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "b40c0935c1f080f868ad92f09fe97d9528ac7a99fdf740550db21365709b83ff",
    "pyparsing-3.1.2-py3-none-any.whl": "b422fdeeec20dcd8540113cbf5f871361708d7a98a86fc2152aef1484440d9aa",
    "python_lsp_jsonrpc-1.0.0-py3-none-any.whl": "c048d3fe8132ce68f303b1e60d99be11cfe8d7d12181ac982309689a9589cedb",
    "setproctitle-1.3.2-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "5c538df39985111bc0ec3d0deccf5ef3a3de9268137f6271bded8f3d672322d5",
    "setuptools-63.4.3-py3-none-any.whl": "4e4fc62fd129faff0b16783574fe9b0c356bf52c1cc6217e61c96a6afb14bbca",
    "six-1.16.0-py2.py3-none-any.whl": "a8dffba69ff16231827d7133867794d7e85295e22c36fbdc496b3d80d6d8d2cc",
    "toml-0.10.2-py2.py3-none-any.whl": "5fb6845639412b75ccc50cbb2cb70716517b59e3cbc6b832b1b652bb5e24bb99",
    "types_PyYAML-6.0.3-py3-none-any.whl": "3fa6b44036ae62cd583d37db411a9a1b1ebe01dc45dd64305cf3e2c2c338fd3e",
    "types_setuptools-62.6.1-py3-none-any.whl": "6d89dff70d523e7df3a9cfa1cb9f103bab6c976f03ab55a0719d361a4ba000e3",
    "types_toml-0.10.8-py3-none-any.whl": "46e8d165f4bd8790d3b526580067cfc1169f98c92a178b8c6ffcd7f6a66aca71",
    "typing_extensions-4.3.0-py3-none-any.whl": "2370627dba47a337f8ecc3207bc70ef64bb02f556b82d78e0d6112982ebbd3c1",
    "ujson-5.9.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl": "f54f9fa90a80781314b55ca217721f3be8ce41424c3fdbce3e5792876cf40d28"
}

So all of those wheels are manylinux2014 or older, other than the Pants one.