update Pants' own Python to 3.11 #21528

cburroughs · 2024-10-15T18:14:15Z

Building on the work in pantsbuild/scie-pants#351 this brings the version of Python used by Pants from 3.9 to 3.11. Why 3.11 and not 3.12 or 3.13? Because that is what we already released on the scie-pants side and two release forward is still a big benefit.

NOTE: I'd like to hold off on stomping out all deprecation warnings and anything else that "depends" on 3.11until one dev release since the release process part is what I'm most nervous about.

cburroughs

I should add Thanks to:

@kaos for doing the hardest part
@thejcannon for the earlier draft
@benjyw for getting the needed versions of Python on to the Mac builders

cburroughs · 2024-10-30T17:15:20Z

build-support/flake8/.flake8

@@ -18,6 +18,8 @@ extend-ignore:
  NIC102,
  # Unnecessary dict call - rewrite as a literal
  C408,
+  # Temporarily exclude during Python upgrade


Note to self post merge.

cburroughs · 2024-10-30T17:18:04Z

pants-plugins/pants_explorer/server/graphql/query/targets.py

@@ -63,7 +63,7 @@ class Target:
    target_type: str = strawberry.field(
        description="The target type, such as `python_sources` or `pex_binary` etc."
    )
-    fields: JSONScalar = strawberry.field(
+    fields: JSONScalar = strawberry.field(  # type: ignore[valid-type]


So this clearly 'should work' given strawberry-graphql/strawberry#1205

but there are various similar looking errors at strawberry-graphql/strawberry#3579 Given the low current usage of the explorer I went with the ignore.

cburroughs · 2024-10-30T17:19:59Z

src/python/pants_release/generate_github_workflows.py

@@ -8,7 +8,7 @@
 import os
 import re
 from dataclasses import dataclass, field
-from enum import Enum
+from enum import Enum, ReprEnum


https://docs.python.org/3/library/enum.html#enum.ReprEnum

cburroughs · 2024-10-30T17:21:29Z

src/python/pants/testutil/python_interpreter_selection.py

@@ -22,6 +22,8 @@
 PY_37 = "3.7"
 PY_38 = "3.8"
 PY_39 = "3.9"
+PY_310 = "3.10"
+PY_311 = "3.11"


Added for completeness. I'm suspicious that some of these existing annotations around Python 3.9 semantically should be "the version of Python used by Pants" but have not pulled harder on that thread.

huonw

This is awesome, thank you! The code looks good, but want to nail down the deprecation/phasing details.

huonw · 2024-10-30T22:27:05Z

src/python/pants/bin/pants_runner.py

-MINIMUM_SCIE_PANTS_VERSION = Version("0.10.0")
+# First version with Python 3.11 support:
+# https://github.com/pantsbuild/scie-pants/releases/tag/v0.12.0
+MINIMUM_SCIE_PANTS_VERSION = Version("0.12.0")


Have we done any deprecations in advance? I'm thinking it's probably best if someone who's, say, got scie-pants 0.11.0 installed doesn't first learn about this by upgrading to 2.25.x and finding it fails hard/immediately.

Maybe we could sneak a warning about <0.12.0 into 2.24? (Either before branching or cherrypicked soon after)

We internally use an explicit version of scie-pants in ci/devcontainers, but I suspect that most people just use the bootstrapping script and end up with the latest and only update if forced to by this check or some issue they run into. 0.12.0 has been out since May so I think anyone is likely to have setup 0.11 and Pant 2.2{4,5} for the first time unless they were following some intentional internal procedure.

My fear we "per-deprectaiton" might be too clever by half and if we end up needing to put out a 0.12.x for some bug in this thing we are doing for the first time (Python upgrade) are are now forcing multiple upgrades.

@sureshjoshi had a PoC of using the new Pex-makes-the-scies support so that the Pants repo could control the python-build-standalone version and we would have less of this dance.

I'm a bit groggy on this at the moment (and am not home to check), but one thing about deprecations... Which Python interpreter do in-repo plugins use? I'm struggling to recall if Pants runs them using it's own Python, or whether the user needs to have a system Python targeting 3.9.

Which Python interpreter do in-repo plugins use?

I believe the answer is... both!

My understanding is that it "runs" them with with the Pants-Python, but if you have them setup as targets with BUILD files the linting/testing/etc happens with a 'regular' Python. (Thus the callout in the release notes plugin section.)

My fear we "per-deprectaiton" might be too clever by half and if we end up needing to put out a 0.12.x for some bug in this thing we are doing for the first time (Python upgrade) are are now forcing multiple upgrades.

Yeah, that's a reasonable concern, but it seems unfortunate to give up on providing a user prompt just because we might get it wrong.

Thinking out loud, if we land this and put a pre-deprecation into 2.24 soon, we've got until 2.24.0 stable is released (i.e. at least a few weeks) to nail down any additional scie-pants changes and cherry-pick version bumps back to 2.24.x.

Having a heads up for plugin authors in the 2.23 blog post seams a reasonable thing we can do at the least.

I'm not opposed to some sort of warn-but-not-too-often in 2.24, I'm just not sure exactly what that looks like. Best effort to do something by 2.24.0 as we learn more in main sounds okay to me.

Filed #21603

docs/notes/2.24.x.md

huonw

Thank you for doing this!

huonw · 2024-11-01T21:03:09Z

src/python/pants/bin/pants_runner.py

-MINIMUM_SCIE_PANTS_VERSION = Version("0.10.0")
+# First version with Python 3.11 support:
+# https://github.com/pantsbuild/scie-pants/releases/tag/v0.12.0
+MINIMUM_SCIE_PANTS_VERSION = Version("0.12.0")


Filed #21603

cburroughs · 2024-11-14T20:01:05Z

:-(

Created the minor followup issues discussed above. Intending to land this as soon as I figure out a path around the build breakage.

While trying to land #21528 to was observed that the wheel building jobs were failing during git checkout with wonderful errors like `/__e/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /__e/node20/bin/node)`. This appears to be the same symptoms as actions/checkout#1809 which pointed the the deprecation notice at <https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/>. If this all sounds familiar that is because we *mostly* cleaned this up in #21133, but kept a fewer uses of the older actions for `manylinux2014` compatibility. However, from the notice: "To opt out of this and continue using Node16 while it is still available in the runner, you can choose to set ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true as an ‘env’ in their workflow or as an environment variable on your runner machine. This will *only work until we upgrade the runner removing Node16 later in the spring*. (emphasis added)" From the wheel job failures during #21528 and my attempts to fiddle with all of `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION`, `ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION`, `ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION` I think the removal time promised for the Spring has finally come, but I'm not that familiar with GitHub Actions and could be missing something. As a consequence the version of both `actions/upload-artifacts` and `actions/checkout` is bumped to `v4`. To make this testable now and in the future, wheels are now build on ci config changes. Why is bumping to `manylinux_2_28` (the next oldest manylinux) not a big deal? The "2.28" refers to glibc [2.28](https://sourceware.org/legacy-ml/libc-alpha/2018-08/msg00003.html) in 2018. That is older than Debian stable and if you need to use software that old you are presumably paying someone for an enterprise distribution. NOTE: I think we should bump the manylinux version in `main` regardless, but if my read if the situation is correct we may need to backport this any active release line. Without this or a more complex change we would also be unable to release after December 5th per #21616. ref #21195 #21616

Building on the work in pantsbuild/scie-pants#351 this brings the version of Python used by Pants from 3.9 to 3.11. Why 3.11 and not 3.12 or 3.13? Because that is what we already released on the scie-pants side and two release forward is still a big benefit. NOTE: I'd like to hold off on stomping out all deprecation warnings until one `dev` release since the release process part is what I'm most nervous about.

While trying to land pantsbuild#21528 to was observed that the wheel building jobs were failing during git checkout with wonderful errors like `/__e/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /__e/node20/bin/node)`. This appears to be the same symptoms as actions/checkout#1809 which pointed the the deprecation notice at <https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/>. If this all sounds familiar that is because we *mostly* cleaned this up in pantsbuild#21133, but kept a fewer uses of the older actions for `manylinux2014` compatibility. However, from the notice: "To opt out of this and continue using Node16 while it is still available in the runner, you can choose to set ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true as an ‘env’ in their workflow or as an environment variable on your runner machine. This will *only work until we upgrade the runner removing Node16 later in the spring*. (emphasis added)" From the wheel job failures during pantsbuild#21528 and my attempts to fiddle with all of `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION`, `ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION`, `ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION` I think the removal time promised for the Spring has finally come, but I'm not that familiar with GitHub Actions and could be missing something. As a consequence the version of both `actions/upload-artifacts` and `actions/checkout` is bumped to `v4`. To make this testable now and in the future, wheels are now build on ci config changes. Why is bumping to `manylinux_2_28` (the next oldest manylinux) not a big deal? The "2.28" refers to glibc [2.28](https://sourceware.org/legacy-ml/libc-alpha/2018-08/msg00003.html) in 2018. That is older than Debian stable and if you need to use software that old you are presumably paying someone for an enterprise distribution. NOTE: I think we should bump the manylinux version in `main` regardless, but if my read if the situation is correct we may need to backport this any active release line. Without this or a more complex change we would also be unable to release after December 5th per pantsbuild#21616. ref pantsbuild#21195 pantsbuild#21616

While trying to land #21528 to was observed that the wheel building jobs were failing during git checkout with wonderful errors like ``/__e/node20/bin/node: /lib64/libm.so.6: version `GLIBC_2.27' not found (required by /__e/node20/bin/node)``. This appears to be the same symptoms as actions/checkout#1809 which pointed the the deprecation notice at <https://github.blog/changelog/2024-03-07-github-actions-all-actions-will-run-on-node20-instead-of-node16-by-default/>. If this all sounds familiar that is because we *mostly* cleaned this up in #21133, but kept a fewer uses of the older actions for `manylinux2014` compatibility. However, from the notice: "To opt out of this and continue using Node16 while it is still available in the runner, you can choose to set ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION=true as an ‘env’ in their workflow or as an environment variable on your runner machine. This will *only work until we upgrade the runner removing Node16 later in the spring*. (emphasis added)" From the wheel job failures during #21528 and my attempts to fiddle with all of `ACTIONS_ALLOW_USE_UNSECURE_NODE_VERSION`, `ACTIONS_RUNNER_FORCED_INTERNAL_NODE_VERSION`, `ACTIONS_RUNNER_FORCE_ACTIONS_NODE_VERSION` I think the removal time promised for the Spring has finally come, but I'm not that familiar with GitHub Actions and could be missing something. As a consequence the version of both `actions/upload-artifacts` and `actions/checkout` is bumped to `v4`. To make this testable now and in the future, wheels are now build on ci config changes. Why is bumping to `manylinux_2_28` (the next oldest manylinux) not a big deal? The "2.28" refers to glibc [2.28](https://sourceware.org/legacy-ml/libc-alpha/2018-08/msg00003.html) in 2018. That is older than Debian stable and if you need to use software that old you are presumably paying someone for an enterprise distribution. NOTE: I think we should bump the manylinux version in `main` regardless, but if my read if the situation is correct we may need to backport this any active release line. Without this or a more complex change we would also be unable to release after December 5th per #21616. ref #21195 #21616 Co-authored-by: cburroughs <[email protected]>

Fixes #424 This makes a few adjustments to support running Pants 2.25.0.dev0 and newer, which run using Python 3.11 (pantsbuild/pants#21528), iterating on #351. Particularly: - expanding tools.pex's interpreter constraints to include 3.11 (and also 3.10, which seems fine) - add a test (NB. Pants 2.25 cannot run on macOS older than 13 / 14, depending on platform, so this test is conditional: pantsbuild/pants#21655) - hardcoding that 2.25.0.dev0 uses Python 3.11, to use the optimised "hit the exact URL first time" codepath, instead of having to try all Python versions: https://github.com/pantsbuild/scie-pants/blob/d50bd33fd67e944384c9548811a66c1cb03113a5/tools/src/scie_pants/pants_version.py#L239-L252 This thus preps scie-pants release 0.12.1 too, which I'll tag once merged.

cburroughs force-pushed the csb/py11-bump branch 6 times, most recently from ca4fae3 to d4fa8dd Compare October 29, 2024 20:55

cburroughs changed the title ~~python3.11 draft~~ update Pants' own Python to 3.11 Oct 29, 2024

cburroughs force-pushed the csb/py11-bump branch 2 times, most recently from 1b4c9c4 to 64b26f0 Compare October 30, 2024 16:41

cburroughs commented Oct 30, 2024

View reviewed changes

cburroughs marked this pull request as ready for review October 30, 2024 17:23

huonw reviewed Oct 30, 2024

View reviewed changes

cognifloyd approved these changes Oct 31, 2024

View reviewed changes

huonw mentioned this pull request Nov 1, 2024

Add deprecation warnings to 2.24 about scie-pants/Python 3.11 #21603

Open

huonw approved these changes Nov 1, 2024

View reviewed changes

cburroughs force-pushed the csb/py11-bump branch from dddf00b to 988ee28 Compare November 14, 2024 19:00

This was referenced Nov 14, 2024

resolve Python 3.11 deprectation/WARNings #21646

Closed

Resolve Temporary flake8 exlcusions from Python 3.11 cutover #21647

Open

cburroughs mentioned this pull request Nov 15, 2024

update the manylinux wheel glibc version to 2.28 #21650

Merged

cburroughs added 2 commits November 16, 2024 18:03

docs version switch

3e8fae6

cburroughs force-pushed the csb/py11-bump branch from 988ee28 to 3e8fae6 Compare November 16, 2024 23:04

cburroughs merged commit 1258080 into pantsbuild:main Nov 17, 2024
24 checks passed

huonw mentioned this pull request Nov 18, 2024

Test and release on newer macOS: 13 (x86-64) and 14 (arm64) #21655

Merged

benjyw mentioned this pull request Nov 18, 2024

update the manylinux wheel glibc version to 2.28 (#21650) #21660

Merged

huonw mentioned this pull request Nov 25, 2024

Fix support for Pants 2.25.x / Python 3.11 pantsbuild/scie-pants#426

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update Pants' own Python to 3.11 #21528

update Pants' own Python to 3.11 #21528

cburroughs commented Oct 15, 2024 •

edited

Loading

cburroughs left a comment

cburroughs Oct 30, 2024

cburroughs Oct 30, 2024

cburroughs Oct 30, 2024

cburroughs Oct 30, 2024

huonw left a comment

huonw Oct 30, 2024

cburroughs Oct 31, 2024

sureshjoshi Oct 31, 2024

cburroughs Oct 31, 2024

huonw Oct 31, 2024

cburroughs Nov 1, 2024

huonw Nov 1, 2024

huonw left a comment

huonw Nov 1, 2024

cburroughs commented Nov 14, 2024

update Pants' own Python to 3.11 #21528

update Pants' own Python to 3.11 #21528

Conversation

cburroughs commented Oct 15, 2024 • edited Loading

cburroughs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

huonw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

huonw left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cburroughs commented Nov 14, 2024

cburroughs commented Oct 15, 2024 •

edited

Loading