Allow the use of arbitrary Pyodide versions

[agriyakhetarpal]

Description

This PR updates the Pyodide build procedure (see #1456) that is enabled with CIBW_PLATFORM: "pyodide" (or with the --platform pyodide CLI equivalent) post the changes in pyodide/pyodide#4882, where pyodide/pyodide-build was unvendored from the main Pyodide repository to accommodate faster updates and fixes.

This means that the Pyodide version and pyodide-build are not going to be in sync going forward, and that the Pyodide xbuildenv to install must be inferred by the versions available to install by pyodide-build through a recently added pyodide xbuildenv search command, which prints out this table:

Tap to expand table

┌────────────┬────────────┬────────────┬───────────────────────────┬────────────┐
│ Version    │ Python     │ Emscripten │ pyodide-build             │ Compatible │
├────────────┼────────────┼────────────┼───────────────────────────┼────────────┤
│ 0.27.0a2   │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
│ 0.26.4     │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
│ 0.26.3     │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
│ 0.26.2     │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
│ 0.26.1     │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
│ 0.26.0     │ 3.12.1     │ 3.1.58     │ 0.26.0 -                  │ Yes        │
└────────────┴────────────┴────────────┴───────────────────────────┴────────────┘

Alternatively, one may use pyodide xbuildenv search --all to return both compatible and non-compatible versions. This would, however, be better received as JSON (please see pyodide/pyodide-build#28).

---

Additionally, in this PR, support has been added for installing arbitrary Pyodide versions, or, more specifically, arbitrary versions for "Pyodide cross-build environments (xbuildenvs)" – though, only the ones that are supported for a given pyodide-build version. This has been implemented through an environment variable CIBW_PYODIDE_VERSION and an associated configuration variable in the schema (through a table implemented via pyodide/pyodide-build#26).

The rationale behind this is that WebAssembly/Pyodide builds are already experimental, and it would be useful to not tie the available Pyodide version to the cibuildwheel version – this would be helpful for downstream projects (statsmodels/statsmodels#9343, scikit-image/scikit-image#7525, scikit-learn/scikit-learn#29791, and so on) to allow testing against Pyodide's alpha releases and/or for the case of greater reproducibility against Pyodide's older releases.

cc: @hoodmane and @ryanking13 for visibility

---

Suggested CHANGELOG entry

Since I didn't find a way to add an entry without the pre-commit hook removing previous entries, I've added a few lines here based on the current state of this PR. Please feel free to suggest changes or modify these lines directly.

- 🛠 Provide [Pyodide version 0.26.4](https://github.com/pyodide/pyodide/releases/tag/0.26.4) with `cp312-pyodide_wasm32` (#2002)
- ✨ Allow the use of a custom Pyodide version to target by the use of the `CIBW_PYODIDE_VERSION` environment variable. This
is an experimental option and users are advised to look at the [compatible Pyodide versions](https://github.com/pyodide/pyodide/blob/main/pyodide-cross-build-environments.json) according to the [`pyodide-build`](https://github.com/pyodide/pyodide-build) version.

[agriyakhetarpal]

The Windows test failures are unrelated. I'll try to fix them later in the day, but happy to step back if someone else does it before me, or wishes to.

[joerick]

Well, I'm pretty confused. But I'll share the minimal repros I'm working with, in case they're helpful @hoodmane.

# docker run -it --rm ghcr.io/astral-sh/uv:debian 
uv run --python=python3.13 --with pyodide-build python <<'EOF'

import subprocess
from pathlib import Path
import sys
import os
import textwrap
import shutil

venv_dir = Path("/tmp/pyodidevenv")

try:
    shutil.rmtree(venv_dir)
except Exception as e:
    pass

def run(args, env=None):
    result = subprocess.run(args, env=env)
    if result.returncode != 0:
        sys.exit(f'{' '.join(args)} returned {result.returncode}')

run(['pyodide', 'venv', venv_dir])

env = os.environ.copy()

env["PATH"] = str(venv_dir / "bin") + os.pathsep + env["PATH"]

print("we're running this pip:")
run(['which', 'pip'], env=env)

run(['pip', 'install', 'pytest'], env=env)
run(['pytest'], env=env)

EOF

This fails on debian - I get the encoding error

I've tried inserting the workaround, as shown here-

# docker run -it --rm ghcr.io/astral-sh/uv:debian 
uv run --python=python3.13 --with pyodide-build python <<'EOF'

import subprocess
from pathlib import Path
import sys
import os
import textwrap
import shutil

venv_dir = Path("/tmp/pyodidevenv")

try:
    shutil.rmtree(venv_dir)
except Exception as e:
    pass

def run(args, env=None):
    result = subprocess.run(args, env=env)
    if result.returncode != 0:
        sys.exit(f'{' '.join(args)} returned {result.returncode}')

run(['pyodide', 'venv', venv_dir])

# --WORKAROUND--
base_python = Path(sys.executable)
(venv_dir / "bin" / "python-host-link").symlink_to(base_python)
(venv_dir / "bin" / "python-host").unlink()
(venv_dir / "bin" / "python-host").write_text(
    textwrap.dedent(f"""
        #!/bin/bash
        export PYTHONHOME={base_python.parent.parent}
        exec {venv_dir / "bin" / "python-host-link"} -s "$@"
    """)
)

(venv_dir / "bin" / "python-host").chmod(0o755)

# make the python3.12-host bin point to the python-host script
(venv_dir / "bin" / "python3.13-host").unlink()
(venv_dir / "bin" / "python3.13-host").symlink_to(venv_dir / "bin" / "python-host")
# --END WORKAROUND--

env = os.environ.copy()

del env["VIRTUAL_ENV"]
env["PATH"] = str(venv_dir / "bin") + os.pathsep + env["PATH"]

print("we're running this pip:")
run(['which', 'pip'], env=env)

run(['pip', 'install', 'pytest'], env=env)
run(['pytest'], env=env)

EOF

But on linux, this fails with the following error:

we're running this pip:
/tmp/pyodidevenv/bin/pip

[notice] A new release of pip is available: 24.3.1 -> 25.0.1
[notice] To update, run: /root/.local/share/uv/python/cpython-3.13.3-linux-aarch64-gnu/bin/python3.13 -m pip install --upgrade pip
error: externally-managed-environment

× This environment is externally managed
╰─> This Python installation is managed by uv and should not be modified.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
pip install pytest returned 1

Which is weird, as we're in a virtualenv, but pip doesn't think so.

[hoodmane]

Thanks! I'll try it out when I get home and see if I can figure out anything that helps.

[hoodmane]

This works as expected for me:

uv run --python=python3.12 --with pyodide-build python <<'EOF'

import subprocess
from pathlib import Path
import sys
import os
import textwrap
import shutil

venv_dir = Path("pyodidevenv")

try:
    shutil.rmtree(venv_dir)
except Exception as e:
    pass

def run(args, env=None):
    result = subprocess.run(args, env=env)
    if result.returncode != 0:
        sys.exit(f'{' '.join(args)} returned {result.returncode}')

run(['pyodide', 'xbuildenv', 'use', '0.27.4'])
run(['pyodide', 'venv', venv_dir])



# --WORKAROUND--
base_python = Path(sys.executable)
(venv_dir / "bin" / "python-host-link").symlink_to(base_python)
(venv_dir / "bin" / "python-host").unlink()
(venv_dir / "bin" / "python-host").write_text(
    textwrap.dedent(f"""
        #!/bin/bash
        export PYTHONHOME={base_python.parent.parent}
        exec {venv_dir / "bin" / "python-host-link"} -s "$@"
    """)
)

(venv_dir / "bin" / "python-host").chmod(0o755)

# make the python3.12-host bin point to the python-host script
(venv_dir / "bin" / "python3.12-host").unlink()
(venv_dir / "bin" / "python3.12-host").symlink_to(venv_dir / "bin" / "python-host")
# --END WORKAROUND--

env = os.environ.copy()

del env["VIRTUAL_ENV"]
env["PATH"] = str(venv_dir / "bin") + os.pathsep + env["PATH"]

print("we're running this pip:")
run(['which', 'pip'], env=env)

run(['pip', 'install', 'pytest'], env=env)
run(['pytest'], env=env)

EOF

Maybe I should try with the docker image.

[joerick]

This is coming along nicely. We'll need to avoid hitting the Github Releases API at runtime, it's rate limiting us. I suppose we can cache the artifacts in nox -s update_constraints nox -s update_pins instead.

The other thing would be to automate pyodide version bumping, so the bot can do it for us. That shouldn't be too hard, we're grabbing the json file to pin pyodide-build already.

[joerick]

I've added auto version bumping, fixed the runtime GitHub API issue, and added support for the (as yet alpha) next version of Pyodide, targeting 3.13. I don't think there's anything left on my list here.

Couple things to note-

• we're still using python -m pytest due to the issues with non-pip entrypoints and the python-build-standalone venvs. I'm happy to just document that as a known issue for now, it's already required on iOS.
• We're referencing on a prerelease version of pyodide. Depending on the timeline for pyodide 0.28.0, we should maybe put that behind a prerelease enable group.

[joerick]

@agriyakhetarpal any idea what's going on with the macOS test? It's working okay on my local machine but installing the xbuildenv is failing with pyodide 0.28.0a1, it fails when installing numpy actually.

[joerick]

Since the PBS code could be useful to #2349, I think for now I'll remove the pyodide 0.28 support, we can add that in a later PR.