-
-
Notifications
You must be signed in to change notification settings - Fork 1.7k
PEP 796: Relative Virtual Environments [initial draft] #4476
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
rickeylev
wants to merge
36
commits into
python:main
Choose a base branch
from
rickeylev:relative.venvs
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+323
−0
Open
Changes from all commits
Commits
Show all changes
36 commits
Select commit
Hold shift + click to select a range
ceeaa3b
initial file
rickeylev d0a9289
add some core content
rickeylev 41d2ea8
rename to motivation
rickeylev 65a415c
note it enables more advanced optimizations
rickeylev 4175b20
add foot notes, relocatable term section, links
rickeylev bbc85b7
update discussion, sponsor headers to pending
rickeylev a0c2529
rst fixups
rickeylev 8e6e21e
fixup italics
rickeylev 37202ca
rename to relative-venv
rickeylev b81c50f
clarify advanced deployment options, reflow some text
rickeylev 55a5e28
add specification, rationale section; doc why env vars arent an option
rickeylev e09b00d
also mention curdir syntax handling
rickeylev c7a6a9b
update some todo text
rickeylev bc1ffbd
update codeowners
rickeylev 2301d2a
add initial discussion footnote link
rickeylev e14b01f
add fr footnote
rickeylev 175f378
revert codeowners change
rickeylev 31cb756
remove packaging-related specs
rickeylev cd46f4d
fix lint errors with headers
rickeylev b532f5c
set sponsor
rickeylev 99434fc
set sponsor as codeowner
rickeylev 862142a
clarify title to focus on core pyvenv.cfg home behavior
rickeylev 1c8fb2d
fix typo; remove host-relocatable mention; remove non-germane copying…
rickeylev c542890
fix typos, grammar
rickeylev 3240227
Apply suggestions from code review
rickeylev 557cb71
fix up markdown and text width
rickeylev 0579623
fix title underlines
rickeylev c9388b3
link to relative venv reference implementation
rickeylev b959f24
Apply suggestions from code review
rickeylev 5b259cc
Apply suggestions from code review
rickeylev 560dadb
Merge branch 'main' of https://github.com/python/peps into relative.v…
rickeylev f41ed51
Merge branch 'relative.venvs' of https://github.com/rickeylev/peps in…
rickeylev 5cbc9f5
better answer why/how relvenvs help; wrap some long lines
rickeylev 0af0eea
Apply AA-Turner suggestions
rickeylev 68dd5e0
add refs, remove CWD acronym, fix grammar
rickeylev b50a80c
address review comments: better split and organize rationale/motivation
rickeylev File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,322 @@ | ||
PEP: 796 | ||
Title: Relative Home Path in Virtual Environments | ||
Author: Richard Levasseur <[email protected]> | ||
Sponsor: Alyssa Coghlan <[email protected]> | ||
Discussions-To: Pending | ||
Status: Draft | ||
Type: Standards Track | ||
Created: 02-Jul-2025 | ||
Python-Version: 3.15 | ||
|
||
|
||
Abstract | ||
======== | ||
|
||
This PEP formally specifies the use of a relative path for ``home`` | ||
in a Python virtual environment's :file:`pyvenv.cfg` file. | ||
|
||
Specifically, we will discuss how such relative paths are understood | ||
by the Python startup process, including their conversion to absolute | ||
paths for use by the runtime. | ||
This is a fundamental building block for virtual environments to | ||
become more portable. | ||
|
||
Motivation | ||
========== | ||
|
||
The ``home`` field in :file:`pyvenv.cfg` is used on interpreter startup to | ||
determine the actual Python interpreter installation that is used to execute | ||
code in that virtual environment. Currently, this path is required to be | ||
absolute for correct virtual environment operation because the original | ||
`PEP 405 <https://peps.python.org/pep-0405/>`__ | ||
specifying virtual environments didn't cover any specific way of processing | ||
relative paths, their behaviour is implementation dependent. CPython releases | ||
up to and including CPython 3.14 resolve them relative to the current process | ||
working directory, making them too unreliable to use in practice. | ||
|
||
The reason to support a relative path is to support portable virtual | ||
environments, which rely on using a host-agnostic relative path to point to | ||
``PYTHONHOME``. | ||
A portable virtual environment is one that can be moved between | ||
platform-compatible hosts, which is an important feature for some projects (see | ||
"Why portable environments matter"). | ||
|
||
The reason support for a relative ``home`` path needs to be | ||
in the interpreter itself is because locating ``PYTHONHOME`` happens | ||
very early in the interpreter startup process, which limits the options for | ||
customizing how it's computed. Without the ability to specify where the | ||
supporting Python runtime files are, the interpreter can't finish startup, | ||
so other hook points (e.g. ``site`` initialization) never trigger. | ||
|
||
Tools that currently look to enable virtual environment portability across | ||
machines do so either by relying on undocumented interpreter behaviour | ||
(Bazel, omitting the ``home`` key entirely to trigger an implementation | ||
dependent fallback to resolving via a symlinked interpreter binary on | ||
non-Windows systems, see `gh-135773`) or by requiring a post-installation script to be executed | ||
after the environment is placed in its target location ( | ||
`venvstacks <https://lmstudio.ai/blog/venvstacks#publishing-environment-layer-archives>`__ | ||
). | ||
|
||
While this PEP on its own isn't sufficient to enable portable virtual | ||
environments, it allows tools like Bazel or venvstacks to more easily prepare | ||
constrained environments that allow for such use cases. | ||
|
||
Why portable virtual environments matter | ||
---------------------------------------- | ||
|
||
Portable virtual environments are important for the efficiency and | ||
reproducibility benefits they bring from being created once and reused multiple | ||
times later in different locations. For example, a build farm can build a | ||
virtual environment once, cache it, and then re-use it as-is to CI jobs. | ||
|
||
|
||
Rationale | ||
========= | ||
|
||
Defining semantics for a relative ``home`` path is the chosen design for the | ||
following reasons. | ||
|
||
First, it is a small change to interpreter startup, in particular of an | ||
unreliable behavior that isn't specified. Currently, relative paths are | ||
resolved to the process's current working directory, which makes them | ||
unreliable for use in practice. | ||
|
||
Second, for portable virtual environments, relative paths allow more | ||
efficient, simple, and correct reproduction of virtual environments between | ||
hosts. This is because they can be copied as-is to different locations. Some | ||
example capabilities this allows are: | ||
|
||
* A build farm creating (and caching) a virtual environment, which is then | ||
served to developers (e.g. Bazel). | ||
* Composing virtual environments together (e.g. venvstacks). | ||
* Installing multiple arbitrary virtual environments into a container to | ||
save disk space. | ||
* Layering a virtual environment atop a container image for faster image | ||
building. | ||
* Not needing virtual environment creation tools on the host that uses a | ||
virtual environment. | ||
* Exact reproduction of an application's virtual environment between a | ||
developer's host and a production host. | ||
|
||
Third, requiring an absolute path is inherently overly proscriptive. The | ||
interpreter itself doesn't care whether paths are relative or absolute, merely | ||
that they point to valid locations, so users should be given the ability to use | ||
a path of their choosing. Given a known anchor point, it's easy to transform a | ||
relative path to an absolute path and still retain predictable and reliable | ||
behavior that produces correct values. | ||
|
||
Fullying designing portable virtual environments | ||
------------------------------------------------ | ||
|
||
This PEP purposely only focuses on the interpreter startup behavior to limit | ||
its scope. There are multiple implementations and many design questions for how | ||
to implement portable virtual environments work (e.g. what installers should | ||
do), but they are separate from the Python runtime initialization phase. | ||
|
||
|
||
Specification | ||
============= | ||
|
||
The ``home`` value in ``pyvenv.cfg`` is permitted to use a relative path value. | ||
These may contain parent-directory references outside of the virtual environment root | ||
directory. | ||
For example: | ||
|
||
* ``subdir/whatever/bin`` (a directory within the virtual environment) | ||
* ``./subdir/whatever/bin`` (same as above) | ||
* ``../../../../../elsewhere/runtime/bin`` (a directory outside the virtual | ||
environment) | ||
|
||
Relative paths are relative to the directory containing :file:`pyvenv.cfg`. | ||
During interpreter startup (i.e. :file:`getpath.py`), the relative path is joined to the | ||
directory containing ``pyvenv.cfg`` to form an absolute path. | ||
Parent-directory references (``../``) and current | ||
directory references (``./``) are resolved syntactically (i.e. not resolving | ||
symlinks). Symlinks are *not* resolved prior to construction of the absolute | ||
path to ensure semantics between a relative path and absolute path remain the | ||
same. | ||
|
||
For example, given | ||
``/home/user/venv/bin/pyvenv.cfg`` with | ||
``home = ../../runtime/./bin``, the result is ``home = /home/user/runtime/bin``, | ||
i.e. it's equivalent to using that value verbatim in ``pyvenv.cfg``. | ||
|
||
|
||
CPython Runtime Changes | ||
----------------------- | ||
|
||
The CPython runtime itself *almost* already supports relative paths. The | ||
primitives are there, so the only change needed is to define how it resolves | ||
relative paths for ``home`` in ``pyvenv.cfg``. | ||
|
||
Currently, relative paths resolve relative to the process's current working | ||
directory. Because the current working directory isn't knowable in advance, it | ||
makes relative paths today effectively impossible. | ||
|
||
Instead, the paths should be relative to the location of the ``pyvenv.cfg`` | ||
file. This file is chosen as the anchor point because the tool that creates the | ||
file also has to know where the Python runtime is, so can easily calculate the | ||
correct relative path. For tools that read the ``pyvenv.cfg``, it is also easy | ||
to simply join the directory name of where ``pyvenv.cfg`` was found with the | ||
path in the config file. When a person reads the config file, they can do | ||
something similar, which results in a lower cognitive burden and helps avoid | ||
the question of "relative to what?" | ||
|
||
This change is only a couple of lines in the startup code. Specifically, when | ||
parsing the ``pyvenv.cfg`` file and finding the ``home`` value, it just needs | ||
to be checked if it's already absolute. If not, then join it to the directory | ||
name of the ``pyvenv.cfg`` file. The code already knows the directory and has | ||
helpers already exist for checking if a path is absolute and joining two | ||
paths. | ||
|
||
A proof-of-concept of this is implemented in the author's branch, | ||
`rickeylev/feat.relative.pyvenv.home <https://github.com/python/cpython/compare/main...rickeylev:cpython:feat.relative.pyvenv.home>`__. | ||
|
||
Backwards Compatibility | ||
======================= | ||
|
||
Tools that work around the absolute ``home`` key limitation the way Bazel | ||
and venvstacks currently do (omitting the ``home`` key, or editing it after | ||
moving the environment) will be unaffected. | ||
|
||
While the PEP author and sponsor aren't aware of any projects that work around | ||
the limitation by carefully controlling the current working directory used to | ||
launch the deployed Python environments on target systems, any such projects | ||
would be unaffected if they already ensured the working directory was set to | ||
the folder containing ``pyvenv.cfg`` (which seems like a plausible choice, | ||
since that is typically the root directory of the virtual environment). In the | ||
even more unlikely case where that assumption doesn't hold, tools generating | ||
relative virtual environment paths will typically be aware of the underlying | ||
base runtime Python version, and hence able to update the emitted relative path | ||
accordingly. | ||
|
||
Security Implications | ||
===================== | ||
|
||
A relative path in :file:`pyvenv.cfg` may resolve differently depending on the | ||
location of the virtual environment. This *could* point to a surprising, | ||
potentially malicious, location. | ||
|
||
However, this risk already exists today because a relative path isn't | ||
_rejected_, but resolved relative to the current working directory. This PEP | ||
just changes the anchor point to ``pyvenv.cfg`` itself. | ||
|
||
Similarly, the same concern exists for absolute paths. The two are | ||
fundamentally the same because they both rely on trusting whoever created | ||
the ``pyvenv.cfg`` file, which requires having run another tool or downloaded | ||
something from elsewhere. | ||
|
||
|
||
How to Teach This | ||
================= | ||
|
||
Teaching this should be straightforward: if you use a relative path in | ||
``pyvenv.cfg``, then it's relative to the directory containing the | ||
``pyvenv.cfg`` file. This is simple to explain and easy to understand for | ||
anyone that is already familiar with handling relative filesystem paths. | ||
|
||
|
||
Reference Implementation | ||
======================== | ||
|
||
A reference implementation is available by using the combination of: | ||
|
||
* Python runtime from `rickeylev/feat.relative.pyvenv.home <https://github.com/python/cpython/compare/main...rickeylev:cpython:feat.relative.pyvenv.home>`__ | ||
* Relative venv from `rickeylev/relvenv <https://github.com/rickeylev/relvenv>`__ | ||
|
||
And following the | ||
`relvenv README <https://github.com/rickeylev/relvenv/blob/main/README.md>`__. | ||
|
||
Open Issues | ||
=========== | ||
|
||
This PEP does not specify how to create a ``pyvenv.cfg`` with a relative path, | ||
nor how downstream tools (e.g. installers) should identify them or process | ||
them. These questions are best addressed separately by tool owners. | ||
|
||
References | ||
========== | ||
|
||
portable virtual environment | ||
A portable virtual environment is one that can be copied from | ||
one host to another that is platform compatible (e.g. same OS, CPU | ||
architecture, etc), with little or no modification or post processing. | ||
|
||
* `rules_python <https://github.com/bazel-contrib/rules_python>`__: implements | ||
host-relocatable virtual environments. | ||
* `rules_py <https://github.com/aspect-build/rules_py>`__: implements | ||
host-relocatable virtual environments. | ||
* `python-build-standalone <https://github.com/astral-sh/python-build-standalone>`__ | ||
* `venvstacks <https://pypi.org/project/venvstacks/>`__: a tool for creating | ||
reproducible distribution artifacts from virtual environments A relocatable | ||
Python runtime. | ||
* `PoC for relative home in Python startup <https://github.com/python/cpython/compare/main...rickeylev:cpython:feat.relative.pyvenv.home>`__ | ||
* `Python Ideas "Making venvs relocatable friendly" discussion <https://discuss.python.org/t/making-venvs-relocatable-friendly/96177>`__ | ||
* `gh-136051: relative pyvenv.cfg home <https://github.com/python/cpython/issues/136051>`__ | ||
|
||
Rejected Ideas | ||
============== | ||
|
||
Relative to virtual env root | ||
---------------------------- | ||
|
||
Having the ``home`` value in ``pyvenv.cfg`` relative to the virtual | ||
environment's root directory would work just as well, but this idea is rejected | ||
because it requires additional effort to compute the virtual env root. | ||
|
||
Unspecified home means to dynamically compute home | ||
---------------------------------------------------- | ||
|
||
Today, if a ``pyvenv.cfg`` file doesn't set ``home``, the runtime will try to | ||
dynamically compute it by checking if the current executable (which is | ||
typically the venv's ``bin/python3`` symlink) is a symlink and, if so, use | ||
where that points as ``PYTHONHOME``. | ||
|
||
While currently used as a workaround by some tools, *standardising* this | ||
behavior is undesirable for a couple reasons: | ||
|
||
1. It presents platform-specific issues, namely with Windows. Windows does | ||
support symlinks, but not by default, and it can require special | ||
permissions to do so. | ||
2. It *requires* that a symlink be used, which precludes using otherwise | ||
equivalent mechanisms for creating an executable (e.g. a wrapper script, | ||
hard links, etc). | ||
|
||
In general, symlinks work best when they aren't special cased by consumers. | ||
|
||
Using the term "relocatable" | ||
---------------------------- | ||
|
||
Discussions pointed out that the term "relocatable" is somewhat ambiguous and | ||
misleading for a couple reasons. | ||
|
||
First, absolute paths make a venv arbitrarily relocatable *within* a host, but | ||
not between hosts, so "relocatable" requires *some* qualification for | ||
clarity. | ||
|
||
Second, when using relative paths that point outside the venv, the venv is only | ||
relocatable insofar as those external artifacts are also relocated. This is an | ||
additional nuance that requires qualification of the term. | ||
|
||
To better avoid this confusion, "relative" is chosen, which more naturally | ||
invites the question *"Relative to what?"*. | ||
|
||
|
||
Using PYTHONHOME at runtime to specify home | ||
------------------------------------------- | ||
|
||
Using the ``PYTHONHOME`` environment variable (or any environment variable) is | ||
problematic because it's difficult to know and control when an environment | ||
variable should or shouldn't be inherited by subprocesses. In some cases, it's | ||
not feasible because of how layers of programs calling programs interact. | ||
|
||
Code generally assumes that any virtual environment will be | ||
automatically detected and activated by the presence of ``pyvenv.cfg``, so | ||
things work better when alterations to the environment aren't a concern. | ||
|
||
|
||
Copyright | ||
========= | ||
|
||
This document is placed in the public domain or under the | ||
CC0-1.0-Universal license, whichever is more permissive. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.