🔖 Release 2.8.906 (#143)

- Removed opinionated OpenSSL version constraint that forbid any version lower than 1.1.1. The reasoning behind this is that some companies expressed (to us) the need to upgrade urllib3 to urllib3-future in (very) old Python 3.7 built against patched OpenSSL 1.0.2 or 1.0.8 and collaborative testing showed us that this constraint is overly protective. Those build often lack TLS 1.3 support and may contain major vulnerabilities, but we have to be optimistic on their awareness. TLS 1.3 / QUIC is also an option for them as it works out of the box on those old distributions. Effective immediately, we added a dedicated pipeline in our CI to verify that urllib3-future works with the oldest Python 3.7 build we found out there. Blindly removing support for those libraries when supporting Python 3.7 ... 3.9 is as we "partially" support this range and end-users have no to little clues for why it's rejected when it clearly works. The only issue that can appear is for users that have Python built against a SSL library that does not support either TLS 1.2 or 1.3, they will encounter errors for sure. - Changed to submodule http2 to subpackage http2. Purely upstream sync. Still no use for us. - Changed minimum (C)Python interpreter version for qh3 automatic pickup to 3.7.11 as it bundle pip 21.2.4 and is the minimum version to pick an appropriate (abi3) pre-built wheel. You may still install ``qh3`` manually by first upgrading your pip installation by running ``python -m pip install -U pip``. - Fixed an issue where a server is yielding an invalid/malformed ``Alt-Svc`` header and urllib3-future may crash upon it. - Fixed an issue where sending a ``str`` body using a ``bytes`` value for Content-Type would induce a crash. This was due to our unicode transparency policy. See #142
jawah · Aug 15, 2024 · b6dc334 · b6dc334
1 parent f2c8bd0
commit b6dc334
Show file tree

Hide file tree

Showing 41 changed files with 361 additions and 140 deletions.
diff --git a/.github/workflows/ci-dead-things.yml b/.github/workflows/ci-dead-things.yml
@@ -0,0 +1,60 @@
+name: CI for EOL 3rd parties
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+
+permissions: "read-all"
+
+concurrency:
+  group: ci-dead-things-${{ github.ref_name }}
+  cancel-in-progress: true
+
+jobs:
+
+  dead-libressl:
+    name: Ensure LibreSSL <2.8
+    runs-on: ubuntu-latest
+    timeout-minutes: 25
+    continue-on-error: true
+
+    steps:
+      - name: "Checkout repository"
+        uses: "actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608"
+
+      - name: "Run against Python 3.7 built with LibreSSL 2.7.4"
+        run: ./ci/run_legacy_libressl.sh
+
+  dead-openssl:
+    name: Ensure OpenSSL <1.1.1
+    runs-on: ubuntu-latest
+    timeout-minutes: 25
+
+    steps:
+      - name: "Checkout repository"
+        uses: "actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608"
+
+      - name: "Run against Python 3.7 built with OpenSSL 1.0.2q"
+        run: ./ci/run_legacy_openssl.sh
+
+  dead-pip:
+    name: Ensure pip <21.2.4
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+
+    steps:
+      - name: "Checkout repository"
+        uses: "actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608"
+
+      - name: "Setup Python"
+        uses: "actions/setup-python@61a6322f88396a6271a6ee3565807d608ecaddd1"
+        with:
+          python-version: "3.7"
+
+      - name: "Enforce pip 20.x"
+        run: python -m pip install "pip<21"
+
+      - name: "Ensure that urllib3-future can be installed"
+        run: python -m pip install .
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -97,8 +97,12 @@ jobs:
         uses: "actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608"
 
       - name: "Traefik: Prerequisites - Colima (MacOS)"
+        uses: nick-fields/retry@v3
         if: ${{ matrix.traefik-server && contains(matrix.os, 'mac') }}
-        run: ./traefik/macos.sh
+        with:
+          timeout_minutes: 10
+          max_attempts: 3
+          command: ./traefik/macos.sh
 
       - name: "Setup Python ${{ matrix.python-version }}"
         uses: "actions/setup-python@61a6322f88396a6271a6ee3565807d608ecaddd1"

diff --git a/CHANGES.rst b/CHANGES.rst
@@ -1,3 +1,26 @@
+2.8.906 (2024-08-15)
+====================
+
+- Removed opinionated OpenSSL version constraint that forbid any version lower than 1.1.1.
+  The reasoning behind this is that some companies expressed (to us) the need to upgrade urllib3 to urllib3-future
+  in (very) old Python 3.7 built against patched OpenSSL 1.0.2 or 1.0.8 and collaborative testing showed us
+  that this constraint is overly protective. Those build often lack TLS 1.3 support and may contain
+  major vulnerabilities, but we have to be optimistic on their awareness.
+  TLS 1.3 / QUIC is also an option for them as it works out of the box on those old distributions.
+  Effective immediately, we added a dedicated pipeline in our CI to verify that urllib3-future works
+  with the oldest Python 3.7 build we found out there.
+  Blindly removing support for those libraries when supporting Python 3.7 ... 3.9 is as we "partially"
+  support this range and end-users have no to little clues for why it's rejected when it clearly works.
+  The only issue that can appear is for users that have Python built against a SSL library that does not
+  support either TLS 1.2 or 1.3, they will encounter errors for sure.
+- Changed to submodule http2 to subpackage http2. Purely upstream sync. Still no use for us.
+- Changed minimum (C)Python interpreter version for qh3 automatic pickup to 3.7.11 as it bundle pip 21.2.4 and
+  is the minimum version to pick an appropriate (abi3) pre-built wheel. You may still install ``qh3`` manually
+  by first upgrading your pip installation by running ``python -m pip install -U pip``.
+- Fixed an issue where a server is yielding an invalid/malformed ``Alt-Svc`` header and urllib3-future may crash upon it.
+- Fixed an issue where sending a ``str`` body using a ``bytes`` value for Content-Type would induce a crash.
+  This was due to our unicode transparency policy. See https://github.com/jawah/urllib3.future/pull/142
+
 2.8.905 (2024-08-04)
 ====================
 

diff --git a/LibreSSL.Dockerfile b/LibreSSL.Dockerfile
@@ -0,0 +1,20 @@
+FROM python:3.7.1-alpine3.8
+
+RUN apk add build-base libffi-dev
+
+WORKDIR /app
+
+RUN pip install --upgrade pip setuptools
+RUN pip install nox
+
+ENV TRAEFIK_HTTPBIN_ENABLE=false
+ENV CI=true
+
+COPY ./src/urllib3 src/urllib3/
+COPY ./test test/
+COPY ./dummyserver dummyserver/
+COPY ./ci ci/
+
+COPY noxfile.py LICENSE.txt pyproject.toml README.md setup.cfg hatch_build.py dev-requirements.txt mypy-requirements.txt ./
+
+CMD nox -s test-3.7
diff --git a/OpenSSL.Dockerfile b/OpenSSL.Dockerfile
@@ -0,0 +1,20 @@
+FROM python:3.7.2-alpine3.7
+
+RUN apk add build-base libffi-dev
+
+WORKDIR /app
+
+RUN pip install --upgrade pip setuptools
+RUN pip install nox
+
+ENV TRAEFIK_HTTPBIN_ENABLE=false
+ENV CI=true
+
+COPY ./src/urllib3 src/urllib3/
+COPY ./test test/
+COPY ./dummyserver dummyserver/
+COPY ./ci ci/
+
+COPY noxfile.py LICENSE.txt pyproject.toml README.md setup.cfg hatch_build.py dev-requirements.txt mypy-requirements.txt ./
+
+CMD nox -s test-3.7
diff --git a/README.md b/README.md
@@ -20,6 +20,7 @@
 - Thread safety.
 - Happy Eyeballs.
 - Connection pooling.
+- Unopinionated about OpenSSL.
 - Client-side SSL/TLS verification.
 - Highly customizable DNS resolution.
 - File uploads with multipart encoding.
@@ -164,6 +165,7 @@ Here are some of the reasons (not exhaustive) we choose to work this way:
 5. There a ton of other improvement you may leverage, but for that you will need to migrate to Niquests or update your code
   to enable specific capabilities, like but not limited to: "DNS over QUIC, HTTP" / "Happy Eyeballs" / "Native Asyncio" / "Advanced Multiplexing".
 6. Non-blocking IO with concurrent streams/requests. And yes, transparently.
+7. It relaxes some constraints established by upstream in their version 2, thus making it easier to upgrade from version 1.
 
 - **Is this funded?**
 

diff --git a/ci/deploy.sh b/ci/deploy.sh
diff --git a/ci/run_legacy_libressl.sh b/ci/run_legacy_libressl.sh
@@ -0,0 +1,3 @@
+#!/usr/bin/env sh
+docker build -f LibreSSL.Dockerfile -t urllib3:legacy-libressl .
+docker run urllib3:legacy-libressl
diff --git a/ci/run_legacy_openssl.sh b/ci/run_legacy_openssl.sh
@@ -0,0 +1,3 @@
+#!/usr/bin/env sh
+docker build -f OpenSSL.Dockerfile -t urllib3:legacy-openssl .
+docker run urllib3:legacy-openssl
diff --git a/dev-requirements.txt b/dev-requirements.txt
@@ -7,9 +7,10 @@ pytest-timeout==2.3.1
 trustme==0.9.0
 # We have to install at most cryptography 39.0.2 for PyPy<7.3.10
 # versions of Python 3.7, 3.8, and 3.9.
-cryptography==39.0.2;implementation_name=="pypy" and implementation_version<"7.3.10"
-cryptography==42.0.5;implementation_name!="pypy" or implementation_version>="7.3.10"
-backports.zoneinfo==0.2.1;python_version<"3.9"
+cryptography==39.0.2; implementation_name=="pypy" and implementation_version<"7.3.10"
+cryptography==42.0.5; implementation_name!="pypy" or implementation_version>="7.3.10"
+backports.zoneinfo==0.2.1; python_version<"3.9"
+tzdata==2024.1; python_version<"3.8"
 towncrier==21.9.0
 pytest-asyncio>=0.21.1,<=0.23.5.post1
 # unblock cffi for Python 3.13

diff --git a/docs/conf.py b/docs/conf.py
@@ -157,4 +157,5 @@
     ("py:class", "AsyncTrafficPolice"),
     ("py:attr", "HTTPResponse.data"),
     ("py:class", "_TYPE_PEER_CERT_RET_DICT"),
+    ("py:class", "_TYPE_ASYNC_BODY"),
 ]
diff --git a/docs/index.rst b/docs/index.rst
@@ -21,18 +21,23 @@ urllib3.future
 - Async.
 - Task safety.
 - Thread safety.
+- Happy Eyeballs.
 - Connection pooling.
+- Unopinionated about OpenSSL.
 - Client-side SSL/TLS verification.
 - Highly customizable DNS resolution.
 - File uploads with multipart encoding.
 - DNS over UDP, TLS, QUIC, or HTTPS. DNSSEC protected.
 - Helpers for retrying requests and dealing with HTTP redirects.
 - Support for gzip, deflate, brotli, and zstd encoding.
+- Support for Python/PyPy 3.7+, no compromise.
 - HTTP/1.1, HTTP/2 and HTTP/3 support.
 - Proxy support for HTTP and SOCKS.
 - Detailed connection inspection.
+- HTTP/2 with prior knowledge.
 - Multiplexed connection.
 - Mirrored Sync & Async.
+- Amazingly Fast.
 
 urllib3 is powerful and easy to use:
 

diff --git a/docs/user-guide.rst b/docs/user-guide.rst
@@ -16,7 +16,7 @@ urllib3 can be installed with `pip <https://pip.pypa.io>`_
 HTTP/2 and HTTP/3 support
 -------------------------
 
-HTTP/2 support is enabled by default via the ``h2`` dependency, HTTP/3 may or not be
+HTTP/2 support is enabled by default via the ``jh2`` dependency, HTTP/3 may or not be
 automatically available depending on the availability of the wheel on your platform.
 
 .. code-block:: bash
@@ -29,6 +29,8 @@ This may require some external toolchain to be available (compilation).
 
 .. caution:: If the requirements aren't fulfilled for HTTP/3, your package manager won't pick qh3 for installation when installing urllib3-future and it will be silently disabled. We choose not to impose compilation and keep a safe pure Python fallback.
 
+.. note:: Very old ``pip`` versions may not be able to pick the pre-built wheel accordingly. Make sure to have the latest ``pip`` version installed first.
+
 Making Requests
 ---------------
 

diff --git a/docs/v2-migration-guide.rst b/docs/v2-migration-guide.rst
@@ -23,11 +23,10 @@ What are the important changes?
 Here's a short summary of which changes in urllib3 v2.0 are most important:
 
 - Python version must be **3.7 or later** (previously supported Python 2.7, 3.5, and 3.6).
-- Removed support for OpenSSL versions older than 1.1.1.
 - Removed support for Python implementations that aren't CPython or PyPy3 (previously supported Google App Engine, Jython).
 - Removed the ``urllib3.contrib.ntlmpool`` module.
 - Deprecated the ``urllib3.contrib.pyopenssl`` module, made inoperant in v2.1.0.
-- Deprecated the ``urllib3.contrib.securetransport`` module, will be removed in v2.1.0.
+- Deprecated the ``urllib3.contrib.securetransport`` module, is removed in v2.1.0.
 - Deprecated the ``urllib3[secure]`` extra, made inoperant in v2.1.0.
 - Deprecated the ``HTTPResponse.getheaders()`` method in favor of ``HTTPResponse.headers``, will be removed in v2.1.0.
 - Deprecated the ``HTTPResponse.getheader(name, default)`` method in favor of ``HTTPResponse.headers.get(name, default)``, will be removed in v2.1.0.
@@ -149,34 +148,10 @@ immediately you will `receive security fixes on the 1.26.x release stream <#secu
 **🤔 Common upgrading issues**
 -------------------------------
 
-ssl module is compiled with OpenSSL 1.0.2.k-fips
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-.. code-block:: text
-
-  ImportError: urllib3 v2.0 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'OpenSSL 1.0.2k-fips  26 Jan 2017'.
-  See: https://github.com/urllib3/urllib3/issues/2168
-
-Remediation depends on your system:
-
-- **AWS Lambda**: Upgrade to the Python3.10 runtime as it uses OpenSSL 1.1.1. Alternatively, you can
-  use a `custom Docker image
-  <https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/>`_ and ensure you
-  use a Python build that uses OpenSSL 1.1.1 or later.
-- **Amazon Linux 2**: Upgrade to `Amazon Linux 2023
-  <https://aws.amazon.com/linux/amazon-linux-2023/>`_. Alternatively, you can install OpenSSL 1.1.1
-  on Amazon Linux 2 using ``yum install openssl11 openssl11-devel`` and then install Python with a
-  tool like pyenv.
-- **Red Hat Enterpritse Linux 7 (RHEL 7)**: Upgrade to RHEL 8 or RHEL 9.
-- **Read the Docs**: Upgrade your `configuration file to use Ubuntu 22.04
-  <https://docs.readthedocs.io/en/stable/config-file/v2.html>`_ by using ``os: ubuntu-22.04`` in the
-  ``build`` section. Feel free to use the `urllib3 configuration
-  <https://github.com/urllib3/urllib3/blob/2.0.0/.readthedocs.yml>`_ as an inspiration.
-
 docker.errors.dockerexception: error while fetching server api version: request() got an unexpected keyword argument 'chunked'
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Upgrade to ``docker==6.1.0`` that is compatible with urllib3 2.0.
+Upgrade to ``docker==6.1.0`` that is compatible with urllib3(-future) 2+
 
 ImportError: cannot import name 'gaecontrib' from 'requests_toolbelt._compat'
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -187,19 +162,6 @@ Engine Standard Python 2.7 support. Most users that reported this issue were usi
 library is unmaintained, but `replacements exist
 <https://github.com/thisbejim/Pyrebase/issues/435>`_.
 
-``ImportError: cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_'``
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-This likely happens because you're using botocore which `does not support urllib3 2.0 yet
-<https://github.com/boto/botocore/issues/2921>`_. The good news is that botocore explicitly declares
-in its dependencies that it only supports ``urllib3<2``. Make sure to use a recent pip. That way, pip
-will install urllib3 1.26.x until botocore starts supporting urllib3 2.0.
-
-If you're deploying to an AWS environment such as Lambda or a host using Amazon Linux 2,
-you'll need to explicitly pin to ``urllib3<2`` in your project to ensure urllib3 2.0 isn't
-brought into your environment. Otherwise, this may result in unintended side effects with
-the default boto3 installation.
-
 AttributeError: module 'urllib3.connectionpool' has no attribute 'VerifiedHTTPSConnection'
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

diff --git a/dummyserver/server.py b/dummyserver/server.py
@@ -279,7 +279,7 @@ async def run() -> None:
         # run asyncio.run in a thread and collect exceptions from *either*
         # the loop failing to start, or failing to close
         ran = tpe.submit(_run_and_close_tornado, run)
-        for f in concurrent.futures.as_completed((loop_started, ran)):  # type: ignore[misc]
+        for f in concurrent.futures.as_completed((loop_started, ran)):
             if f is loop_started:
                 io_loop, stop_event = loop_started.result()
                 try:

diff --git a/mypy-requirements.txt b/mypy-requirements.txt
@@ -1,4 +1,4 @@
-mypy==1.8.0; python_version >= '3.8'
+mypy==1.11.1; python_version >= '3.8'
 mypy==1.4.1; python_version < '3.8'
 idna>=2.0.0
 cryptography==42.0.5

diff --git a/noxfile.py b/noxfile.py
@@ -196,11 +196,12 @@ def traefik_boot(session: nox.Session) -> typing.Generator[None, None, None]:
 
 def tests_impl(
     session: nox.Session,
-    extras: str = "socks,brotli",
+    extras: str = "socks,brotli,zstd",
     byte_string_comparisons: bool = False,
 ) -> None:
     with traefik_boot(session):
         # Install deps and the package itself.
+        session.install("-U", "pip", "setuptools", silent=False)
         session.install("-r", "dev-requirements.txt", silent=False)
         session.install(f".[{extras}]", silent=False)
 
@@ -209,6 +210,7 @@ def tests_impl(
         # Print the Python version and bytesize.
         session.run("python", "--version")
         session.run("python", "-c", "import struct; print(struct.calcsize('P') * 8)")
+        session.run("python", "-c", "import ssl; print(ssl.OPENSSL_VERSION)")
 
         # Inspired from https://hynek.me/articles/ditch-codecov-python/
         # We use parallel mode and then combine in a later CI step

diff --git a/pyproject.toml b/pyproject.toml
@@ -40,7 +40,7 @@ classifiers = [
 requires-python = ">=3.7"
 dynamic = ["version"]
 dependencies = [
-  "qh3>=1.0.3,<2.0.0; (platform_system == 'Darwin' or platform_system == 'Windows' or platform_system == 'Linux') and (platform_machine == 'x86_64' or platform_machine == 's390x' or platform_machine == 'aarch64' or platform_machine == 'armv7l' or platform_machine == 'ppc64le' or platform_machine == 'ppc64' or platform_machine == 'AMD64' or platform_machine == 'arm64' or platform_machine == 'ARM64') and (platform_python_implementation == 'CPython' or (platform_python_implementation == 'PyPy' and python_version < '3.11'))",
+  "qh3>=1.0.3,<2.0.0; (platform_python_implementation != 'CPython' or python_full_version > '3.7.10') and (platform_system == 'Darwin' or platform_system == 'Windows' or platform_system == 'Linux') and (platform_machine == 'x86_64' or platform_machine == 's390x' or platform_machine == 'aarch64' or platform_machine == 'armv7l' or platform_machine == 'ppc64le' or platform_machine == 'ppc64' or platform_machine == 'AMD64' or platform_machine == 'arm64' or platform_machine == 'ARM64') and (platform_python_implementation == 'CPython' or (platform_python_implementation == 'PyPy' and python_version < '3.11'))",
   "h11>=0.11.0,<1.0.0",
   "jh2>=5.0.3,<6.0.0",
 ]
@@ -103,8 +103,6 @@ markers = ["limit_memory"]
 log_level = "DEBUG"
 filterwarnings = [
     "error",
-    '''default:urllib3 v2.0 only supports OpenSSL 1.1.1+.*''',
-    '''default:'urllib3\[secure\]' extra is deprecated and will be removed in urllib3 v2\.1\.0.*:DeprecationWarning''',
     '''default:No IPv6 support. Falling back to IPv4:urllib3.exceptions.HTTPWarning''',
     '''default:No IPv6 support. skipping:urllib3.exceptions.HTTPWarning''',
     '''default:ssl\.TLSVersion\.TLSv1 is deprecated:DeprecationWarning''',
@@ -126,6 +124,7 @@ filterwarnings = [
     '''ignore:A plugin raised an exception during''',
     '''ignore:Exception ignored in:pytest.PytestUnraisableExceptionWarning''',
     '''ignore:Exception in thread:pytest.PytestUnhandledThreadExceptionWarning''',
+    '''ignore:function _SSLProtocolTransport\.__del__:pytest.PytestUnraisableExceptionWarning''',
     '''ignore:The `hash` argument is deprecated in favor of `unsafe_hash`:DeprecationWarning''',
 ]
 

diff --git a/src/urllib3/__init__.py b/src/urllib3/__init__.py
@@ -31,26 +31,6 @@
 from .util.retry import Retry
 from .util.timeout import Timeout
 
-# Ensure that Python is compiled with OpenSSL 1.1.1+
-# If the 'ssl' module isn't available at all that's
-# fine, we only care if the module is available.
-try:
-    import ssl
-except ImportError:
-    pass
-else:
-    if ssl.OPENSSL_VERSION.startswith("OpenSSL ") and ssl.OPENSSL_VERSION_INFO < (
-        1,
-        1,
-        1,
-    ):  # Defensive:
-        raise ImportError(
-            "urllib3 v2.0 only supports OpenSSL 1.1.1+, currently "
-            f"the 'ssl' module is compiled with {ssl.OPENSSL_VERSION!r}. "
-            "See: https://github.com/urllib3/urllib3/issues/2168"
-        )
-
-
 __author__ = "Andrey Petrov ([email protected])"
 __license__ = "MIT"
 __version__ = __version__