Release 2.10.903 (#157)

2.10.903 (2024-10-12)
=====================

- Fixed exception leaks in ExtensionFromHTTP plugins. Now every
extension behave and raise urllib3 own exceptions.
- Added automatic connection downgrade HTTP/2 -> HTTP/1.1 or HTTP/3 ->
(HTTP/2 or HTTP/1.1) in case of known recoverable issues.
jawah/niquests#150
jawah/niquests#134
- Fixed a rare issue where the write semaphore (async context) for a
datagram socket would be locked forever in case of an error.

.github/workflows/ci-dead-things.yml

@@ -41,7 +41,7 @@ jobs:
 
   dead-pip:
     name: Ensure pip <21.2.4
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-20.04
     timeout-minutes: 5
 
     steps:

.github/workflows/ci.yml

@@ -47,7 +47,7 @@ jobs:
           - macos-13
           - windows-2022
           - ubuntu-20.04  # OpenSSL 1.1.1
-          - ubuntu-latest  # OpenSSL 3.0
+          - ubuntu-22.04  # OpenSSL 3.0
         nox-session: ['']
         include:
           - experimental: false
@@ -91,7 +91,7 @@ jobs:
             os: ubuntu-22.04
 
     runs-on: ${{ matrix.os }}
-    name: ${{ fromJson('{"macos-13":"macOS","windows-2022":"Windows","ubuntu-latest":"Ubuntu","ubuntu-20.04":"Ubuntu 20.04 (OpenSSL 1.1.1)","ubuntu-latest":"Ubuntu Latest (OpenSSL 3+)"}')[matrix.os] }} ${{ matrix.python-version }} ${{ matrix.nox-session }}
+    name: ${{ fromJson('{"macos-13":"macOS","windows-2022":"Windows","ubuntu-20.04":"Ubuntu 20.04 (OpenSSL 1.1.1)","ubuntu-22.04":"Ubuntu 22 (OpenSSL 3+)"}')[matrix.os] }} ${{ matrix.python-version }} ${{ matrix.nox-session }}
     continue-on-error: ${{ matrix.experimental }}
     timeout-minutes: 40
     steps:

CHANGES.rst

@@ -1,3 +1,10 @@
+2.10.903 (2024-10-12)
+=====================
+
+- Fixed exception leaks in ExtensionFromHTTP plugins. Now every extension behave and raise urllib3 own exceptions.
+- Added automatic connection downgrade HTTP/2 -> HTTP/1.1 or HTTP/3 -> (HTTP/2 or HTTP/1.1) in case of known recoverable issues.
+- Fixed a rare issue where the write semaphore (async context) for a datagram socket would be locked forever in case of an error.
+
 2.10.902 (2024-10-09)
 =====================
 

README.md

@@ -28,6 +28,7 @@
 - Helpers for retrying requests and dealing with HTTP redirects.
 - Support for gzip, deflate, brotli, and zstd encoding.
 - Support for Python/PyPy 3.7+, no compromise.
+- Automatic Connection Upgrade / Downgrade.
 - Early (Informational) Responses / Hints.
 - HTTP/1.1, HTTP/2 and HTTP/3 support.
 - WebSocket over HTTP/2+ (RFC8441).

docs/index.rst

@@ -15,7 +15,7 @@ urllib3.future
 
 - ⚡ urllib3.future is a powerful, *user-friendly* HTTP client for Python.
 - ⚡ urllib3.future goes beyond supported features while remaining compatible.
-- ⚡ urllib3.future brings many critical features that are missing from both the Python standard libraries and **urllib3**:
+- ⚡ urllib3.future brings many critical features that are missing from both the Python standard libraries and **urllib3**!
 
 
 - Async.
@@ -31,13 +31,19 @@ urllib3.future
 - Helpers for retrying requests and dealing with HTTP redirects.
 - Support for gzip, deflate, brotli, and zstd encoding.
 - Support for Python/PyPy 3.7+, no compromise.
+- Automatic Connection Upgrade / Downgrade.
+- Early (Informational) Responses / Hints.
 - HTTP/1.1, HTTP/2 and HTTP/3 support.
+- WebSocket over HTTP/2+ (RFC8441).
 - Proxy support for HTTP and SOCKS.
+- Post-Quantum Security with QUIC.
 - Detailed connection inspection.
 - HTTP/2 with prior knowledge.
 - Multiplexed connection.
 - Mirrored Sync & Async.
+- Trailer Headers.
 - Amazingly Fast.
+- WebSocket.
 
 urllib3 is powerful and easy to use:
 
@@ -49,6 +55,8 @@ urllib3 is powerful and easy to use:
    200
    >>> resp.data
    b"User-agent: *\nDisallow: /deny\n"
+   >>> resp.version
+   20
 
 Installing
 ----------

src/urllib3/_async/connection.py

@@ -195,6 +195,16 @@ async def _new_conn(self) -> AsyncSocket:  # type: ignore[override]
         """
         await super()._new_conn()
 
+        backup_timeout: float | None = -1.0
+
+        # we want to purposely mitigate the following scenario:
+        #   "A server yield its support for HTTP/2 or HTTP/3 through Alt-Svc, but
+        #    it cannot connect to the alt-svc, thus confusing the end-user on why it
+        #    waits forever for the 2nd request."
+        if self._max_tolerable_delay_for_upgrade is not None:
+            backup_timeout = self.timeout
+            self.timeout = self._max_tolerable_delay_for_upgrade
+
         try:
             sock = await self._resolver.create_connection(
                 (self._dns_host, self.port or self.default_port),
@@ -220,6 +230,9 @@ async def _new_conn(self) -> AsyncSocket:  # type: ignore[override]
             raise NewConnectionError(
                 self, f"Failed to establish a new connection: {e}"
             ) from e
+        finally:
+            if backup_timeout != -1:
+                self.timeout = backup_timeout
 
         # We can, migrate to a DGRAM socket if DNS HTTPS/RR record exist and yield HTTP/3+QUIC support.
         if sock.type == socket.SOCK_DGRAM and self.socket_kind == socket.SOCK_STREAM:

src/urllib3/_async/connectionpool.py

@@ -42,10 +42,12 @@
     InsecureRequestWarning,
     LocationValueError,
     MaxRetryError,
+    MustDowngradeError,
     NewConnectionError,
     ProtocolError,
     ProxyError,
     ReadTimeoutError,
+    RecoverableError,
     SSLError,
     TimeoutError,
 )
@@ -1569,6 +1571,7 @@ async def urlopen(
             SSLError,
             CertificateError,
             ProxyError,
+            RecoverableError,
         ) as e:
             # Discard the connection for these exceptions. It will be
             # replaced during the next _get_conn() call.
@@ -1594,6 +1597,16 @@ async def urlopen(
             )
             await retries.async_sleep()
 
+            # todo: allow the conn to be reusable.
+            #       the MustDowngradeError only means that a single request cannot be
+            #       served over the current svn. does not means all requests to endpoint
+            #       are concerned.
+            if isinstance(new_e, MustDowngradeError) and conn is not None:
+                if "disabled_svn" not in self.conn_kw:
+                    self.conn_kw["disabled_svn"] = set()
+
+                self.conn_kw["disabled_svn"].add(conn._svn)
+
             # Keep track of the error for the retry warning.
             err = e
 

src/urllib3/_version.py

@@ -1,4 +1,4 @@
 # This file is protected via CODEOWNERS
 from __future__ import annotations
 
-__version__ = "2.10.902"
+__version__ = "2.10.903"

src/urllib3/backend/_async/hface.py

@@ -4,6 +4,7 @@
 import typing
 from datetime import datetime, timezone
 from socket import SOCK_DGRAM, SOCK_STREAM
+from socket import timeout as SocketTimeout
 
 try:  # Compiled with SSL?
     import ssl
@@ -41,6 +42,7 @@
     EarlyResponse,
     IncompleteRead,
     InvalidHeader,
+    MustDowngradeError,
     ProtocolError,
     ResponseNotReady,
     SSLError,
@@ -111,8 +113,12 @@ def __init__(
         # h3 specifics
         self.__custom_tls_settings: QuicTLSConfig | None = None
         self.__alt_authority: tuple[str, int] | None = None
+        self.__origin_port: int | None = None
         self.__session_ticket: typing.Any | None = None
 
+        # automatic upgrade shield against errors!
+        self._max_tolerable_delay_for_upgrade: float | None = None
+
     @property
     def is_saturated(self) -> bool:
         if self._protocol is None:
@@ -231,6 +237,24 @@ async def _upgrade(self) -> None:  # type: ignore[override]
                 self.__alt_authority = self.__altsvc_probe(svc="h2")
 
         if self.__alt_authority:
+            # we want to infer a "best delay" to wait for silent upgrade.
+            # for that we use the previous known delay for handshake or establishment.
+            # and apply a "safe" margin of 50%.
+            if (
+                self.conn_info is not None
+                and self.conn_info.established_latency is not None
+            ):
+                self._max_tolerable_delay_for_upgrade = (
+                    self.conn_info.established_latency.total_seconds()
+                )
+                if self.conn_info.tls_handshake_latency is not None:
+                    self._max_tolerable_delay_for_upgrade += (
+                        self.conn_info.tls_handshake_latency.total_seconds()
+                    )
+                self._max_tolerable_delay_for_upgrade *= 10.0
+            else:  # by default (safe/conservative fallback) to 3000ms
+                self._max_tolerable_delay_for_upgrade = 3.0
+
             if upgradable_svn == HttpVersion.h3:
                 if self._preemptive_quic_cache is not None:
                     self._preemptive_quic_cache[
@@ -241,6 +265,7 @@ async def _upgrade(self) -> None:  # type: ignore[override]
                         return
 
             self._svn = upgradable_svn
+            self.__origin_port = self.port or 443
             # We purposely ignore setting the Hostname. Avoid MITM attack from local cache attack.
             self.port = self.__alt_authority[1]
             await self.close()
@@ -504,23 +529,56 @@ async def _post_conn(self) -> None:  # type: ignore[override]
 
             return
 
+        # we want to purposely mitigate the following scenario:
+        #   "A server yield its support for HTTP/2 or HTTP/3 through Alt-Svc, but
+        #    it cannot connect to the alt-svc, thus confusing the end-user on why it
+        #    waits forever for the 2nd request."
+        if self._max_tolerable_delay_for_upgrade is not None:
+            self.sock.settimeout(self._max_tolerable_delay_for_upgrade)
+
         # it may be required to send some initial data, aka. magic header (PRI * HTTP/2..)
         try:
             await self.__exchange_until(
                 HandshakeCompleted,
                 receive_first=False,
             )
-        except ProtocolError as e:
+        except (
+            ProtocolError,
+            TimeoutError,
+            SocketTimeout,
+            ConnectionRefusedError,
+            ConnectionResetError,
+        ) as e:
             if (
                 isinstance(self._protocol, HTTPOverQUICProtocol)
                 and self.__alt_authority is not None
             ):
-                raise ProtocolError(
-                    "The server yielded its support for HTTP/3 through the Alt-Svc header while unable to do so. "
-                    "To remediate that issue, either disable http3 or reach out to the server admin."
+                # we want to remove invalid quic cache capability
+                # because the alt-svc was probably bogus...
+                if (
+                    self._svn == HttpVersion.h3
+                    and self._preemptive_quic_cache is not None
+                ):
+                    alt_key = (self.host, self.__origin_port or 443)
+                    if alt_key in self._preemptive_quic_cache:
+                        del self._preemptive_quic_cache[alt_key]
+
+                # this avoid the close() to attempt re-use the (dead) sock
+                self._protocol = None
+
+                raise MustDowngradeError(
+                    f"The server yielded its support for {self._svn} through the Alt-Svc header while unable to do so. "
+                    f"To remediate that issue, either disable {self._svn} or reach out to the server admin."
                 ) from e
             raise
 
+        if self._max_tolerable_delay_for_upgrade is not None:
+            self.sock.settimeout(self.timeout)
+
+        self._max_tolerable_delay_for_upgrade = (
+            None  # upgrade went fine. discard the value!
+        )
+
         if isinstance(self._protocol, HTTPOverQUICProtocol):
             self.conn_info.certificate_der = self._protocol.getpeercert(
                 binary_form=True
@@ -775,6 +833,16 @@ async def __exchange_until(
 
                     raise ProtocolError(event.message)
                 elif stream_related_event and isinstance(event, StreamResetReceived):
+                    # we want to catch MUST_USE_HTTP_1_1 or H3_VERSION_FALLBACK
+                    # HTTP/2 https://www.rfc-editor.org/rfc/rfc9113.html#name-error-codes
+                    # HTTP/3 https://www.iana.org/assignments/http3-parameters/http3-parameters.xhtml#http3-parameters-error-codes
+                    if (self._svn == HttpVersion.h2 and event.error_code == 0xD) or (
+                        self._svn == HttpVersion.h3 and event.error_code == 0x0110
+                    ):
+                        raise MustDowngradeError(
+                            f"The remote server is unable to serve this resource over {self._svn}"
+                        )
+
                     raise ProtocolError(
                         f"Stream {event.stream_id} was reset by remote peer. Reason: {hex(event.error_code)}."
                     )