✔️ improve pypy support (#94)

.github/workflows/run-tests.yml

@@ -20,13 +20,16 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        python-version: ["3.7", "3.8", "3.9", "3.10", "3.11", "3.12", "pypy-3.8", "pypy-3.9", "pypy-3.10"]
+        python-version: ["3.7", "3.8", "3.9", "3.10", "3.11", "3.12", "pypy-3.9", "pypy-3.10"]
         os: [ubuntu-latest, macOS-latest, windows-latest]
         include:
-          # pypy-3.7 on Windows and Mac OS currently fails trying to compile
-          # cryptography. Moving pypy-3.7 to only test linux.
+          # pypy-3.7, pypy-3.8 may fail due to missing cryptography wheels. Adapting.
           - python-version: pypy-3.7
             os: ubuntu-latest
+          - python-version: pypy-3.8
+            os: ubuntu-latest
+          - python-version: pypy-3.8
+            os: macOS-latest
 
     steps:
     - uses: actions/checkout@8ade135a41bc03ea155e62e844d188df1ea18608 # v4.1.0

HISTORY.md

@@ -1,6 +1,16 @@
 Release History
 ===============
 
+3.5.3 (2024-03-06)
+------------------
+
+**Fixed**
+- A rare error that occurs on PyPy, especially on Windows, complaining about a missing release call.
+
+**Misc**
+- Allow latest dependencies version for httpbin, Flask and werkzeug in tests.
+- Remove wheel from test dependencies.
+
 3.5.2 (2024-03-05)
 ------------------
 

README.md

@@ -39,18 +39,6 @@ Niquests, is the “**Safest**, **Fastest[^10]**, **Easiest**, and **Most advanc
 | `Package / SLSA Signed`             |    ✅     |     ❌     |       ❌       | ✅             |
 </details>
 
-[^1]: aiohttp has no support for synchronous request.
-[^2]: requests has no support for asynchronous request.
-[^3]: while the HTTP/2 connection object can handle concurrent requests, you cannot leverage its true potential.
-[^4]: loading client certificate without file can't be done.
-[^5]: httpx officially claim to be thread safe but recent tests demonstrate otherwise as of february 2024.
-[^6]: they do not expose anything to control network aspects such as IPv4/IPv6 toggles, and timings (e.g. DNS response time, established delay, TLS handshake delay, etc...) and such.
-[^7]: while advertised as possible, they refuse to make it the default due to performance issues. as of february 2024 an extra is required to enable it manually.
-[^8]: they don't support HTTP/3 at all.
-[^9]: you must use a custom DNS resolver so that it can preemptively connect using HTTP/3 over QUIC when remote is compatible.
-[^10]: performance measured when leveraging a multiplexed connection with or without uses of any form of concurrency as of november 2023. The research compared `httpx`, `requests`, `aiohttp` against `niquests`.
-[^11]: enabled when using a custom DNS resolver.
-
 ```python
 >>> import niquests
 >>> s = niquests.Session(resolver="doh+google://", multiplexed=True)
@@ -74,15 +62,19 @@ True
 >>> r.conn_info.established_latency
 datetime.timedelta(microseconds=38)
 ```
-or using async/await! <small>you'll need to enclose the code within proper async function, see the docs for more.</small>
+or using async/await!
 ```python
 import niquests
->>> s = niquests.AsyncSession(resolver="doh+google://")
->>> r = await s.get('https://pie.dev/basic-auth/user/pass', auth=('user', 'pass'), stream=True)
->>> r
-<Response HTTP/3 [200]>
->>> await r.json()
-{'authenticated': True, ...}
+import asyncio
+
+async def main() -> None:
+    async with niquests.AsyncSession(resolver="doh+google://") as s:
+        r = await s.get('https://pie.dev/basic-auth/user/pass', auth=('user', 'pass'), stream=True)
+        print(r)  # Output: <Response HTTP/3 [200]>
+        payload = await r.json()
+        print(payload)  # Output: {'authenticated': True, ...}
+
+asyncio.run(main())
 ```
 
 Niquests allows you to send HTTP requests extremely easily. There’s no need to manually add query strings to your URLs, or to form-encode your `PUT` & `POST` data — just use the `json` method!
@@ -147,15 +139,25 @@ How about a nice refresher with a mere `CTRL+H` _import requests_ **to** _import
 ## 💼 For Enterprise
 
 Professional support for Niquests is available as part of the [Tidelift
-Subscription][12]. Tidelift gives software development teams a single source for
+Subscription](https://tidelift.com/subscription/pkg/pypi-niquests?utm_source=pypi-niquests&utm_medium=readme). Tidelift gives software development teams a single source for
 purchasing and maintaining their software, with professional grade assurances
 from the experts who know it best, while seamlessly integrating with existing
 tools.
 
-[12]: https://tidelift.com/subscription/pkg/pypi-niquests?utm_source=pypi-niquests&utm_medium=readme
-
 You may also be interested in unlocking specific advantages by looking at our [GitHub sponsor tiers](https://github.com/sponsors/Ousret).
 
 ---
 
 Niquests is a highly improved HTTP client that is based (forked) on Requests. The previous project original author is Kenneth Reitz and actually left the maintenance of Requests years ago.
+
+[^1]: aiohttp has no support for synchronous request.
+[^2]: requests has no support for asynchronous request.
+[^3]: while the HTTP/2 connection object can handle concurrent requests, you cannot leverage its true potential.
+[^4]: loading client certificate without file can't be done.
+[^5]: httpx officially claim to be thread safe but recent tests demonstrate otherwise as of february 2024.
+[^6]: they do not expose anything to control network aspects such as IPv4/IPv6 toggles, and timings (e.g. DNS response time, established delay, TLS handshake delay, etc...) and such.
+[^7]: while advertised as possible, they refuse to make it the default due to performance issues. as of february 2024 an extra is required to enable it manually.
+[^8]: they don't support HTTP/3 at all.
+[^9]: you must use a custom DNS resolver so that it can preemptively connect using HTTP/3 over QUIC when remote is compatible.
+[^10]: performance measured when leveraging a multiplexed connection with or without uses of any form of concurrency as of november 2023. The research compared `httpx`, `requests`, `aiohttp` against `niquests`.
+[^11]: enabled when using a custom DNS resolver.

docs/community/faq.rst

@@ -43,3 +43,56 @@ Python 3 already includes native support for SNI in their SSL modules.
 
 .. _`Server-Name-Indication`: https://en.wikipedia.org/wiki/Server_Name_Indication
 .. _`virtual hosting`: https://en.wikipedia.org/wiki/Virtual_hosting
+
+
+What are "OverwhelmedTraffic" errors?
+-------------------------------------
+
+You may witness: " Cannot select a disposable connection to ease the charge ".
+
+Basically, it means that your pool of connections is saturated and we were unable to open a new connection.
+If you wanted to run 32 threads sharing the same ``Session`` objects, you want to allow
+up to 32 connections per host.
+
+Do as follow::
+
+    import niquests
+
+    with niquests.Session(pool_maxsize=32) as s:
+        ...
+
+
+What is "urllib3.future"?
+-------------------------
+
+It is a fork of the well know **urllib3** library, you can easily imagine that
+Niquests would have been completely unable to serve that much feature with the
+existing **urllib3** library.
+
+**urllib3.future** is independent, managed separately and completely compatible with
+its counterpart (API-wise).
+
+Shadow-Naming
+~~~~~~~~~~~~~
+
+Your environment may or may not include the legacy urllib3 package in addition to urllib3.future.
+So doing::
+
+    import urllib3
+
+May actually import either urllib3 or urllib3.future.
+But fear not, if your script was compatible with urllib3, it will most certainly work
+out-of-the-box with urllib3.future.
+
+This behavior was chosen to ensure the highest level of compatibility with your migration,
+ensuring the minimum friction during the migration between Requests and Niquests.
+
+Cohabitation
+~~~~~~~~~~~~
+
+You may have both urllib3 and urllib3.future installed if wished.
+Niquests will use the secondary entrypoint for urllib3.future internally.
+
+It does not change anything for you. You may still pass ``urllib3.Retry`` and
+``urllib3.Timeout`` regardless of the cohabitation, Niquests will do
+the translation internally.

docs/community/recommended.rst

@@ -25,6 +25,7 @@ requested by users within the community.
 
 .. _Requests-Toolbelt: https://toolbelt.readthedocs.io/en/latest/index.html
 
+.. warning:: Requests-Toolbelt actually require Requests as a dependency thus making you have duplicate dependencies.
 
 Requests-OAuthlib
 -----------------

pyproject.toml

@@ -110,5 +110,5 @@ filterwarnings = [
     '''ignore:unclosed .*:ResourceWarning''',
     '''ignore:Parsed a negative serial number:cryptography.utils.CryptographyDeprecationWarning''',
     '''ignore:A plugin raised an exception during an old-style hookwrapper teardown''',
-    '''ignore:Exception ignored in:pytest.PytestUnraisableExceptionWarning''',
+    '''ignore:.*:pytest.PytestUnraisableExceptionWarning''',
 ]

requirements-dev.txt

@@ -3,8 +3,7 @@ pytest>=2.8.0,<=7.4.4
 pytest-cov
 pytest-httpbin==2.0.0
 pytest-asyncio>=0.21.1,<1.0
-httpbin==0.10.1
+httpbin==0.10.2
 trustme
-wheel
 cryptography<40.0.0; python_version <= '3.7' and platform_python_implementation == 'PyPy'
-werkzeug<3
+werkzeug<4

src/niquests/__version__.py

@@ -9,9 +9,9 @@
 __url__: str = "https://niquests.readthedocs.io"
 
 __version__: str
-__version__ = "3.5.2"
+__version__ = "3.5.3"
 
-__build__: int = 0x030502
+__build__: int = 0x030503
 __author__: str = "Kenneth Reitz"
 __author_email__: str = "[email protected]"
 __license__: str = "Apache-2.0"

src/niquests/adapters.py

@@ -840,8 +840,19 @@ def send(
                 multiplexed=multiplexed,
             )
 
+            # branch for urllib3.future 2.5+ with advanced conn/multiplexing scheduler/mapper. aka. TrafficPolice.
+            # we are bypassing the PoolManager.request to directly invoke the concerned HttpPool, so we missed
+            # a required call to TrafficPolice::memorize(...).
             if hasattr(self.poolmanager.pools, "memorize"):
-                self.poolmanager.pools.memorize(resp_or_promise, conn)
+                proxy = select_proxy(request.url, proxies)
+
+                if proxy is not None:
+                    self.proxy_manager[proxy].pools.memorize(resp_or_promise, conn)
+                    self.proxy_manager[proxy].pools.release()
+                else:
+                    self.poolmanager.pools.memorize(resp_or_promise, conn)
+                    self.poolmanager.pools.release()
+
         except (ProtocolError, OSError) as err:
             if "illegal header" in str(err).lower():
                 raise InvalidHeader(err, request=request)
@@ -1267,7 +1278,7 @@ def __init__(
         self._max_in_flight_multiplexed = (
             max_in_flight_multiplexed
             if max_in_flight_multiplexed is not None
-            else self._pool_connections * 124
+            else self._pool_connections * 250
         )
 
         disabled_svn = set()
@@ -1748,8 +1759,10 @@ async def send(
 
                 if proxy is not None:
                     self.proxy_manager[proxy].pools.memorize(resp_or_promise, conn)
+                    self.proxy_manager[proxy].pools.release()
                 else:
                     self.poolmanager.pools.memorize(resp_or_promise, conn)
+                    self.poolmanager.pools.release()
 
         except (ProtocolError, OSError) as err:
             if "illegal header" in str(err).lower():

tests/test_lowlevel.py

@@ -1,5 +1,7 @@
 from __future__ import annotations
 
+import platform
+import sys
 import threading
 from json import JSONDecodeError
 
@@ -122,6 +124,15 @@ def multiple_content_length_response_handler(sock):
         close_server.set()
 
 
+@pytest.mark.xfail(
+    platform.python_implementation() == "PyPy"
+    and sys.version_info
+    < (
+        3,
+        8,
+    ),
+    reason="PyPy 3.7 bug with socket unexpected close server side",
+)
 def test_digestauth_401_count_reset_on_redirect():
     """Ensure we correctly reset num_401_calls after a successful digest auth,
     followed by a 302 redirect to another digest auth prompt.
@@ -188,6 +199,15 @@ def digest_response_handler(sock):
         close_server.set()
 
 
+@pytest.mark.xfail(
+    platform.python_implementation() == "PyPy"
+    and sys.version_info
+    < (
+        3,
+        8,
+    ),
+    reason="PyPy 3.7 bug with socket unexpected close server side",
+)
 def test_digestauth_401_only_sent_once():
     """Ensure we correctly respond to a 401 challenge once, and then
     stop responding if challenged again.
@@ -310,6 +330,15 @@ def test_use_proxy_from_environment(httpbin, var, scheme):
         assert len(fake_proxy.handler_results[0]) > 0
 
 
+@pytest.mark.xfail(
+    platform.python_implementation() == "PyPy"
+    and sys.version_info
+    < (
+        3,
+        8,
+    ),
+    reason="PyPy 3.7 bug with socket unexpected close server side",
+)
 def test_redirect_rfc1808_to_non_ascii_location():
     path = "š"
     expected_path = b"%C5%A1"
@@ -367,6 +396,15 @@ def test_fragment_not_sent_with_request():
         close_server.set()
 
 
+@pytest.mark.xfail(
+    platform.python_implementation() == "PyPy"
+    and sys.version_info
+    < (
+        3,
+        8,
+    ),
+    reason="PyPy 3.7 bug with socket unexpected close server side",
+)
 def test_fragment_update_on_redirect():
     """Verify we only append previous fragment if one doesn't exist on new
     location. If a new fragment is encountered in a Location header, it should

tests/testserver/server.py

@@ -112,7 +112,7 @@ def _accept_connection(self):
                 return None
 
             return self.server_sock.accept()[0]
-        except OSError:
+        except (OSError, ValueError):
             return None
 
     def __enter__(self):