Requoting does not unquote %27

[albertvillanova]

When requoting a redirection URL, the single quotation mark does not get unquoted: %27 => %27:

In [1]: from requests.utils import requote_uri

In [2]: url = "https://netloc/path?param=param%27%27value"

In [3]: print(url)
Out[3]: 'https://netloc/path?param=param%27%27value'

In [4]: print(requote_uri(url))
Out[4]: 'https://netloc/path?param=param%27%27value'

However, the aiohttp library uses yarl.ULR and this does unquote %27: %27 => '

In [5]: from yarl import URL

In [6]: print(url)
Out[6]: 'https://netloc/path?param=param%27%27value'

In [7]: print(str(URL(url)))
Out[7]: "https://netloc/path?param=param''value"

Expected Result

I think both libraries should have the same behavior.

Actual Result

The requests library does not unquote %27, whereas the aiohttp library (that uses yarl) does unquote %27.

Reproduction Steps

from requests.utils import requote_uri
from yarl import URL

url = "https://netloc/path?param=param%27%27value"

print(url)

print(requote_uri(url))

print(URL(url))

System Information

$ python -m requests.help

{
  "chardet": {
    "version": null
  },
  "charset_normalizer": {
    "version": "2.0.8"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.10"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.9.7"
  },
  "platform": {
    "release": "5.14.0-1055-oem",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.26.0"
  },
  "system_ssl": {
    "version": "101010cf"
  },
  "urllib3": {
    "version": "1.26.7"
  },
  "using_charset_normalizer": true,
  "using_pyopenssl": false
}

[sigmavirus24]

Arguably the RFCs are in Requests' favor. It's safer to leave those quotes because they should be quoted by RFC 3986 and even the What WG URL spec suggests they should be. Not all servers are going to reject aiohttp's behavior but some will. It's safer to be conservative in what is sent (quote reserved characters, etc.) then it is not to.

[albertvillanova]

Thank you for your explanation @sigmavirus24.

Indeed, we found this discrepancy between requests and aiohttp when we got a 403 error from our server at Hugging Face in streaming mode (that uses aiohttp) and no error in non-streaming mode (that uses requests)

See:

• Disable aiohttp requoting of redirection URL huggingface/datasets#5459