Refactor: replace `requests` with `urllib3`

[woodruffw]

urllib3 has everything we need from requests (and is the library beneath it), including connection pooling. It's much more actively maintained and has a more consistent API, so we should switch to it.

[jku]

linking to theupdateframework/python-tuf#2762 FYI (a related PR for python-tuf).

One thing to note that already came up:

• urllib3 defaults to system certificates while requests defaults to certifi

[woodruffw]

• urllib3 defaults to system certificates while requests defaults to certifi

Yep, this is a "plus" IMO: the fact that PyPI is a CA bundle delivery mechanism (for certifi) has always been one of those "don't look but this is very scary" realities about HTTPS in Python 🙂

[jku]

Yeah I definitely agree with your concerns -- what I'm not sure is how to combine the two views:

• scary-http-security-bootstrap-situation you mentioned + some users would definitely like to use system certificates
• python ecosystem seems to like certifi for platform-compat concerns. I am not familiar with the compatibility issue in question but I guess it's windows?

Basically, can we just switch to using system certificates for https without collateral damage? is there a way for downstream (applications/distributors/users) to change that or is that unnecessary (I remember requests had a REQUESTS_CA_BUNDLE hack)?

I see Seth has a potential solution https://github.com/sethmlarson/truststore -- I suppose this means we could just use system certs and point people to truststore (and potentially use truststore in the sigstore-python CLI ourselves)

[woodruffw]

• python ecosystem seems to like certifi for platform-compat concerns. I am not familiar with the compatibility issue in question but I guess it's windows?

Windows may have been a concern, but (IIRC) the original motivation for certifi was that Linux distributions didn't do a consistently good job of updating/maintaining their CA bundles, nor did users do a consistently good job of retrieving distribution-pushed updates. The Mozilla root program was (and is) the best at the time, so certifi just redistributed that.

In practice, I think this problems have been mostly ameliorated: the Web PKI is in much better shape than it was a decade+ ago, and root program/OS providers tend to be much more stringent about security updates.

With urllib3 we could allow an override via urllib3.PoolManager(ca_certs="..."), although IMO we shouldn't expose that as a public interface until a user actually explicitly requests it (and presents a good justification -- 99% of the time they should be configuring their system roots instead). Thoughts on that?

[jku]

I think that sounds very reasonable.

@sethmlarson if you have the time, would you mind commenting on this one?

• are we overlooking any potential problems when switching from certifi to system certs
• should we (as library) mention truststore in docs?
• should we (as CLI app) use truststore explicitly?

[jku]

Documenting the other concern with urllib3 here: it does not handle proxy environment variables, and cannot be configured to do so.

I think this is something we can't just overlook:

• I believe a lot of people are behind dumb corporate proxies where the only way to get software working is proxy configuration in some way
• since sigstore (the library) is not implement "sans I/O" style, we can't really leave this as an application problem unless we expose new API for this -- and that seems very unappealing
• Supporting "https_proxy" and other related environment variables seems like the least unappealing path even if it's not great

I've done the proxy variable support in a python-tuf PR: it's <100 lines of code so not entirely out of question