Finalize design for egress metering and billing

[reidlw]

What

We need to be billing people for egress, which is currently not happening, even though egress costs are called out in the public pricing info.

This ticket is to finalize a design for how we can do this, and roadmap units of work required to get there

Why

Egress is a big source of cost for us and it's critical that we start recouping that cost from customers.

Also:

• doing the work to enable this also likely is useful in being able to block usage on delinquent accounts
• doing the work to enable this also helps address the problems we're currently solving through implementing manual content blocks through cloudflare dashboard.

Design Strategy

Early design strategy (captured here -- https://www.notion.so/w3sat/Egress-accounting-designs-9d26747203bd4adcbe494bf55c4a5748?pvs=4
) has landed us on the following approach:

• Introduce heavy rate limiting on our current public URLs, which are extremely hard to accurately bill egress for. Includes Bitswap/hoverboard as well
• Introduce a new signed URL schema, where the account to be billed is contained in the URL as well the CID requested. These URLs would not be rate limited, but we would closely track and bill for egress

A URL change for retrieval is quite significant and requires planning before implementation and roll out.

Recommended approach:

• Arrive at a final URL format. Ideally, we should avoid changing our URLs multiple times.

  • Worth considering whether UCAN's should be a part of it
  • @Gozala has some thoughts about URLs and read permissions that we should probably consider.
  • Worth considering how these URLs might work under various future payment schemes, like retriever pays, or content provider pays for n retrievals
  • We don't have to figure it out all now but we should enable some way for us to do extensibility

• Design a simple mechanism for tracking egress based on the account in the URL. Develop an implementation plan. We will probably need to redesign payment mechanisms in the future, so this doesn't have to be a complete rewrite of our existing accounting mechanisms.

• Develop a rollout plan -- how would we ship this? How would and when would we announce the change? When would we roll out rate limiting on existing URLs

• Develop a migration plan for generating new URLs for old content.

[Gozala]

I think this is more complicated than it may appear. I have provided some feedback in the design doc and I will call out specific unknowns here

1. As far as I know we are utilizing cloudflare cache reserve and I suspect it would be really difficult for us to rate-limit or even count reads from cache. I think we need to

   1. Investigate this before we can device a plan for rate limiting or read metering
   2. Assesses how plan would look like without CF cache reserve, mostly so that we do not tie ourselves to a specific feature of the platform (like we did with S3 events)

2. Gating reads will have performance implications. We have designed system to remove as much as possible out of the way from read pipeline so it can handle significant throughput and remain cost efficient. Likely we'll need some way to

   1. Asses rate limits on every read
   2. Charge someone for the read
   3. Deny service if we are unable to bill someone

   Right now we have nothing in place to do any of the above. Also doing dynamo lookup for every read is going to increase operational costs. All in all deciding whether we serve and async decide who to charge vs we decide sync before we serve is a serious decision and probably requires careful consideration and some cost analysis.

3. I think we need a decision and some input from the market research on what is reasonable DX here. I've been advocating for use of same gateway URI for reads and supply billing info through an auth headers. It is unclear if that is reasonable DX. My guess is it is given that for pre-signed S3 / R3 PUT urls you do have to pass some HTTP headers. That said it may not be so for GET URLs e.g. if someone wants to embed image on the web-page they can't pass headers. Passing auth data via query params might be another option, but again I think we need someone perhaps @prodalex to do some market research to make an informed decision here.

   • There are other authorization schemes available e.g. HTTP basic auth allowing us to embed auth in URLs. I think it would make sense to have a table of all possible options and evaluate pros and cons

[travis]

PR with RFC for this design here: storacha/RFC#28