402 Payment Required from cloudsmith

[roman-belkov]

Previously discussed here: #114

E: Failed to fetch https://dl.cloudsmith.io/public/caddy/stable/deb/debian/dists/any-version/InRelease 402 Payment Required

This blocks updates & upgrades on machines.

It appears that all bandwidth is gone in only 5 days.

[mstaack]

yeah same for me.....

Err:7 https://dl.cloudsmith.io/public/caddy/stable/deb/debian any-version InRelease
  402  Payment Required [IP: 2600:9000:275b:ea00:* 443]

[mohammed90]

It's strange that we've hit our 2TB bandwidth quota in merely 3 days. Our apt repo appears to be abused as part of CI builds. If any of you are using the apt package in CI, please change your approach.

Until then, we're working on a solution.

N.B. Please refrain from making more comments of "me too" unless you're adding more details or context.

[ob6160]

If it helps anybody, we had success using the following to install Caddy in our CI build:

# Define Caddy version and URL for the binary and checksum
CADDY_VERSION="2.7.6"
CADDY_URL="https://github.com/caddyserver/caddy/releases/download/v${CADDY_VERSION}/caddy_${CADDY_VERSION}_linux_amd64.tar.gz"
CHECKSUM_URL="https://github.com/caddyserver/caddy/releases/download/v${CADDY_VERSION}/caddy_${CADDY_VERSION}_checksums.txt"

# Download Caddy and checksum files
curl -sOL ${CADDY_URL}
curl -sOL ${CHECKSUM_URL}

# Verify checksum
grep " caddy_${CADDY_VERSION}_linux_amd64.tar.gz$" caddy_${CADDY_VERSION}_checksums.txt | shasum -c

# Extract Caddy binary
tar xzf caddy_${CADDY_VERSION}_linux_amd64.tar.gz caddy

# Move Caddy to the PATH
sudo mv caddy /usr/local/bin/

# Verify installation
caddy version

There might be room for improvement with the script of course, but it got us up and running again in our pipeline (sorry for using the apt repo originally!!)

[andris9]

I gave up using Cloudsmith and started to install from releases directly (Ubuntu 22.04)

CADDY_VERSION="2.7.6"
CADDY_ARCH="amd64"

wget -q "https://github.com/caddyserver/caddy/releases/download/v${CADDY_VERSION}/caddy_${CADDY_VERSION}_linux_${CADDY_ARCH}.deb"
apt install "./caddy_${CADDY_VERSION}_linux_${CADDY_ARCH}.deb"

[francislavoie]

This is due to abusive users hammering Cloudsmith. There's absolutely no way we received 2.5TB of legitimate traffic within 3 days. For the record, here's our usage tracking for the past year:

/cc @lskillen @BartoszBlizniak from Cloudsmith

[mholt]

We're also going to ask that anyone using the Cloudsmith repos in CI to use xcaddy or to download from GitHub releases instead.

We're not sure but it seems likely that an extremely busy CI or testing environment is relentlessly downloading Caddy from Cloudsmith.

[bakerds]

It's strange that we've hit our 2TB bandwidth quota in merely 3 days

There are many public mirrors that would host your repository for free, with no bandwidth limits. Have you considered a distributed approach?

[mohammed90]

It's strange that we've hit our 2TB bandwidth quota in merely 3 days

There are many public mirrors that would host your repository for free, with no bandwidth limits. Have you considered a distributed approach?

Can you elaborate?

[bakerds]

It's strange that we've hit our 2TB bandwidth quota in merely 3 days

There are many public mirrors that would host your repository for free, with no bandwidth limits. Have you considered a distributed approach?

Can you elaborate?

For example, the many organizations listed here or here provide their services for free, and would very likely be happy to host your repo as well. Then you just need to refer users to a nearby healthy mirror.

[mcint]

It might be still more helpful to talk to CDNs, who have expertise and host projects like these anyway. Importantly they also have expertise in dealing with abuse, and still a desire for good availability and speed, as developer targeted mindshare.

Fastly definitely comes to mind, runs mirrors, Cloudflare might.

This seems like a separate program, https://www.fastly.com/fast-forward.

[compuguy]

We're also going to ask that anyone using the Cloudsmith repos in CI to use xcaddy instead.

That's not a 1:1 replacement for a Debian/Ubuntu APT repository. It looks like you're using Fedora's COPR for the RHEL/Fedora. Maybe someone should move Debian/deb repo to an Ubuntu PPA @mholt?
https://caddyserver.com/docs/install#fedora-redhat-centos
https://launchpad.net/ubuntu/+ppas

This is due to abusive users hammering Cloudsmith. There's absolutely no way we received 2.5TB of legitimate traffic within 3 days. For the record, here's our usage tracking for the past year:
Link to image

That's a heck of a lot of caddyserver downloads!?!?

[francislavoie]

@compuguy the best solution we can offer for now is that you download the .deb from our Github releases (which is what we upload to Cloudsmith anyway).

We're still waiting for communication from Cloudsmith about this.

[andyshinn]

Talk to Fastly. They can front the Cloudsmith repo as a CDN. It was something I worked with them on for the Alpine Linux package CDN.

[mholt]

(Following this discussion closely but am traveling through the end of the week and will get to this when I have a chance)

[BartoszBlizniak]

Hey everyone 👋

@francislavoie - I've increased the limit once more to get everyone unblocked. I'm also going to take some time to look at our logs to determine where the majority of requests are coming from.

[BartoszBlizniak]

Hey @francislavoie - I will reach out via email so we can continue the conversation via our support portal. For the wider audience, we have currently increased the bandwidth to 5TB while we address the bandwidth usage.

[mholt]

Hey @BartoszBlizniak , thank you, although I have been emailing Glenn and asked that the limit not be increased any further. We have already utilized much of your generosity and do not want to take advantage of that while we look into mirrors and such.

[mholt]

Cloudsmith has notified me that they have implemented measures, so we appreciate their help and cooperation 😃