too many requests error again

[DChevrier1]

We upgraded to 0.28 when it came out and these errors went away, but they have come back again. What can we do to help fix this as it is part of our CI/CD pipeline? Thanks

ERROR [vulndb] Failed to download artifact repo="ghcr.io/aquasecurity/trivy-db:2" err="oci download error: failed to fetch the layer: GET https://ghcr.io/v2/aquasecurity/trivy-db/blobs/sha256:c3afeb28c808216cc2fa69d1e682164efd3c9967451a061778329ce94ce1a069: TOOMANYREQUESTS: retry-after: 217.944µs, allowed: 44000/minute"
1902024-11-05T13:55:48Z FATAL Fatal error init error: DB error: failed to download vulnerability DB: OCI artifact error: failed to download vulnerability DB: failed to download artifact from any source

[ACipkowski1]

Thank you for looking into this

[JBrown413]

I'd love to see a fix!!

[CDixson1]

Received same error, TOOMANYREQUESTS. A fix would be greatly appreciated. Thanks.

[RBlanc1]

A fix for this would be helpful in our pipeline process - thx

[tliebert1]

Have run into this problem also.

[ogruene]

Hi @DChevrier1,

as far as I see, you can simply add the following env to your aquasecurity/trivy-action step:

env:
    TRIVY_DB_REPOSITORY: "public.ecr.aws/aquasecurity/trivy-db:2"

This should work - at least when using current aquasecurity/trivy-action@master. Not sure if it also works with the last release.

_{Oliver Grüneberg <[email protected]>, Mercedes-Benz Tech Innovation GmbH
Provider Information}

[arareko]

Seems related to #107

[ogruene]

As far as I see, it's a DB download error rather related #389

[arareko]

As far as I see, it's a DB download error rather related #389

@ogruene Yes, both are related.

[RichardoC]

Hi @DChevrier1,

as far as I see, you can simply add the following env to your aquasecurity/trivy-action step:

env:
    TRIVY_DB_REPOSITORY: "public.ecr.aws/aquasecurity/trivy-db:2"

This should work - at least when using current aquasecurity/trivy-action@master. Not sure if it also works with the last release.

Oliver Grüneberg <[email protected]>, Mercedes-Benz Tech Innovation GmbH Provider Information

Mind providing a source for that repo? https://pkg.go.dev/github.com/aquasecurity/trivy-db#readme-download-the-vulnerability-database only refers to the github container registry

[brianjmurrell]

So, it seems that 0.26 was supposed to have added caching and the README says specifically: _ The action has a built-in functionality for caching and restoring [the vulnerability DB]_.

While I am seeing some caching happening in our runs:

Run aquasecurity/trivy-action@915b19bbe73b92a6cf82a1bc12b087c9a19a5fe2
Run aquasecurity/[email protected]
Run echo "dir=$HOME/.local/bin/trivy-bin" >> $GITHUB_OUTPUT
Run actions/cache@v4
Cache Size: ~36 MB (37525313 B)
/usr/bin/tar -xf /home/runner/work/_temp/bbc2d040-9129-4c68-941e-38abe1985a7d/cache.tzst -P -C /home/runner/work/[redacted]/[redacted] --use-compress-program unzstd
Cache restored successfully
Cache restored from key: trivy-binary-v0.56.1-Linux-X64
Run echo /home/runner/.local/bin/trivy-bin >> $GITHUB_PATH
Run echo "date=$(date +'%Y-%m-%d')" >> $GITHUB_OUTPUT
Run actions/cache@v4
Cache Size: ~38 MB (39917276 B)
/usr/bin/tar -xf /home/runner/work/_temp/aa0062e0-a74f-49e4-8884-48eb8665af21/cache.tzst -P -C /home/runner/work/[redacted]/[redacted] --use-compress-program unzstd
Received 39917276 of 39917276 (100.0%), 38.0 MBs/sec
Cache restored successfully
Cache restored from key: cache-trivy-2024-11-13

when it comes to getting the vulnerabilities DB, the cache does not seem to be getting used:

2024-11-18T14:51:53Z	INFO	[vulndb] Need to update DB
2024-11-18T14:51:53Z	INFO	[vulndb] Downloading vulnerability DB...
2024-11-18T14:51:53Z	INFO	[vulndb] Downloading artifact...	repo="ghcr.io/aquasecurity/trivy-db:2"
2024-11-18T14:51:53Z	ERROR	[vulndb] Failed to download artifact	repo="ghcr.io/aquasecurity/trivy-db:2" err="OCI repository error: 1 error occurred:\n\t* GET https://ghcr.io/v2/aquasecurity/trivy-db/manifests/2: TOOMANYREQUESTS: retry-after: 95.725µs, allowed: 44000/minute\n\n"
2024-11-18T14:51:53Z	FATAL	Fatal error

So why is caching happening only partially? Is there a bug related to vulnerability DB caching?

[RichardoC]

So, it seems that 0.26 was supposed to have added caching and the README says specifically: _ The action has a built-in functionality for caching and restoring [the vulnerability DB]_.

While I am seeing some caching happening in our runs:

Run aquasecurity/trivy-action@915b19bbe73b92a6cf82a1bc12b087c9a19a5fe2
Run aquasecurity/[email protected]
Run echo "dir=$HOME/.local/bin/trivy-bin" >> $GITHUB_OUTPUT
Run actions/cache@v4
Cache Size: ~36 MB (37525313 B)
/usr/bin/tar -xf /home/runner/work/_temp/bbc2d040-9129-4c68-941e-38abe1985a7d/cache.tzst -P -C /home/runner/work/[redacted]/[redacted] --use-compress-program unzstd
Cache restored successfully
Cache restored from key: trivy-binary-v0.56.1-Linux-X64
Run echo /home/runner/.local/bin/trivy-bin >> $GITHUB_PATH
Run echo "date=$(date +'%Y-%m-%d')" >> $GITHUB_OUTPUT
Run actions/cache@v4
Cache Size: ~38 MB (39917276 B)
/usr/bin/tar -xf /home/runner/work/_temp/aa0062e0-a74f-49e4-8884-48eb8665af21/cache.tzst -P -C /home/runner/work/[redacted]/[redacted] --use-compress-program unzstd
Received 39917276 of 39917276 (100.0%), 38.0 MBs/sec
Cache restored successfully
Cache restored from key: cache-trivy-2024-11-13

when it comes to getting the vulnerabilities DB, the cache does not seem to be getting used:

2024-11-18T14:51:53Z	INFO	[vulndb] Need to update DB
2024-11-18T14:51:53Z	INFO	[vulndb] Downloading vulnerability DB...
2024-11-18T14:51:53Z	INFO	[vulndb] Downloading artifact...	repo="ghcr.io/aquasecurity/trivy-db:2"
2024-11-18T14:51:53Z	ERROR	[vulndb] Failed to download artifact	repo="ghcr.io/aquasecurity/trivy-db:2" err="OCI repository error: 1 error occurred:\n\t* GET https://ghcr.io/v2/aquasecurity/trivy-db/manifests/2: TOOMANYREQUESTS: retry-after: 95.725µs, allowed: 44000/minute\n\n"
2024-11-18T14:51:53Z	FATAL	Fatal error

So why is caching happening only partially? Is there a bug related to vulnerability DB caching?

Afaik these containers get released every 6 hours, is it possible that those runs were 6 hours apart?

[brianjmurrell]

Afaik these containers get released every 6 hours, is it possible that those runs were 6 hours apart?

I don't think so. It even happens when I "rerun failed jobs" in GitHub Actions repeatedly in quick succession when the jobs fail (i.e. minutes apart).