Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix Snyk scanning and upload results #343

Merged
merged 3 commits into from
Dec 19, 2024
Merged

Fix Snyk scanning and upload results #343

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
60b788e
Fix Snyk scanning and upload results
julienp Dec 17, 2024
5ad3090
Move script to a separate file and document fail-fast
julienp Dec 17, 2024
0cc8a46
add script …
julienp Dec 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .github/scripts/filter-sarif.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
#!/usr/bin/python
# GitHub Code Scanning does not allow more than 20 runs per file. We filter out
# empty runs to avoid this limit. Note that the file needs to include at least
# one run.

import json

with open("snyk.sarif") as f:
sarif = json.load(f)

# Remove runs with no results
runs = [run for run in sarif["runs"] if len(run["results"]) > 0]

# Keep at least one run
runs = runs if len(runs) > 0 else [sarif["runs"][0]]

sarif["runs"] = runs

with open("out.sarif", "w") as out:
json.dump(sarif, out, indent=2)
77 changes: 68 additions & 9 deletions .github/workflows/snyk-scan.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,14 +17,16 @@ jobs:
name: All SDKs images
runs-on: ubuntu-latest
strategy:
# Try to complete the scan for all images, even if one fails.
fail-fast: false
julienp marked this conversation as resolved.
Show resolved Hide resolved
matrix:
image: ["pulumi", "pulumi-provider-build-environment"]
include:
# For the pulumi image add a the nonroot variant
- suffix: -nonroot
image: pulumi
suffix: ["", "-nonroot"]
steps:
- uses: actions/checkout@master
- name: Free Disk Space (Ubuntu)
uses: jlumbroso/free-disk-space@main
with:
tool-cache: false
- name: Set version
run: |
[ -z "${{ env.DISPATCH_REF }}" ] && echo "PULUMI_VERSION=$(curl https://www.pulumi.com/latest-version)" >> $GITHUB_ENV || echo "PULUMI_VERSION=${{ env.DISPATCH_REF }}" >> $GITHUB_ENV
Expand All @@ -34,13 +36,47 @@ jobs:
env:
SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
with:
image: ${{ env.DOCKER_ORG }}/${{ matrix.image }}:${{ env.PULUMI_VERSION }}${{ matrix.suffix }}
image: ${{ env.DOCKER_ORG }}/pulumi:${{ env.PULUMI_VERSION }}${{ matrix.suffix }}-amd64
args: --severity-threshold=high --file=docker/pulumi/Dockerfile
- name: Filter Sarif File
run: python ./.github/scripts/filter-sarif.py
- name: Upload result to GitHub Code Scanning
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: out.sarif

provider-build-environment:
name: Provider Build Environment
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@master
- name: Free Disk Space (Ubuntu)
uses: jlumbroso/free-disk-space@main
with:
tool-cache: false
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What leads to tons of disk usage here? Is it the outputs of scanning?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The image itself is fairly large, plus it download some snyk image. Not sure what precisely causes us to run out of space, but for kitchen-sink it failed with ENOSPC before :/

- name: Set version
run: |
[ -z "${{ env.DISPATCH_REF }}" ] && echo "PULUMI_VERSION=$(curl https://www.pulumi.com/latest-version)" >> $GITHUB_ENV || echo "PULUMI_VERSION=${{ env.DISPATCH_REF }}" >> $GITHUB_ENV
- name: Snyk scan
continue-on-error: true
uses: snyk/actions/docker@master
env:
SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
with:
image: ${{ env.DOCKER_ORG }}/pulumi-provider-build-environment:${{ env.PULUMI_VERSION }}-amd64
args: --severity-threshold=high --file=docker/pulumi/Dockerfile
- name: Filter Sarif File
run: python ./.github/scripts/filter-sarif.py
- name: Upload result to GitHub Code Scanning
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: out.sarif

base:
name: Base image
runs-on: ubuntu-latest
strategy:
# Try to complete the scan for all images, even if one fails.
fail-fast: false
matrix:
os: ["debian", "ubi"]
Expand All @@ -57,7 +93,14 @@ jobs:
SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
with:
image: ${{ env.DOCKER_ORG }}/pulumi-base:${{ env.PULUMI_VERSION }}-${{ matrix.os }}-${{ matrix.arch }}
args: --severity-threshold=high --file=docker/base/Dockerfile.${{ matrix.os }}
args: --severity-threshold=high --file=docker/base/Dockerfile.${{ matrix.os }} --platform=linux/${{ matrix.arch }}
- name: Filter Sarif File
run: python ./.github/scripts/filter-sarif.py
- name: Upload result to GitHub Code Scanning
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: out.sarif

define-debian-matrix:
runs-on: ubuntu-latest
outputs:
Expand All @@ -68,11 +111,13 @@ jobs:
id: define-matrix
run: |
echo matrix=$(python ./.github/scripts/matrix/gen-matrix.py) >> "$GITHUB_OUTPUT"

debian-sdk:
name: Debian SDK images
runs-on: ubuntu-latest
needs: define-debian-matrix
strategy:
# Try to complete the scan for all images, even if one fails.
fail-fast: false
matrix: ${{ fromJSON(needs.define-debian-matrix.outputs.matrix) }}
steps:
Expand All @@ -90,11 +135,19 @@ jobs:
SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
with:
image: ${{ env.IMAGE_NAME }}
args: --severity-threshold=high --file=docker/${{ matrix.sdk }}/Dockerfile.debian
args: --severity-threshold=high --file=docker/${{ matrix.sdk }}/Dockerfile.debian --platform=linux/${{ matrix.arch }}
- name: Filter Sarif File
run: python ./.github/scripts/filter-sarif.py
- name: Upload result to GitHub Code Scanning
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: out.sarif

ubi-sdk:
name: UBI SDK images
runs-on: ubuntu-latest
strategy:
# Try to complete the scan for all images, even if one fails.
fail-fast: false
matrix:
sdk: ["nodejs", "python", "dotnet", "go"]
Expand All @@ -110,4 +163,10 @@ jobs:
SNYK_TOKEN: ${{ secrets.SNYK_TOKEN }}
with:
image: ${{ env.DOCKER_ORG }}/pulumi-${{ matrix.sdk }}:${{ env.PULUMI_VERSION }}-ubi
args: --severity-threshold=high --file=docker/${{ matrix.sdk }}/Dockerfile.ubi
args: --severity-threshold=high --file=docker/${{ matrix.sdk }}/Dockerfile.ubi --platform=linux/amd64
- name: Filter Sarif File
run: python ./.github/scripts/filter-sarif.py
- name: Upload result to GitHub Code Scanning
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: out.sarif
Loading