Speed up ci (#383)

* Super broken commit of my throwing my brain at refactoring the Docker Compose config logic * responsive mostly WIP * began migrating to muiv5 * updated to MUI 5, finished first responsive pass * cleanup * fix integration tests * quieter cypress * add seo stuff * Refactor CLI to be more modular * got rid of unnecessary cypress step * extend cypress first page load timeout * try to catch cy bug * mode in header' * Switch to Yarn 2 and fix CLI Env bugs * Update Yarn in gitignore * Don't ignore nested yarn.lock files * fix bugs that were related to the demo app running after migration to using yarn 3 * Fix bugs introduced, also compile static assets in local mode now * Remove Lerna from CI scripts * Attempt to fix CI again * Add missing build scripts * Add more type imports everywhere * Cleanup versions and update Cypress * Docs are still broken, but it is at least closer now! * more permissive visit call * fix local scripts * Fix ESLint configuration in IntelliJ * change urls to make sure that demo and tests are working. use xfvb-run to bring up integration tests. * fix docs by preventing hoisting of docs deps * make lockfile deletion in CI also regenerate * yarn install during no lockfile test * trick yarn install about ci mode * remove extra yarn install * update lockfile * lockfile hacks * try CI=false * set ci blank * turn off fancy lockfile stuff * remove go tests * ignore linting * cypress try global listener * restore tmuxp and sign up for cypress dashboard * reintroduced initial visit call to try to make cy tests work in CI * more cypress headers * get simple tokenizer working and restore go tests * create a new workflow for checking evil deps * simplify deps job * simplify the RIGHT job * simplify CI env var * make sure buildkit is on inside cli container * rewrite analytics inserter to support demo app * make lint work again, update a ton of lint deps * attempt to prevent lerna-bootstrap double run * added some eslint ignores * got all lints passing, definitely took on some tech-debt here * improve bucket script * fix lint * fix lint bug with react-scripts and hopefully fix aws_resources path bug * Another pass of the CI * Move Golang tests into their own job * Fix lint errors * free changes made * fix cypress verbose logging env var * Split up the CI job into re-usable actions * Rename jobs * Fix what is likely a bad path * Fix what is likely a bad path * Remove action to test where bad syntax is * Test if variable syntax is the problem * Fix variable syntax * Test uses syntax with version * Another round of checks * Another pass of tweaking * Use underscores instead of dashes * Fix invalid YAML * Move actions around * No nested actions * Make input required * Minimum test * rename job * Rename job * id -> name * id -> name * Check variable syntax * Double equals perhaps? * Try with actual GitHub action script now * Pass event through * Remove cancel step * Move steps back. maybe * Move steps back. maybe * Only call second script now * Inline second workflow * Pass through a shell * Inline at the end * Try inline again * Bring back script * Another test * Just test merge branch check * Test with an echo * Test with a basic echo * Remove conditional * Smush everything together * Remove echo line * Remove conditional * made cypress dashboard more smarter * Remove all conditional logic by splitting up action * Remove Yarn cache * Cleanup * update docusaurus to fix blogs and fix the awful looking admonition from beta 12 * make demo backend use pm2 * update yarn.lock * Regenerate lockfile, fix CLI arguments for hosted demo * Disable Nginx volume * Fix a yarn error * Fix yarn.lock * Remove extraneous workdir statement * Uncomment nginx * Remove second build step from demo back end build * Revert build change * no fork pm2 * switch to pm2-runtime * add mui types * verbose start in CI * yarn install * made bucket script wait for file and brought back the use of a precache container * switch cli workdir to repo * try again in CI * increase max mocked s3 body size in nginx for live demo * fix nginx args * fix admonitions * yarn stuff * include more install steps in the precache * stop demo back end from starting prematurely * delete &&, it wasnt needed * rename integration test workflow to just test Co-authored-by: Free Wortley <[email protected]> Co-authored-by: breadchris <[email protected]> Former-commit-id: 98ad19a Former-commit-id: 4f64e33b0949d9bd3ee8cb91f78d3d77e8f17c69
lunasec-io · Dec 22, 2021 · c27bdbf · c27bdbf
1 parent a45854a
commit c27bdbf
Show file tree

Hide file tree

Showing 136 changed files with 43,041 additions and 30,694 deletions.
diff --git a/.dockerignore b/.dockerignore
@@ -2,8 +2,6 @@
 .idea/
 .github/
 
-docs/
-
 docker-compose.yaml
 docker-compose.*.yaml
 
@@ -12,4 +10,4 @@ outputs/
 **/build/
 **/node_modules/
 node_modules/
-.npmrc
+.npmrc
diff --git a/.editorconfig b/.editorconfig
@@ -0,0 +1,10 @@
+root = true
+
+[*]
+end_of_line = lf
+insert_final_newline = true
+
+[*.{js,json,yml}]
+charset = utf-8
+indent_style = space
+indent_size = 2
diff --git a/.eslintignore b/.eslintignore
@@ -4,4 +4,9 @@
 **/build
 **/generated
 js/demo-apps/packages/react-front-end/cypress/integration/secure_components_spec.ts
-**/ContactForm.jsx
+.pnp.cjs
+*.mjs
+deploy-apigateway-to-firehose.ts
+**/*.vue
+**/metrics-server-backend/*
+**/ContactForm.jsx
diff --git a/.eslintrc.js b/.eslintrc.js
@@ -21,10 +21,10 @@ module.exports = {
     node: true
   },
   extends: [
-    "plugin:vue/vue3-essential",
-    "@vue/typescript/recommended",
-    "@vue/prettier",
-    "@vue/prettier/@typescript-eslint",
+    // "plugin:vue/vue3-essential",
+    // "@vue/typescript/recommended",
+    // "@vue/prettier",
+    // "@vue/prettier/@typescript-eslint",
     'eslint:recommended',
     'plugin:@typescript-eslint/recommended',
     'plugin:@typescript-eslint/recommended-requiring-type-checking',
@@ -54,10 +54,12 @@ module.exports = {
   },
   plugins: [
     'react',
-    '@typescript-eslint'
+    '@typescript-eslint',
+    'jest'
   ],
   rules: {
-    'no-console': process.env.NODE_ENV === 'production' ? 'error' : 'off',
+    "@typescript-eslint/no-unsafe-argument": 1, // TODO: Re-enable this rule and fix all errors
+    'no-console': process.env.NODE_ENV === 'production' ? 'error' : 'off', // These never error, currently
     'no-debugger': process.env.NODE_ENV === 'production' ? 'error' : 'off',
     eqeqeq: 'error',
     quotes: ['error', 'single', { allowTemplateLiterals: true, avoidEscape: true }],

diff --git a/.github/ISSUE_TEMPLATE/bug_report.md b/.github/ISSUE_TEMPLATE/bug_report.md
@@ -1,3 +1,16 @@
+<!--
+  ~ Copyright by LunaSec (owned by Refinery Labs, Inc)
+  ~
+  ~ Licensed under the Creative Commons Attribution-ShareAlike 4.0 International
+  ~ (the "License"); you may not use this file except in compliance with the
+  ~ License. You may obtain a copy of the License at
+  ~
+  ~ https://creativecommons.org/licenses/by-sa/4.0/legalcode
+  ~
+  ~ See the License for the specific language governing permissions and
+  ~ limitations under the License.
+  ~
+-->
 ---
 name: Bug report
 about: Create a report to help us improve

diff --git a/.github/workflows/check-deps.yaml b/.github/workflows/check-deps.yaml
@@ -0,0 +1,23 @@
+name: Check Dependencies
+
+# This workflow checks to make sure that all dependency binaries are valid, to prevent a sneaky commit doing something malicious
+on:
+  pull_request:
+    branches: ['**', '**']
+  push:
+    branches: [master]
+
+jobs:
+  check-deps:
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v2
+    - uses: actions/setup-node@v2
+      with:
+        node-version: '16'
+
+    - name: dependencies
+      env:
+        IS_LUNASEC_CI: true
+      run: yarn install --immutable --immutable-cache --check-cache --inline-builds
diff --git a/.github/workflows/composite/merge-branch/action.yml b/.github/workflows/composite/merge-branch/action.yml
@@ -0,0 +1,22 @@
+name: 'Merge Master Branch'
+description: 'Merges the master Git branch into the current branch.'
+
+runs:
+  using: "composite"
+  steps:
+    # TODO: there may be a more idiomatic way to do this
+    - name: Check if we should skip this build
+      shell: bash
+      id: job-canceller
+      run: echo "::set-output name=cancelled::${{ github.event_name != 'pull_request' }}"
+
+     # Cancel the merge copy of this build(see the matrix above) if we are not in a PR
+    - name: cancelling
+      uses: andymckay/[email protected]
+      if: ${{ steps.job-canceller.outputs.cancelled == 'true' }}
+
+    # Merge with master (or whatever target branch) so we are actually testing what will happen after PR merges, not just this branch
+    - name: Merge target branch
+      shell: bash
+      run: git merge origin/${{ github.event.pull_request.base.ref }}
+      if: ${{ github.event_name == 'pull_request' }}
diff --git a/.github/workflows/composite/setup-docker-ci/action.yml b/.github/workflows/composite/setup-docker-ci/action.yml
@@ -0,0 +1,40 @@
+name: 'Setup Docker CI Environment'
+description: 'Creates the LunaSec CI build environment'
+
+inputs:
+  merge_master:
+    description: 'If true, this merges the master branch during setup.'
+    required: true
+
+runs:
+  using: "composite"
+  steps:
+    - name: Set up Docker BuildKit
+      id: buildx
+      uses: docker/setup-buildx-action@v1
+      with:
+        install: true # sets buildx as the default for docker, which should apply to docker-compose commands
+        driver: docker
+
+    - uses: actions/setup-node@v2
+      with:
+        node-version: '16'
+
+    # specifically tag this bootstrap container to prevent rebuilds
+    - name: Build Lerna Bootstrap container
+      shell: bash
+      env:
+        # avoid warnings like "tput: No value for $TERM and no -T specified"
+        TERM: xterm
+        DOCKER_BUILDKIT: 1
+        COMPOSE_DOCKER_CLI_BUILD: 1
+      run: docker build --progress plain -f ./js/docker/demo.dockerfile -t lerna-bootstrap --target lerna-bootstrap .
+
+    - name: Build CLI Container
+      shell: bash
+      env:
+        # avoid warnings like "tput: No value for $TERM and no -T specified"
+        TERM: xterm
+        DOCKER_BUILDKIT: 1
+        COMPOSE_DOCKER_CLI_BUILD: 1
+      run: docker build --progress plain -f ./js/docker/demo.dockerfile -t repo_lunasec-cli --target lunasec-cli .
diff --git a/.github/workflows/documentation.yaml b/.github/workflows/documentation.yaml
@@ -41,19 +41,13 @@ jobs:
         with:
           node-version: '16'
 
-      # Speed up builds by caching node_modules
-      - uses: actions/cache@v2
-        with:
-          path: |
-            node_modules
-            */*/node_modules
-          key: ${{ runner.os }}-modules-${{ hashFiles('**/yarn.lock') }}
-
       - name: Lerna Bootstrap and Build Deploy Tool
+        env:
+          IS_LUNASEC_CI: true
+          CI: true
         run: |
-          npx lerna bootstrap --ci
+          yarn install --immutable --immutable-cache --inline-builds
           yarn compile:dev:infrastructure
-          npx lerna link
 
       - name: Build Repo
         working-directory: js/sdks

diff --git a/.github/workflows/integration.yaml b/.github/workflows/integration.yaml
@@ -14,7 +14,7 @@
 #
 # This is a basic workflow to help you get started with Actions
 
-name: CI
+name: Tests
 
 # Controls when the workflow will run
 on:
@@ -36,53 +36,27 @@ jobs:
     env:
       # avoid warnings like "tput: No value for $TERM and no -T specified"
       TERM: xterm
-      RUNNING_IN_CI: true
       DOCKER_BUILDKIT: 1
       COMPOSE_DOCKER_CLI_BUILD: 1
     strategy:
       matrix:
         merge: ["merged", "not-merged"]
-        lockfile: ["lockfile","no-lockfile"]
+        lockfile: ["lockfile"] #,"no-lockfile"]
         exclude:
           - merge: "merged"
             lockfile: "no-lockfile"
 
     steps:
-
     - uses: actions/checkout@v2
       with:
         fetch-depth: 0
 
-    #TODO: there may be a more idiomatic way to do this
-    - name: Check if we should skip this build
-      id: job-canceller
-      run: echo "::set-output name=cancelled::${{ matrix.merge == 'merged' && github.event_name != 'pull_request'}}"
-
-    # Cancel the merge copy of this build(see the matrix above) if we are not in a PR
-    - name: cancelling
-      uses: andymckay/[email protected]
-      if: ${{ steps.job-canceller.outputs.cancelled == 'true' }}
-
-      # merge with master(or whatever target branch) so we are actually testing what will happen after PR merges, not just this branch
-    - name: Merge target branch
-      run: git merge origin/${{ github.event.pull_request.base.ref }}
-      if: ${{ matrix.merge == 'merged' && github.event_name == 'pull_request' }}
-
-    - uses: actions/setup-node@v2
-      with:
-        node-version: '14'
-        cache: yarn
-
-    - uses: actions/setup-go@v2
-      with:
-        go-version: '^1.17.1'
+    - name: Setup Branch
+      uses: ./.github/workflows/composite/merge-branch
+      if: ${{ matrix.merge == 'merged' }}
 
-    - name: Set up Docker BuildKit
-      id: buildx
-      uses: docker/setup-buildx-action@v1
-      with:
-        install: true # sets buildx as the default for docker, which should apply to docker-compose commands
-        driver: docker
+    - name: Setup Environment
+      uses: ./.github/workflows/composite/setup-docker-ci
 
     # Enable tmate debugging of manually-triggered workflows if the input option was provided
     - name: Setup tmate session
@@ -92,9 +66,13 @@ jobs:
       if: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.debug_enabled != 'false' }}
       timeout-minutes: 15
 
-    - name: Delete Lockfile
-      if: ${{ matrix.lockfile == 'no-lockfile' }}
-      run: rm yarn.lock
+    - uses: actions/setup-node@v2
+      with:
+        node-version: '16'
+
+#    - name: Delete and Regenerate Lockfile
+#      if: ${{ matrix.lockfile == 'no-lockfile' }}
+#      run: rm yarn.lock && CI="" yarn install #currently broken
 
 #    - uses: actions/setup-go@v2
 #      with:
@@ -116,22 +94,8 @@ jobs:
 #        GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 #      run: npm run license:check
 
-    # TODO: move this into docker, using the same container that the dedicated tokenizer uses
-    - name: Go Tests
-      run: yarn run test:unit:go
-
-    - name: Build CLI Container
-      run: docker build --progress plain -f ./js/docker/demo.dockerfile -t repo_lunasec-cli --target lunasec-cli .
-
-    # Hijack the cli image to quickly run linting and unit testing
-    - name: Lint
-      run: docker run --entrypoint yarn repo_lunasec-cli lint
-
-    - name: Unit Test
-      run: docker run --entrypoint yarn repo_lunasec-cli test:unit:js
-
     - name: Use CLI Container with docker.sock mounted to launch all other containers
-      run: docker run -v /var/run/docker.sock:/var/run/docker.sock -e HOST_MACHINE_PWD=$(pwd) repo_lunasec-cli start --no-sudo --local-build --env=tests
+      run: docker run -v /var/run/docker.sock:/var/run/docker.sock -e HOST_MACHINE_PWD=$(pwd) repo_lunasec-cli start --no-sudo --local-build --env=tests --verbose
 
     - name: docker logs
       if: ${{ always() && steps.job-canceller.outputs.cancelled == 'false' }}
@@ -158,4 +122,38 @@ jobs:
       env:
         SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
         SLACK_TITLE: PR Build ${{ github.head_ref }} ${{ job.status }}
-        SLACK_COLOR: ${{ job.status }}
+        SLACK_COLOR: ${{ job.status }}
+
+  golang-unit-tests:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        merge: [ "merged", "not-merged" ]
+        lockfile: [ "lockfile" ] #,"no-lockfile"]
+        exclude:
+          - merge: "merged"
+            lockfile: "no-lockfile"
+    steps:
+      - uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
+
+      - name: Setup Branch
+        uses: ./.github/workflows/composite/merge-branch
+        if: ${{ matrix.merge == 'merged' }}
+
+      - uses: actions/setup-go@v2
+        with:
+          go-version: '^1.17.1'
+
+      # TODO: move this into docker, using the same container that the dedicated tokenizer uses
+      - name: Go Tests
+        run: yarn run test:unit:go
+
+      - name: Slack Notification
+        uses: rtCamp/action-slack-notify@v2
+        if: ${{ steps.job-canceller.outputs.cancelled == 'false' }}
+        env:
+          SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
+          SLACK_TITLE: PR Build ${{ github.head_ref }} ${{ job.status }}
+          SLACK_COLOR: ${{ job.status }}
diff --git a/.github/workflows/lint-code.yaml b/.github/workflows/lint-code.yaml
@@ -0,0 +1,28 @@
+name: Lint Code plus JS Unit Tests
+
+# This workflow checks to make sure that all dependency binaries are valid, to prevent a sneaky commit doing something malicious
+on:
+  pull_request:
+    branches: ['**', '**']
+  push:
+    branches: [master]
+
+jobs:
+  lint-code:
+    runs-on: ubuntu-latest
+
+    steps:
+    - uses: actions/checkout@v2
+      with:
+        fetch-depth: 0
+
+    - name: Setup Environment
+      uses: ./.github/workflows/composite/setup-docker-ci
+
+    # Use the bootstrap image to quickly run linting and unit testing
+    - name: Lint
+      run: docker run --entrypoint yarn lerna-bootstrap lint
+
+    - name: Unit Test
+      run: docker run --entrypoint yarn lerna-bootstrap test:unit:js
+
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
@@ -51,10 +51,11 @@ jobs:
       - name: install lerna
         run: npm i -g lerna
 
-      - name: lerna bootstrap
+      - name: dependencies
         env:
           IS_LUNASEC_CI: true
-        run: lerna bootstrap --ci -- --frozen-lockfile
+          CI: true
+        run: yarn install --immutable --immutable-cache --check-cache --inline-builds
 
       - name: Configure CI Git User
         run: |
@@ -132,4 +133,4 @@ jobs:
       - name: git push new tags
         run: git push --tags
         env:
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}