Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: swrlab/ard-eventhub
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v1.1.1
Choose a base ref
...
head repository: swrlab/ard-eventhub
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: main
Choose a head ref
Loading
Showing with 7,233 additions and 8,748 deletions.
  1. +42 −0 .editorconfig
  2. +25 −5 .env.example
  3. +4 −16 .eslintrc.json
  4. +12 −11 .github/dependabot.yaml
  5. +57 −62 .github/workflows/ingest-pull.yml
  6. +140 −164 .github/workflows/ingest-push.yml
  7. +3 −5 .gitignore
  8. +109 −0 .license-compliancerc.js
  9. +35 −0 .markdownlint.yaml
  10. +3 −0 .vscode/extensions.json
  11. +19 −0 .vscode/settings.json
  12. +255 −36 CHANGELOG.md
  13. +26 −0 Dockerfile
  14. +0 −22 Dockerfile.ingest
  15. +43 −22 README.md
  16. +5 −5 test/README.md → TESTING.md
  17. +38 −0 biome.json
  18. +2,584 −0 bun.lock
  19. +8 −0 bunfig.toml
  20. 0 config/{coreIdPrefixes.json → core-id-prefixes.json}
  21. +6 −0 config/dts-keys.js
  22. +33 −0 config/event-duplication.json
  23. +43 −38 config/index.js
  24. +2 −2 config/{swaggerUI.js → swagger-ui.js}
  25. +7 −14 docs/AUTHENTICATION.md
  26. +8 −4 docs/EVENTS.md
  27. +17 −0 docs/EXTERNAL_IDS.md
  28. +2 −7 docs/NAMING.md
  29. +2 −2 docs/OPENAPI.md
  30. +40 −0 docs/PLUGINS.md
  31. +18 −28 docs/QUICKSTART.md
  32. +8 −14 docs/SECRETS.md
  33. +1 −6 docs/STAGES.md
  34. +34 −18 docs/TYPES.md
  35. +6 −11 docs/USERS.md
  36. +3 −1 docs/_SIDEBAR.md
  37. +2 −2 index.html
  38. 0 keys/.gitkeep
  39. +0 −102 license.sh
  40. +1,588 −1,359 openapi.json
  41. +235 −59 openapi.yaml
  42. +49 −44 package.json
  43. +2 −2 src/coreId/index.js
  44. +537 −459 src/data/coreApi.json
  45. +32 −25 src/ingest/README.md
  46. +20 −19 src/ingest/auth/login/post.js
  47. +22 −7 src/ingest/auth/middleware/verify.js
  48. +20 −19 src/ingest/auth/refresh/post.js
  49. +23 −23 src/ingest/auth/reset/post.js
  50. +140 −212 src/ingest/events/post.js
  51. +20 −14 src/ingest/index.js
  52. +247 −60 {test → src/ingest}/ingest.test.js
  53. +5 −3 src/ingest/pubsub/index.js
  54. +3 −2 src/ingest/pubsub/verify.js
  55. +25 −20 src/ingest/router.js
  56. +16 −17 src/ingest/subscriptions/delete.js
  57. +6 −4 src/ingest/subscriptions/get.js
  58. +5 −5 src/ingest/subscriptions/list.js
  59. +6 −6 src/ingest/subscriptions/post.js
  60. +4 −2 src/ingest/topics/list.js
  61. +0 −16 src/utils/core/createHashedId.js
  62. +1 −1 src/utils/core/getPublisher.js
  63. +1 −3 src/utils/core/index.js
  64. +1 −1 src/utils/datastore/_client.js
  65. +1 −1 src/utils/datastore/delete.js
  66. +8 −4 src/utils/datastore/index.js
  67. +2 −2 src/utils/datastore/load.js
  68. +2 −2 src/utils/datastore/save.js
  69. +77 −0 src/utils/events/createNewTopic.js
  70. +14 −0 src/utils/events/index.js
  71. +97 −0 src/utils/events/processServices.js
  72. +10 −5 src/utils/firebase/index.js
  73. +18 −22 src/utils/firebase/refreshToken.js
  74. +18 −24 src/utils/firebase/sendPasswordResetEmail.js
  75. +18 −22 src/utils/firebase/signInWithEmailAndPassword.js
  76. +1 −1 src/utils/firebase/verifyToken.js
  77. +27 −16 src/utils/logger/index.js
  78. +132 −221 src/utils/plugins/dts/event.js
  79. +1 −1 src/utils/plugins/dts/index.js
  80. +1 −1 src/utils/plugins/index.js
  81. +1 −1 src/utils/pubsub/_client.js
  82. +1 −1 src/utils/pubsub/_publisherClient.js
  83. +1 −1 src/utils/pubsub/_subscriberClient.js
  84. +3 −3 src/utils/pubsub/buildId.js
  85. +1 −1 src/utils/pubsub/convertId.js
  86. +3 −3 src/utils/pubsub/createSubscription.js
  87. +3 −3 src/utils/pubsub/createTopic.js
  88. +1 −1 src/utils/pubsub/deleteSubscription.js
  89. +1 −1 src/utils/pubsub/getSubscription.js
  90. +1 −1 src/utils/pubsub/getSubscriptions.js
  91. +1 −1 src/utils/pubsub/getTopic.js
  92. +1 −1 src/utils/pubsub/getTopics.js
  93. +1 −1 src/utils/pubsub/index.js
  94. +2 −2 src/utils/pubsub/mapSubscription.js
  95. +4 −13 src/utils/pubsub/publishMessage.js
  96. +2 −2 src/utils/response/badRequest.js
  97. +37 −0 src/utils/response/errors/expiredStartTime.js
  98. +15 −0 src/utils/response/errors/index.js
  99. +38 −0 src/utils/response/errors/mismatchingEventName.js
  100. +12 −5 src/utils/response/index.js
  101. +3 −3 src/utils/response/internalServerError.js
  102. +3 −3 src/utils/response/notFound.js
  103. +3 −3 src/utils/response/ok.js
  104. +1 −1 src/utils/secrets/_client.js
  105. +2 −2 src/utils/secrets/get.js
  106. +1 −1 src/utils/secrets/index.js
  107. +1 −2 src/utils/tracer/index.js
  108. +15 −0 src/utils/undici/index.js
  109. +0 −10 src/utils/user/index.js
  110. +0 −22 test/example.test.js
  111. +0 −5,359 yarn.lock
42 changes: 42 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# docs: https://editorconfig.org
# please make sure to keep this file in sync between repositories
# to main file should be edited in `swr-prettier-config` and copied where needed

root = true

[*]
charset = utf-8
end_of_line = lf
indent_style = tab
indent_size = 8
print_width = 120
trim_trailing_whitespace = true
insert_final_newline = true

[*.md]
trim_trailing_whitespace = false
indent_style = space
indent_size = 2

[*.{yml,yaml}]
indent_style = space
indent_size = 2

# LiquidSoap config files
[*.liq]
indent_style = space
indent_size = 2

# Terraform config files
[*.tf]
indent_style = space
indent_size = 2

# Linting toml config files
[*.toml]
indent_style = space
indent_size = 2

# Auto-generated CSV Files
[*.csv]
insert_final_newline = false
30 changes: 25 additions & 5 deletions .env.example
Original file line number Diff line number Diff line change
@@ -1,11 +1,31 @@
STAGE=dev
GCP_PROJECT_ID=ard-eventhub
#
# ard-eventhub
#

# Disable dotenv-linter quote character
# dotenv-linter:off QuoteCharacter

# set service name
SERVICE_NAME=ard-eventhub-ingest

# set node env
IS_LOCAL=true
# PORT=<custom-port>
STAGE=dev

# set datadog tracing
DD_TRACE_ENABLED=false

# set auth
DTS_KEYS="<base64-of-key>"
FIREBASE_API_KEY=<firebase-api-key>
GOOGLE_APPLICATION_CREDENTIALS=./keys/ingest.json

# set Pub/Sub
GCP_PROJECT_ID=ard-eventhub
PUBSUB_SERVICE_ACCOUNT_EMAIL_INTERNAL=<service-account-email>
#PORT=<custom-port>
#DEBUG=true

# set test vars
TEST_USER=<test-user-email>
TEST_USER_PW=<test-user-password>
#TEST_USER_RESET=true
# TEST_USER_RESET=true
20 changes: 4 additions & 16 deletions .eslintrc.json
Original file line number Diff line number Diff line change
@@ -1,21 +1,9 @@
{
"parserOptions": {
"ecmaVersion": 2020
},
"root": true,
"plugins": ["@swrlab/eslint-plugin-swr", "chai-friendly"],
"extends": ["plugin:@swrlab/eslint-plugin-swr/recommended", "plugin:chai-friendly/recommended"],
"ignorePatterns": ["docs/_SIDEBAR.md"],
"env": {},
"extends": [],
"rules": {
"radix": 0,
"no-unused-vars": [
"error",
{
"argsIgnorePattern": "next",
"varsIgnorePattern": "should"
}
],
"no-param-reassign": [2, { "props": false }],
"no-restricted-syntax": 0
"no-undef": "off",
"no-unused-vars": "off"
}
}
23 changes: 12 additions & 11 deletions .github/dependabot.yaml
Original file line number Diff line number Diff line change
@@ -1,13 +1,14 @@
version: 2
updates:
- package-ecosystem: npm
directory: "/"
open-pull-requests-limit: 10
schedule:
interval: "weekly"
day: "monday"
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "weekly"
day: "monday"
- package-ecosystem: "github-actions"
directory: "/"
open-pull-requests-limit: 20
schedule:
interval: "weekly"
day: "sunday"
- package-ecosystem: npm
directory: "/"
open-pull-requests-limit: 20
schedule:
interval: "weekly"
day: "sunday"
119 changes: 57 additions & 62 deletions .github/workflows/ingest-pull.yml
Original file line number Diff line number Diff line change
@@ -1,90 +1,85 @@

name: ingest pull

on:
pull_request_target:
types: [labeled]

env:
NODE_VERSION: 14.17
NODE_VERSION: 22

jobs:
security:
name: Check permissions for pull request runs
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
if: contains(github.event.pull_request.labels.*.name, 'safe to test')
steps:
- run: |
echo "Security checks passed!"
- run: |
echo "Security checks passed!"
test:
needs: security
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
strategy:
matrix:
STAGES: [dev]
steps:
- uses: actions/setup-node@v2.1.5
with:
node-version: ${{ env.NODE_VERSION }}

- name: Checkout repo 👀
uses: actions/checkout@v2.3.4

- name: Get yarn cache directory path
id: yarn-cache-dir-path
run: echo "::set-output name=dir::$(yarn cache dir)"

- uses: actions/cache@v2.1.6
with:
path: ${{ steps.yarn-cache-dir-path.outputs.dir }}
key: ${{ runner.os }}-node-${{ hashFiles('**/yarn.lock') }}
restore-keys: |
${{ runner.os }}-node-
- name: Install Dependencies
run: yarn

- name: 🔑 Setup Google Cloud Auth
uses: google-github-actions/setup-gcloud@master
with:
service_account_key: ${{ secrets.GCP_GITHUB_SERVICE_ACCOUNT_KEY }}
export_default_credentials: true

- name: Run Ingest Tests
env:
STAGE: ${{ matrix.STAGES }}
GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
FIREBASE_API_KEY: ${{ secrets.TEST_FIREBASE_API_KEY }}
TEST_USER: ${{ secrets.TEST_USER }}
TEST_USER_PW: ${{ secrets.TEST_USER_PW }}
PUBSUB_SERVICE_ACCOUNT_EMAIL_INTERNAL: ${{ secrets.PUBSUB_SERVICE_ACCOUNT_EMAIL }}
run: yarn ingest:test
- name: 👀 Checkout Code
uses: actions/checkout@v4

- name: 🏗 Install Node.js
uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}

- name: 🛠 Setup Bun
uses: oven-sh/setup-bun@v2
with:
bun-version: latest

- name: 📦 Install Dependencies
run: bun install

- name: 🔑 Setup Google Cloud Auth
uses: google-github-actions/auth@v2
with:
credentials_json: ${{ secrets.GCP_GITHUB_SERVICE_ACCOUNT_KEY }}
create_credentials_file: true

- name: 📚 Lint Code
run: bun lint

- name: 🧪 Run Ingest Tests
env:
DD_TRACE_ENABLED: false
DTS_KEYS: ${{ secrets.TEST_DTS_KEYS }}
FIREBASE_API_KEY: ${{ secrets.TEST_FIREBASE_API_KEY }}
GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
PUBSUB_SERVICE_ACCOUNT_EMAIL_INTERNAL: ${{ secrets.PUBSUB_SERVICE_ACCOUNT_EMAIL }}
SERVICE_NAME: ard-eventhub-ingest
STAGE: ${{ matrix.STAGES }}
TEST_USER: ${{ secrets.TEST_USER }}
TEST_USER_PW: ${{ secrets.TEST_USER_PW }}
run: bun ingest:test

license:
needs: security
runs-on: ubuntu-latest
runs-on: ubuntu-24.04
steps:
- uses: actions/setup-node@v2.1.5
with:
node-version: ${{ env.NODE_VERSION }}

- name: Checkout repo 👀
uses: actions/checkout@v2.3.4
- name: 👀 Checkout Code
uses: actions/checkout@v4

- name: Get yarn cache directory path
id: yarn-cache-dir-path
run: echo "::set-output name=dir::$(yarn cache dir)"
- name: ⛺️ Install Node.js
uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}

- uses: actions/cache@v2.1.6
with:
path: ${{ steps.yarn-cache-dir-path.outputs.dir }}
key: ${{ runner.os }}-node-${{ hashFiles('**/yarn.lock') }}
restore-keys: |
${{ runner.os }}-node-
- name: 🛠 Setup Bun
uses: oven-sh/setup-bun@v2
with:
bun-version: latest

- name: Install Dependencies
run: yarn
- name: 📦 Install Dependencies
run: bun install

- name: Run License Check
run: yarn license
- name: 📚 Run License Check
run: bun license
Loading