Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: swrlab/ard-eventhub
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v1.0.0
Choose a base ref
...
head repository: swrlab/ard-eventhub
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: main
Choose a head ref
Loading
Showing with 7,803 additions and 8,521 deletions.
  1. +42 −0 .editorconfig
  2. +26 −5 .env.example
  3. +4 −16 .eslintrc.json
  4. +12 −11 .github/dependabot.yaml
  5. +66 −62 .github/workflows/ingest-pull.yml
  6. +148 −147 .github/workflows/ingest-push.yml
  7. +3 −5 .gitignore
  8. +109 −0 .license-compliancerc.js
  9. +35 −0 .markdownlint.yaml
  10. +3 −0 .vscode/extensions.json
  11. +19 −0 .vscode/settings.json
  12. +271 −24 CHANGELOG.md
  13. +26 −0 Dockerfile
  14. +0 −22 Dockerfile.ingest
  15. +66 −44 README.md
  16. +5 −5 test/README.md → TESTING.md
  17. +38 −0 biome.json
  18. +2,584 −0 bun.lock
  19. +8 −0 bunfig.toml
  20. 0 config/{coreIdPrefixes.json → core-id-prefixes.json}
  21. +6 −0 config/dts-keys.js
  22. +33 −0 config/event-duplication.json
  23. +44 −35 config/index.js
  24. +15 −0 config/swagger-ui.js
  25. +0 −12 config/swaggerUI.js
  26. +7 −14 docs/AUTHENTICATION.md
  27. +8 −4 docs/EVENTS.md
  28. +17 −0 docs/EXTERNAL_IDS.md
  29. +2 −7 docs/NAMING.md
  30. +3 −3 docs/OPENAPI.md
  31. +40 −0 docs/PLUGINS.md
  32. +20 −30 docs/QUICKSTART.md
  33. +8 −14 docs/SECRETS.md
  34. +1 −6 docs/STAGES.md
  35. +132 −26 docs/TYPES.md
  36. +6 −11 docs/USERS.md
  37. +3 −1 docs/_SIDEBAR.md
  38. +2 −2 index.html
  39. 0 keys/.gitkeep
  40. +0 −102 license.sh
  41. +1,588 −1,220 openapi.json
  42. +378 −82 openapi.yaml
  43. +51 −44 package.json
  44. +21 −0 src/coreId/index.js
  45. +537 −459 src/data/coreApi.json
  46. +33 −25 src/ingest/README.md
  47. +20 −19 src/ingest/auth/login/post.js
  48. +22 −7 src/ingest/auth/middleware/verify.js
  49. +20 −19 src/ingest/auth/refresh/post.js
  50. +23 −23 src/ingest/auth/reset/post.js
  51. +147 −197 src/ingest/events/post.js
  52. +20 −14 src/ingest/index.js
  53. +247 −60 {test → src/ingest}/ingest.test.js
  54. +56 −0 src/ingest/pubsub/index.js
  55. +58 −0 src/ingest/pubsub/verify.js
  56. +31 −22 src/ingest/router.js
  57. +16 −17 src/ingest/subscriptions/delete.js
  58. +6 −4 src/ingest/subscriptions/get.js
  59. +5 −5 src/ingest/subscriptions/list.js
  60. +6 −6 src/ingest/subscriptions/post.js
  61. +4 −2 src/ingest/topics/list.js
  62. +0 −16 src/utils/core/createHashedId.js
  63. +1 −1 src/utils/core/getPublisher.js
  64. +1 −3 src/utils/core/index.js
  65. +1 −1 src/utils/datastore/_client.js
  66. +1 −1 src/utils/datastore/delete.js
  67. +8 −4 src/utils/datastore/index.js
  68. +2 −2 src/utils/datastore/load.js
  69. +2 −2 src/utils/datastore/save.js
  70. +77 −0 src/utils/events/createNewTopic.js
  71. +14 −0 src/utils/events/index.js
  72. +97 −0 src/utils/events/processServices.js
  73. +10 −5 src/utils/firebase/index.js
  74. +18 −22 src/utils/firebase/refreshToken.js
  75. +18 −24 src/utils/firebase/sendPasswordResetEmail.js
  76. +18 −22 src/utils/firebase/signInWithEmailAndPassword.js
  77. +1 −1 src/utils/firebase/verifyToken.js
  78. +27 −16 src/utils/logger/index.js
  79. +169 −0 src/utils/plugins/dts/event.js
  80. +12 −0 src/utils/plugins/dts/index.js
  81. +12 −0 src/utils/plugins/index.js
  82. +1 −1 src/utils/pubsub/_client.js
  83. +1 −1 src/utils/pubsub/_publisherClient.js
  84. +1 −1 src/utils/pubsub/_subscriberClient.js
  85. +3 −3 src/utils/pubsub/buildId.js
  86. +1 −1 src/utils/pubsub/convertId.js
  87. +3 −3 src/utils/pubsub/createSubscription.js
  88. +3 −3 src/utils/pubsub/createTopic.js
  89. +1 −1 src/utils/pubsub/deleteSubscription.js
  90. +1 −1 src/utils/pubsub/getSubscription.js
  91. +1 −1 src/utils/pubsub/getSubscriptions.js
  92. +1 −1 src/utils/pubsub/getTopic.js
  93. +1 −1 src/utils/pubsub/getTopics.js
  94. +1 −1 src/utils/pubsub/index.js
  95. +2 −2 src/utils/pubsub/mapSubscription.js
  96. +4 −13 src/utils/pubsub/publishMessage.js
  97. +2 −2 src/utils/response/badRequest.js
  98. +37 −0 src/utils/response/errors/expiredStartTime.js
  99. +15 −0 src/utils/response/errors/index.js
  100. +38 −0 src/utils/response/errors/mismatchingEventName.js
  101. +12 −5 src/utils/response/index.js
  102. +3 −3 src/utils/response/internalServerError.js
  103. +3 −3 src/utils/response/notFound.js
  104. +3 −3 src/utils/response/ok.js
  105. +16 −0 src/utils/secrets/_client.js
  106. +29 −0 src/utils/secrets/get.js
  107. +14 −0 src/utils/secrets/index.js
  108. +1 −2 src/utils/tracer/index.js
  109. +15 −0 src/utils/undici/index.js
  110. +0 −10 src/utils/user/index.js
  111. +0 −22 test/example.test.js
  112. +0 −5,484 yarn.lock
42 changes: 42 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# docs: https://editorconfig.org
# please make sure to keep this file in sync between repositories
# to main file should be edited in `swr-prettier-config` and copied where needed

root = true

[*]
charset = utf-8
end_of_line = lf
indent_style = tab
indent_size = 8
print_width = 120
trim_trailing_whitespace = true
insert_final_newline = true

[*.md]
trim_trailing_whitespace = false
indent_style = space
indent_size = 2

[*.{yml,yaml}]
indent_style = space
indent_size = 2

# LiquidSoap config files
[*.liq]
indent_style = space
indent_size = 2

# Terraform config files
[*.tf]
indent_style = space
indent_size = 2

# Linting toml config files
[*.toml]
indent_style = space
indent_size = 2

# Auto-generated CSV Files
[*.csv]
insert_final_newline = false
31 changes: 26 additions & 5 deletions .env.example
Original file line number Diff line number Diff line change
@@ -1,10 +1,31 @@
STAGE=dev
GCP_PROJECT_ID=ard-eventhub
#
# ard-eventhub
#

# Disable dotenv-linter quote character
# dotenv-linter:off QuoteCharacter

# set service name
SERVICE_NAME=ard-eventhub-ingest

# set node env
IS_LOCAL=true
# PORT=<custom-port>
STAGE=dev

# set datadog tracing
DD_TRACE_ENABLED=false

# set auth
DTS_KEYS="<base64-of-key>"
FIREBASE_API_KEY=<firebase-api-key>
GOOGLE_APPLICATION_CREDENTIALS=./keys/ingest.json
#PORT=<custom-port>
#DEBUG=true

# set Pub/Sub
GCP_PROJECT_ID=ard-eventhub
PUBSUB_SERVICE_ACCOUNT_EMAIL_INTERNAL=<service-account-email>

# set test vars
TEST_USER=<test-user-email>
TEST_USER_PW=<test-user-password>
#TEST_USER_RESET=true
# TEST_USER_RESET=true
20 changes: 4 additions & 16 deletions .eslintrc.json
Original file line number Diff line number Diff line change
@@ -1,21 +1,9 @@
{
"parserOptions": {
"ecmaVersion": 2020
},
"root": true,
"plugins": ["@swrlab/eslint-plugin-swr", "chai-friendly"],
"extends": ["plugin:@swrlab/eslint-plugin-swr/recommended", "plugin:chai-friendly/recommended"],
"ignorePatterns": ["docs/_SIDEBAR.md"],
"env": {},
"extends": [],
"rules": {
"radix": 0,
"no-unused-vars": [
"error",
{
"argsIgnorePattern": "next",
"varsIgnorePattern": "should"
}
],
"no-param-reassign": [2, { "props": false }],
"no-restricted-syntax": 0
"no-undef": "off",
"no-unused-vars": "off"
}
}
23 changes: 12 additions & 11 deletions .github/dependabot.yaml
Original file line number Diff line number Diff line change
@@ -1,13 +1,14 @@
version: 2
updates:
- package-ecosystem: npm
directory: "/"
open-pull-requests-limit: 10
schedule:
interval: daily
time: '08:00'
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "daily"
time: '08:00'
- package-ecosystem: "github-actions"
directory: "/"
open-pull-requests-limit: 20
schedule:
interval: "weekly"
day: "sunday"
- package-ecosystem: npm
directory: "/"
open-pull-requests-limit: 20
schedule:
interval: "weekly"
day: "sunday"
128 changes: 66 additions & 62 deletions .github/workflows/ingest-pull.yml
Original file line number Diff line number Diff line change
@@ -1,81 +1,85 @@

name: ingest pull


on:
pull_request:

pull_request_target:
types: [labeled]

env:
NODE_VERSION: 14.5

NODE_VERSION: 22

jobs:
security:
name: Check permissions for pull request runs
runs-on: ubuntu-24.04
if: contains(github.event.pull_request.labels.*.name, 'safe to test')
steps:
- run: |
echo "Security checks passed!"
test:
runs-on: ubuntu-latest
needs: security
runs-on: ubuntu-24.04
strategy:
matrix:
STAGES: [dev]
steps:
- uses: actions/setup-node@v2
with:
node-version: ${{ env.NODE_VERSION }}

- name: Checkout repo 👀
uses: actions/checkout@v2

- name: Get yarn cache directory path
id: yarn-cache-dir-path
run: echo "::set-output name=dir::$(yarn cache dir)"

- uses: actions/cache@v2
with:
path: ${{ steps.yarn-cache-dir-path.outputs.dir }}
key: ${{ runner.os }}-node-${{ hashFiles('**/yarn.lock') }}
restore-keys: |
${{ runner.os }}-node-
- name: Install Dependencies
run: yarn

- name: 🔑 Setup Google Cloud Auth
uses: google-github-actions/setup-gcloud@master
with:
service_account_key: ${{ secrets.GCP_GITHUB_SERVICE_ACCOUNT_KEY }}
export_default_credentials: true

- name: Run Ingest Tests
env:
STAGE: ${{ matrix.STAGES }}
GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
FIREBASE_API_KEY: ${{ secrets.TEST_FIREBASE_API_KEY }}
TEST_USER: ${{ secrets.TEST_USER }}
TEST_USER_PW: ${{ secrets.TEST_USER_PW }}
run: yarn ingest:test
- name: 👀 Checkout Code
uses: actions/checkout@v4

- name: 🏗 Install Node.js
uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}

- name: 🛠 Setup Bun
uses: oven-sh/setup-bun@v2
with:
bun-version: latest

- name: 📦 Install Dependencies
run: bun install

- name: 🔑 Setup Google Cloud Auth
uses: google-github-actions/auth@v2
with:
credentials_json: ${{ secrets.GCP_GITHUB_SERVICE_ACCOUNT_KEY }}
create_credentials_file: true

- name: 📚 Lint Code
run: bun lint

- name: 🧪 Run Ingest Tests
env:
DD_TRACE_ENABLED: false
DTS_KEYS: ${{ secrets.TEST_DTS_KEYS }}
FIREBASE_API_KEY: ${{ secrets.TEST_FIREBASE_API_KEY }}
GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }}
PUBSUB_SERVICE_ACCOUNT_EMAIL_INTERNAL: ${{ secrets.PUBSUB_SERVICE_ACCOUNT_EMAIL }}
SERVICE_NAME: ard-eventhub-ingest
STAGE: ${{ matrix.STAGES }}
TEST_USER: ${{ secrets.TEST_USER }}
TEST_USER_PW: ${{ secrets.TEST_USER_PW }}
run: bun ingest:test

license:
runs-on: ubuntu-latest
needs: security
runs-on: ubuntu-24.04
steps:
- uses: actions/setup-node@v2
with:
node-version: ${{ env.NODE_VERSION }}

- name: Checkout repo 👀
uses: actions/checkout@v2
- name: 👀 Checkout Code
uses: actions/checkout@v4

- name: Get yarn cache directory path
id: yarn-cache-dir-path
run: echo "::set-output name=dir::$(yarn cache dir)"
- name: ⛺️ Install Node.js
uses: actions/setup-node@v4
with:
node-version: ${{ env.NODE_VERSION }}

- uses: actions/cache@v2
with:
path: ${{ steps.yarn-cache-dir-path.outputs.dir }}
key: ${{ runner.os }}-node-${{ hashFiles('**/yarn.lock') }}
restore-keys: |
${{ runner.os }}-node-
- name: 🛠 Setup Bun
uses: oven-sh/setup-bun@v2
with:
bun-version: latest

- name: Install Dependencies
run: yarn
- name: 📦 Install Dependencies
run: bun install

- name: Run License Check
run: yarn license
- name: 📚 Run License Check
run: bun license
Loading