Merge pull request #60 from swrlab/dev/beta-fixes

Dev/beta fixes

.env.example

@@ -0,0 +1,10 @@
+STAGE=dev 
+GCP_PROJECT_ID=ard-eventhub
+FIREBASE_API_KEY=<firebase-api-key>
+GOOGLE_APPLICATION_CREDENTIALS=./keys/ingest.json
+#PORT=<custom-port>
+#DEBUG=true
+
+TEST_USER=<test-user-email>
+TEST_USER_PW=<test-user-password>
+#TEST_USER_RESET=true

CHANGELOG.md

@@ -5,6 +5,12 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.0.0-beta2] - 2021-03-25
+
+### Added
+
+- Add trailing 0 in service publisher-ids if number is only 5 digits
+
 ## [1.0.0-beta1] - 2021-03-24
 
 🚧 BREAKING CHANGES for `serviceIds` 🚧

README.md

@@ -7,8 +7,10 @@ ARD system to distribute real-time (live) metadata for primarily radio broadcast
   - [Get involved](#get-involved)
   - [Modules](#modules)
     - [Ingest](#ingest)
+  - [Changelog](#changelog)
   - [License](#license)
   - [Third-Party Components](#third-party-components)
+  - [Authors](#authors)
 
 ## Get Started and Documentation
 
@@ -33,6 +35,10 @@ This project will include two modules: Ingest and API. The first development ste
 
 The Ingest service is responsible for receiving and publishing events and managing subscriptions. You'll find the core code in [`./src/ingest/`](./src/ingest/).
 
+## Changelog
+
+See [CHANGELOG.md](CHANGELOG.md) for latest changes.
+
 ## License
 
 This project is licensed under **European Union Public License 1.2** ([`EUPL-1.2`](https://spdx.org/licenses/EUPL-1.2.html)), which can be found in [LICENSE.txt](LICENSE.txt). Detailed information and translations to all 23 official languages of the European Union are available at [joinup.ec.europa.eu](https://joinup.ec.europa.eu/collection/eupl/eupl-text-eupl-12). The usage of this license does not imply any relationship to or endorsement by the European Union, the joinup initiative or other participating parties.  
@@ -66,11 +72,19 @@ This source code is provided under EUPL v1.2, except for the [`spdx-exceptions`]
 | NPM DEV | `chai`                        | [MIT](https://github.com/chaijs/chai/blob/master/LICENSE)                                 |
 | NPM DEV | `chai-http`                   | [MIT](https://github.com/chaijs/chai-http/blob/master/package.json)                       |
 | NPM DEV | `docsify-cli`                 | [MIT](https://github.com/docsifyjs/docsify-cli/blob/master/LICENSE)                       |
-| NPM DEV | `dotenv`                      | [BSD-2-Clause](<[BSD-2-Clause](https://github.com/motdotla/dotenv/blob/master/LICENSE)>)  |
+| NPM DEV | `dotenv`                      | [BSD-2-Clause](https://github.com/motdotla/dotenv/blob/master/LICENSE)                    |
 | NPM DEV | `eslint`                      | [MIT](https://github.com/eslint/eslint/blob/master/LICENSE)                               |
 | NPM DEV | `eslint-plugin-swr`           | [ISC](https://github.com/swrlab/eslint-plugin-swr/blob/main/package.json)                 |
 | NPM DEV | `eslint-plugin-chai-friendly` | [MIT](https://github.com/ihordiachenko/eslint-plugin-chai-friendly/blob/master/LICENSE)   |
 | NPM DEV | `license-compliance`          | [MIT](https://github.com/tmorell/license-compliance/blob/master/LICENSE)                  |
 | NPM DEV | `mocha`                       | [MIT](https://github.com/mochajs/mocha/blob/master/LICENSE)                               |
 | NPM DEV | `nodemon`                     | [MIT](https://github.com/remy/nodemon/blob/master/LICENSE)                                |
 | NPM DEV | `prettier`                    | [MIT](https://github.com/prettier/prettier/blob/main/LICENSE)                             |
+
+## Authors
+
+This project was realized by
+
+- [Daniel Freytag](https://github.com/frytg)
+- [Rafael Mäuer](https://github.com/rafaelmaeuer)
+- [Christian Hufnagel](https://github.com/chhufnagel)

openapi.json

@@ -1,5 +1,5 @@
 {
-  "openapi": "3.0.1",
+  "openapi": "3.0.3",
   "info": {
     "title": "ARD-Eventhub",
     "description": "ARD system to distribute real-time (live) metadata for primarily radio broadcasts.",
@@ -239,6 +239,36 @@
                 }
               }
             }
+          },
+          "401": {
+            "description": "Missing authentication",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/errorUnauthorized"
+                }
+              }
+            }
+          },
+          "403": {
+            "description": "Invalid authorization",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/errorForbidden"
+                }
+              }
+            }
+          },
+          "500": {
+            "description": "Internal server error",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/errorInternalServerError"
+                }
+              }
+            }
           }
         }
       }
@@ -272,6 +302,36 @@
                 }
               }
             }
+          },
+          "401": {
+            "description": "Missing authentication",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/errorUnauthorized"
+                }
+              }
+            }
+          },
+          "403": {
+            "description": "Invalid authorization",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/errorForbidden"
+                }
+              }
+            }
+          },
+          "500": {
+            "description": "Internal server error",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/errorInternalServerError"
+                }
+              }
+            }
           }
         }
       }
@@ -420,6 +480,16 @@
                 }
               }
             }
+          },
+          "500": {
+            "description": "Internal server error",
+            "content": {
+              "application/json": {
+                "schema": {
+                  "$ref": "#/components/schemas/errorInternalServerError"
+                }
+              }
+            }
           }
         }
       }

openapi.yaml

@@ -1,4 +1,4 @@
-openapi: 3.0.1
+openapi: 3.0.3
 info:
   title: ARD-Eventhub
   description: >-
@@ -151,6 +151,24 @@ paths:
             application/json:
               schema:
                 $ref: '#/components/schemas/errorBadRequest'
+        '401':
+          description: Missing authentication
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/errorUnauthorized'
+        '403':
+          description: Invalid authorization
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/errorForbidden'
+        '500':
+          description: Internal server error
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/errorInternalServerError'
   '/events/de.ard.eventhub.v1.radio.track.playing':
     post:
       tags:
@@ -171,6 +189,24 @@ paths:
             application/json:
               schema:
                 $ref: '#/components/schemas/errorBadRequest'
+        '401':
+          description: Missing authentication
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/errorUnauthorized'
+        '403':
+          description: Invalid authorization
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/errorForbidden'
+        '500':
+          description: Internal server error
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/errorInternalServerError'
   /subscriptions:
     get:
       tags:
@@ -263,6 +299,12 @@ paths:
             application/json:
               schema:
                 $ref: '#/components/schemas/errorNotFound'
+        '500':
+          description: Internal server error
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/errorInternalServerError'
   '/subscriptions/{name}':
     get:
       tags:

package.json

@@ -5,9 +5,9 @@
 	"main": "./src/ingest/index.js",
 	"scripts": {
 		"docs:serve": "docsify serve . --port 3000",
-		"ingest:test": "mocha test/ingest.test.js --timeout 15000 --exit",
+		"ingest:test": "mocha test/ingest.test.js --timeout 15000 --exit -r dotenv/config",
 		"ingest:local": "nodemon -r dotenv/config ./src/ingest/index.js",
-		"ingest:cloud": "node ./src/ingest/index.js",
+		"ingest:cloud": "node -r dotenv/config ./src/ingest/index.js",
 		"license": "chmod +x license.sh && ./license.sh",
 		"test": "mocha test/example.test.js"
 	},
@@ -23,10 +23,10 @@
 		"@google-cloud/pubsub": "^2.10.0",
 		"body-parser": "^1.19.0",
 		"compression": "^1.7.4",
-		"dd-trace": "^0.31.2",
+		"dd-trace": "^0.31.4",
 		"express": "4.17.1",
-		"express-openapi-validator": "^4.12.6",
-		"firebase-admin": "^9.5.0",
+		"express-openapi-validator": "^4.12.7",
+		"firebase-admin": "^9.6.0",
 		"jsonwebtoken": "^8.5.1",
 		"moment": "^2.29.1",
 		"node-crc": "^1.3.0",
@@ -43,10 +43,10 @@
 		"chai-http": "^4.3.0",
 		"docsify-cli": "^4.4.3",
 		"dotenv": "^8.2.0",
-		"eslint": "^7.22.0",
+		"eslint": "^7.23.0",
 		"eslint-plugin-chai-friendly": "^0.6.0",
 		"eslint-plugin-swr": "0.0.5",
-		"license-compliance": "^1.1.0",
+		"license-compliance": "^1.2.0",
 		"mocha": "^8.3.2",
 		"nodemon": "^2.0.7",
 		"prettier": "^2.2.1"

src/ingest/README.md

@@ -8,13 +8,14 @@ The Ingest service is used to accept incoming events, distribute them via Pub/Su
 
 Designated host is Kubernetes but the Docker container will also be used in other environments such as Google Cloud Run for testing purposes.
 
-It needs several environment variables to work:
+Several environment variables need to be set in `.env` config in order to run the project:
 
-- REQUIRED `FIREBASE_API_KEY` - corresponding `API_KET` which matches the `GCP_PROJECT_ID`
-- REQUIRED `GOOGLE_APPLICATION_CREDENTIALS` - where the Google Cloud Service Account Key can be found (usually a path to a .json file)
 - REQUIRED `GCP_PROJECT_ID` - which GCP project ID to use for Pub/Sub and Datastore requests
-- OPTIONAL `PORT` - override server port setting, default is 8080
+- REQUIRED `FIREBASE_API_KEY` - corresponding `API_KEY` which matches the `GCP_PROJECT_ID`
+- REQUIRED `GOOGLE_APPLICATION_CREDENTIALS` - where the Google Cloud Service Account Key can be found (usually a path to a .json file)
 - REQUIRED `STAGE` - can be one of the Stages below to switch several settings
+- OPTIONAL `PORT` - override server port setting, default is 8080
+- OPTIONAL `DEBUG` - set true to enable more detailed logging
 
 ### Stages
 
@@ -42,12 +43,9 @@ To run this project locally in your development environment you'll need these pr
    6. `roles/monitoring.metricWriter`
    7. `roles/pubsub.admin`
 3. Install dependencies (`yarn`)
-4. Run the project (replace `gcp-project` and `fb-api-key`)
+4. Run the project
 
    ```sh
-   STAGE=dev \
-   GCP_PROJECT_ID=gcp-project \
-   FIREBASE_API_KEY=fb-api-key \
    yarn ingest:local
    ```
 

src/ingest/events/post.js

@@ -28,11 +28,11 @@ module.exports = async (req, res) => {
 		const { user } = req
 
 		// check eventName
-		if (message.event && eventName !== message.event) {
+		if (message?.event !== eventName) {
 			// log access attempt
 			logger.log({
 				level: 'warning',
-				message: 'User attempted event with missmatching names',
+				message: 'User attempted event with mismatching names',
 				source,
 				data: {
 					email: req.user.email,
@@ -183,7 +183,7 @@ module.exports = async (req, res) => {
 				// handle errors
 				if (messageId === 'TOPIC_ERROR') {
 					// insert error message and empty id
-					service.topic.status = 'TOPIC_ERROR_1'
+					service.topic.status = 'TOPIC_ERROR'
 					service.topic.messageId = null
 				} else if (messageId === 'TOPIC_NOT_FOUND') {
 					// fetch publisher
@@ -228,7 +228,7 @@ module.exports = async (req, res) => {
 						})
 					} else {
 						// Update api result that topic was not created
-						service.topic.status = 'TOPIC_ERROR_2'
+						service.topic.status = 'TOPIC_NOT_CREATED'
 
 						logger.log({
 							level: 'error',

src/ingest/router.js

@@ -64,7 +64,7 @@ router.delete('/subscriptions/:subscriptionName', authVerify, require('./subscri
 router.get('/topics/', authVerify, require('./topics/list'))
 router.get('/topics/:topicName', authVerify, require('./topics/list'))
 
-// send healthcheck ok
+// send health-check ok
 router.get(['/', '/health'], (req, res) => {
 	res.sendStatus(200)
 })

test/README.md

@@ -7,27 +7,16 @@ As test-environment [Mocha](https://mochajs.org/) is used in combination with [C
 
 ### Environments
 
-It needs several environment variables to work:
+In addition to the [ingest-env](../src/ingest/README.md#Environments), following variables are needed in `.env` config for unit tests to work:
 
-- REQUIRED `STAGE` - can be one of the Stages DEV / PROD
-- REQUIRED `GCP_PROJECT_ID` - which GCP project ID to use for Pub/Sub and Datastore requests
-- REQUIRED `FIREBASE_API_KEY` - corresponding `API_KEY` which matches the `GCP_PROJECT_ID`
-- OPTIONAL `PORT` - override server port setting, default is 8080
 - REQUIRED `TEST_USER` - test user email
 - REQUIRED `TEST_USER_PW` - test user password
-- OPTIONAL `TEST_USER_RESET` - test email reset (request limit)
+- OPTIONAL `TEST_USER_RESET` - set true for email reset (request limit)
 
 ## Setup
 
-To run the tests follow the [ingest-setup](../src/ingest/README.md) first
-
-Run the project (replace `gcp-project`, `fb-api-key`, `test-user-email` and `test-user-password`)
+Follow the [ingest-setup](../src/ingest/README.md) first, then run tests with
 
    ```sh
-   STAGE=dev \
-   GCP_PROJECT_ID=gcp-project \
-   FIREBASE_API_KEY=fb-api-key \
-   TEST_USER=test-user-email \
-   TEST_USER_PW=test-user-password \
    yarn ingest:test
    ```

test/ingest.test.js

@@ -97,7 +97,7 @@ describe(`POST ${refreshPath}`, () => {
 // 🚨 firebase limit is 150 requests per day 🚨
 const resetPath = '/auth/reset'
 
-if (process.env.TEST_USER_RESET) {
+if (process.env.TEST_USER_RESET === true) {
 	describe(`POST ${resetPath}`, () => {
 		it('request password reset email', (done) => {
 			const resetRequest = {
@@ -128,7 +128,6 @@ function testEventKeys(body) {
 const eventName = 'de.ard.eventhub.v1.radio.track.playing'
 const eventPath = `/events/${eventName}`
 
-const swrTV = '990030'
 const event = {
 	event: eventName,
 	type: 'music',