✨ 454 - implement ega job flow

[anncatton]

Work for #454

Implements (mostly happy path) EGA permission reconciliation flow. Error handling needs to be improved in main job function and service calls - in most cases errors encountered should not stop the script from running, but would be useful to report/collect somehow, for debugging/possible retry scenarios.

Note: The logger setup is becoming a PITA, might be useful to replace with more robust logger that implements what buildMessage is doing automatically

EgaClient

• adds error cases for 400 errors in axios interceptor
• adds error handling for 400 - Permission does not exist to revokePermissions - this was caught while debugging ega api pagination issue

Job

• adds main function for reconciliation flow, using functions found in /services

Utils

• adds createPermissionApprovalRequest func for creating permission approval request objects

Types

• adds nullable() to some of the response type fields. These values are not used in the recon flow
• updates RevokePermissionsFailure type to include PERMISSION_DOES_NOT_EXIST
• adds type for combined EGA - DACO user type for creating a map to check existing permissions when iterating over user list
• moves errors.ts to types dir, adds custom errors for BadRequest and ServerError

Code cleanup/readability

• creates a services/ folder for service functions related to users and permissions need in the main reconciliation job
• adds a constants file for pagination values

[joneubank]

I think this is functional and no major problems so I approved with bits of code I would fix. In particular, we suppress the axios error from that one request and don't log what happened.

[joneubank]

you aren't throwing or logging any of the information from the axios error!

[anncatton]

Will be adding more error mgmt/retries in #455

[joneubank]

Our function naming is unclear for reader. For the two functions getUsers and getApprovedUsers, one is fetching from the DACO DB, and the other from the EGA API. There is no indication of which is which.

[joneubank]

Dataset is super generic type name, not sure what this contains... I looked up the type and its just an id, name, and description... still not sure.

Perhaps just prefixing Ega to it is enough to indicate its a structure from their API? EgaDataset?

[joneubank]

Success length of 0 means create request, but success length of 1 or greater means error fetching?

I feel like i don't understand whats happening here but potentially this logic is reversed.

[anncatton]

Yeah it's not written very clearly - the else is actually handling the catch clause from getPermissionByDatasetAndUserId, which returns a failure('SERVER_ERROR'). I'll rewrite as a switch statement so it's easier to read.
The GET /permissions endpoints always return an array, so an empty list means the approved user has no permissions for the requested dataset.

[joneubank]

This just logs outcomes? No data returned to compile a report?

[anncatton]

Will add reporting in #455

[joneubank]

Copy variable, use once, then use original variable...

[joneubank]

Food for thought: I would consider being less strict with all the response validations, only check for data that you need to use. For example, I don't think we ever use EgaPermissionRequest.full_name or EgaPermissionRequest.email, so we should be fine accepting a response without that property. This will prevent our script from failing if they change part of their API that we aren't using.