Provide a way to load data programmatically

[torimcd]

Operators have requested the ability to invoke the load_data and load_granule lambda functions via an API call or curl. We may need to add support via something like this: https://docs.aws.amazon.com/lambda/latest/dg/lambda-urls.html

[nikki-t]

I like the idea of the Lambda URL but it looks like each HTTPS request needs to be signed with "AWS Signature Version 4 (SigV4)". And as I was reading through the documentation I found the process to be a little confusing but maybe it is easy to use once we figure out?

Another option might be to invoke the granule with AWS CLI, like so:

aws lambda invoke \
    --cli-binary-format raw-in-base64-out \
    --function-name svc-hydrocron-{venue}-load_data-lambda \
    --invocation-type Event \
    --profile {name_of_aws_profile} \
    --payload ''{"body":{"table_name": "hydrocron-swot-reach-table", "start_date": "2024-01-25T00:00:00", "end_date": "2024-01-30T23:59:59", "obscure_data": "False", "load_benchmarking_data": "False"}}' \
    lambda-out.json

I don't think modifying the payload parameters is too bad since there are not a lot required.

[ymchenjpl]

FYI. We are very familiar with the CLI for HiTIDE Backfill Tool that gets deployed to our on-premise OPS machine and submits backfilling jobs for Forge/TIG/DMRPP to the Services OPS venue. Jim and Simon have incorporated several improvements for us in their CLI tool. That would be another option if the default AWS CLI command doesn't work.

[frankinspace]

Agree with @nikki-t that short term aws cli lambda invoke would be the preferred way to start the lambda programmatically. An ops CLI tool could be scoped out and developed as well.

I don't think we want to enable load data via URL because that gets into Authentication and Authorization concerns very quickly.