Skip to content

Commit c0f1f0c

Browse files
wphyojplwphyojpl
authored
feat: Collection Creation endpoint with DAPA format (#79)
* chore: move the unit test * feat: add create collection for cumulus * feat: adding stac to cumulus for json * feat: finished transforming from stac to cumulus * fix: try creating collection with min requirement * feat: add class to create dapa collection obj * fix: add class to create dapa collection object for unity * feat: add class to create STAC collection * feat: update to python3.9 + new lambda for collection creation * chore: python3.9 for github action * fix: ci/cd on develop branch with additional release postfix * fix: testing different output number * fix: testing different output number * fix: testing different output number * fix: testing different output number * fix: testing different output number * fix: testing different output number * fix: testing different output number * fix: testing different output number * fix: testing different output number * chore: add testcase.. in progress * fix: wrong handler for new lambda * fix: require libraries at the worng location * fix: add mock url if url_path is misisng * feat: add rule when creating a collection * fix: wrong quotes in terraform * fix: use thread to run the actual collection creation in background thread * fix: revert background process. does not work as expected in lambda * fix: addig facade for collection creation * fix: api gateway wants string for body. not dict * fix: no need to hash the dict * feat: add single collection details * feat: add single collection detail logic * fix: update testcase + yaml for api gateway * feat: delete collection if rule creation fails * chore: update version
1 parent f10f913 commit c0f1f0c

30 files changed

+1338
-53
lines changed

‎.github/workflows/dockerbuild.yml

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -15,7 +15,7 @@ jobs:
1515
- uses: actions/checkout@v3
1616
- uses: actions/setup-python@v3
1717
with:
18-
python-version: '3.7'
18+
python-version: '3.9'
1919
- run: |
2020
# make file runnable, might not be necessary
2121
chmod +x "${GITHUB_WORKSPACE}/ci.cd/store_version.sh"

‎.github/workflows/makefile.yml

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2,9 +2,9 @@ name: Makefile CI
22

33
on:
44
push:
5-
branches: [ main ]
6-
# pull_request:
7-
# branches: [ main ]
5+
branches: [ main, develop ]
6+
pull_request:
7+
branches: [ develop ]
88

99
env:
1010
ARTIFACT_BASE_NAME: cumulus_lambda_functions
@@ -17,7 +17,7 @@ jobs:
1717
- uses: actions/checkout@v3
1818
- uses: actions/setup-python@v3
1919
with:
20-
python-version: '3.7'
20+
python-version: '3.9'
2121
- run: |
2222
python3 "${GITHUB_WORKSPACE}/setup.py" install
2323
- run: |
@@ -52,7 +52,7 @@ jobs:
5252
prerelease: false
5353
- name: Create PreRelease
5454
id: create_prerelease
55-
if: ${{ contains(github.ref, 'main') }}
55+
# if: ${{ contains(github.ref, 'main') }}
5656
uses: actions/create-release@v1
5757
env:
5858
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} # This token is provided by Actions, you do not need to create your own token
@@ -66,7 +66,7 @@ jobs:
6666
prerelease: true
6767
- name: Upload PreRelease Asset 1
6868
id: upload-prerelease-asset-1
69-
if: ${{ contains(github.ref, 'main') }}
69+
# if: ${{ contains(github.ref, 'main') }}
7070
uses: actions/upload-release-asset@v1
7171
env:
7272
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
@@ -77,7 +77,7 @@ jobs:
7777
asset_content_type: application/zip
7878
- name: Upload PreRelease Asset 2
7979
id: upload-prerelease-asset-2
80-
if: ${{ contains(github.ref, 'main') }}
80+
# if: ${{ contains(github.ref, 'main') }}
8181
uses: actions/upload-release-asset@v1
8282
env:
8383
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

‎ci.cd/Makefile

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -25,3 +25,7 @@ update_lambda_function_2:
2525
aws --profile saml-pub lambda update-function-code --s3-key unity_cumulus_lambda/cumulus_lambda_functions_deployment.zip --s3-bucket am-uds-dev-cumulus-tf-state --function-name arn:aws:lambda:us-west-2:884500545225:function:am-uds-dev-cumulus-cumulus_granules_dapa --publish &>/dev/null
2626
update_lambda_function_3:
2727
aws --profile saml-pub lambda update-function-code --s3-key unity_cumulus_lambda/cumulus_lambda_functions_deployment.zip --s3-bucket am-uds-dev-cumulus-tf-state --function-name arn:aws:lambda:us-west-2:884500545225:function:am-uds-dev-cumulus-cumulus_collections_ingest_cnm_dapa --publish &>/dev/null
28+
update_lambda_function_4:
29+
aws --profile saml-pub lambda update-function-code --s3-key unity_cumulus_lambda/cumulus_lambda_functions_deployment.zip --s3-bucket am-uds-dev-cumulus-tf-state --function-name arn:aws:lambda:us-west-2:884500545225:function:am-uds-dev-cumulus-cumulus_collections_creation_dapa --publish &>/dev/null
30+
update_lambda_function_5:
31+
aws --profile saml-pub lambda update-function-code --s3-key unity_cumulus_lambda/cumulus_lambda_functions_deployment.zip --s3-bucket am-uds-dev-cumulus-tf-state --function-name arn:aws:lambda:us-west-2:884500545225:function:am-uds-dev-cumulus-cumulus_collections_creation_dapa_facade --publish &>/dev/null

‎ci.cd/create_aws_lambda_zip.sh

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -22,6 +22,16 @@ cp ${zip_file} build/
2222
cd $project_root_dir/tf-module/unity-cumulus
2323
zip -9 ${terraform_zip_file} * **/*
2424

25+
# github.job
26+
github_branch=${GITHUB_REF##*/}
27+
software_version_trailing=""
28+
main_branch="main"
29+
if [ "$github_branch" = "$main_branch" ];
30+
then
31+
software_version=""
32+
else
33+
software_version_trailing="-${github_branch}-${GITHUB_RUN_ID}"
34+
fi
2535
software_version=`python3 ${project_root_dir}/setup.py --version`
26-
echo "software_version=${software_version}" >> ${GITHUB_ENV}
36+
echo "software_version=${software_version}${software_version_trailing}" >> ${GITHUB_ENV}
2737
cat ${GITHUB_ENV}

‎cumulus_lambda_functions/cumulus_collections_dapa/cumulus_collections_dapa.py

Lines changed: 16 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,19 @@ def __init__(self, event):
2828
self.__cumulus = CollectionsQuery(self.__cumulus_base, self.__jwt_token)
2929
self.__cumulus.with_limit(self.__limit)
3030
self.__cumulus.with_page_number(self.__page_number)
31+
self.__get_collection_id()
32+
33+
def __get_collection_id(self):
34+
if 'pathParameters' not in self.__event:
35+
return self
36+
path_param_dict = self.__event['pathParameters']
37+
if 'collectionId' not in path_param_dict:
38+
return self
39+
collection_id = path_param_dict['collectionId']
40+
if collection_id == '*':
41+
return self
42+
self.__cumulus.with_collection_id(path_param_dict['collectionId'])
43+
return self
3144

3245
def __assign_values(self):
3346
if 'queryStringParameters' not in self.__event or self.__event['queryStringParameters'] is None:
@@ -61,12 +74,12 @@ def start(self):
6174
if 'server_error' in cumulus_result:
6275
return {
6376
'statusCode': 500,
64-
'body': {'message': cumulus_result['server_error']}
77+
'body': json.dumps({'message': cumulus_result['server_error']})
6578
}
6679
if 'client_error' in cumulus_result:
6780
return {
6881
'statusCode': 400,
69-
'body': {'message': cumulus_result['client_error']}
82+
'body': json.dumps({'message': cumulus_result['client_error']})
7083
}
7184
cumulus_size = self.__get_size()
7285
return {
@@ -84,5 +97,5 @@ def start(self):
8497
LOGGER.exception(f'unexpected error')
8598
return {
8699
'statusCode': 500,
87-
'body': {'message': f'unpredicted error: {str(e)}'}
100+
'body': json.dumps({'message': f'unpredicted error: {str(e)}'})
88101
}

‎cumulus_lambda_functions/cumulus_collections_dapa/cumulus_create_collection_dapa.py

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
import json
2+
import os
3+
4+
import pystac
5+
6+
from cumulus_lambda_functions.cumulus_stac.collection_transformer import CollectionTransformer
7+
from cumulus_lambda_functions.cumulus_wrapper.query_collections import CollectionsQuery
8+
from cumulus_lambda_functions.lib.aws.aws_lambda import AwsLambda
9+
from cumulus_lambda_functions.lib.lambda_logger_generator import LambdaLoggerGenerator
10+
11+
LOGGER = LambdaLoggerGenerator.get_logger(__name__, LambdaLoggerGenerator.get_level_from_env())
12+
13+
14+
class CumulusCreateCollectionDapa:
15+
def __init__(self, event):
16+
required_env = ['CUMULUS_LAMBDA_PREFIX', 'CUMULUS_WORKFLOW_SQS_URL']
17+
if not all([k in os.environ for k in required_env]):
18+
raise EnvironmentError(f'one or more missing env: {required_env}')
19+
self.__event = event
20+
self.__request_body = None
21+
self.__cumulus_collection_query = CollectionsQuery('', '')
22+
self.__cumulus_lambda_prefix = os.getenv('CUMULUS_LAMBDA_PREFIX')
23+
self.__ingest_sqs_url = os.getenv('CUMULUS_WORKFLOW_SQS_URL')
24+
self.__workflow_name = os.getenv('CUMULUS_WORKFLOW_NAME', 'CatalogGranule')
25+
self.__provider_id = '' # TODO. need this?
26+
self.__collection_creation_lambda_name = os.environ.get('COLLECTION_CREATION_LAMBDA_NAME', '').strip()
27+
28+
def execute_creation(self):
29+
try:
30+
cumulus_collection_doc = CollectionTransformer().from_stac(self.__request_body)
31+
creation_result = self.__cumulus_collection_query.create_collection(cumulus_collection_doc, self.__cumulus_lambda_prefix)
32+
if 'status' not in creation_result:
33+
LOGGER.error(f'status not in creation_result: {creation_result}')
34+
return {
35+
'statusCode': 500,
36+
'body': json.dumps({
37+
'message': creation_result
38+
})
39+
}
40+
rule_creation_result = self.__cumulus_collection_query.create_sqs_rules(
41+
cumulus_collection_doc,
42+
self.__cumulus_lambda_prefix,
43+
self.__ingest_sqs_url,
44+
self.__provider_id,
45+
self.__workflow_name,
46+
)
47+
if 'status' not in rule_creation_result:
48+
LOGGER.error(f'status not in rule_creation_result. deleting collection: {rule_creation_result}')
49+
delete_collection_result = self.__cumulus_collection_query.delete_collection(self.__cumulus_lambda_prefix, cumulus_collection_doc['name'], cumulus_collection_doc['version'])
50+
return {
51+
'statusCode': 500,
52+
'body': json.dumps({
53+
'message': {rule_creation_result},
54+
'details': f'collection deletion result: {delete_collection_result}'
55+
})
56+
}
57+
except Exception as e:
58+
LOGGER.exception('error while creating new collection in Cumulus')
59+
return {
60+
'statusCode': 500,
61+
'body': json.dumps({
62+
'message': f'error while creating new collection in Cumulus. check details',
63+
'details': str(e)
64+
})
65+
}
66+
LOGGER.info(f'creation_result: {creation_result}')
67+
return {
68+
'statusCode': 200,
69+
'body': json.dumps({
70+
'message': creation_result
71+
})
72+
}
73+
74+
def start(self):
75+
if 'body' not in self.__event:
76+
raise ValueError(f'missing body in {self.__event}')
77+
self.__request_body = json.loads(self.__event['body'])
78+
LOGGER.debug(f'request body: {self.__request_body}')
79+
validation_result = pystac.Collection.from_dict(self.__request_body).validate()
80+
if not isinstance(validation_result, list):
81+
LOGGER.error(f'request body is not valid STAC collection: {validation_result}')
82+
return {
83+
'statusCode': 500,
84+
'body': json.dumps({'message': f'request body is not valid STAC Collection schema. check details',
85+
'details': validation_result})
86+
}
87+
if self.__collection_creation_lambda_name != '':
88+
response = AwsLambda().invoke_function(
89+
function_name=self.__collection_creation_lambda_name,
90+
payload=self.__event,
91+
)
92+
LOGGER.debug(f'async function started: {response}')
93+
return {
94+
'statusCode': 202,
95+
'body': json.dumps({
96+
'message': 'processing'
97+
})
98+
}
99+
LOGGER.debug(f'creating collection.')
100+
return self.execute_creation()

‎cumulus_lambda_functions/cumulus_collections_dapa/lambda_function.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
from cumulus_lambda_functions.cumulus_collections_dapa.cumulus_collections_dapa import CumulusCollectionsDapa
2+
from cumulus_lambda_functions.cumulus_collections_dapa.cumulus_create_collection_dapa import CumulusCreateCollectionDapa
23
from cumulus_lambda_functions.lib.lambda_logger_generator import LambdaLoggerGenerator
34

45

@@ -12,3 +13,8 @@ def lambda_handler(event, context):
1213
LambdaLoggerGenerator.remove_default_handlers()
1314
# TODO implement
1415
return CumulusCollectionsDapa(event).start()
16+
17+
18+
def lambda_handler_ingestion(event, context):
19+
LambdaLoggerGenerator.remove_default_handlers()
20+
return CumulusCreateCollectionDapa(event).start()

‎cumulus_lambda_functions/cumulus_granules_dapa/cumulus_granules_dapa.py

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -110,12 +110,12 @@ def start(self):
110110
if 'server_error' in cumulus_result:
111111
return {
112112
'statusCode': 500,
113-
'body': {'message': cumulus_result['server_error']}
113+
'body': json.dumps({'message': cumulus_result['server_error']})
114114
}
115115
if 'client_error' in cumulus_result:
116116
return {
117117
'statusCode': 400,
118-
'body': {'message': cumulus_result['client_error']}
118+
'body': json.dumps({'message': cumulus_result['client_error']})
119119
}
120120
cumulus_size = self.__get_size()
121121
return {
@@ -133,5 +133,5 @@ def start(self):
133133
LOGGER.exception(f'unexpected error')
134134
return {
135135
'statusCode': 500,
136-
'body': {'message': f'unpredicted error: {str(e)}'}
136+
'body': json.dumps({'message': f'unpredicted error: {str(e)}'})
137137
}

‎cumulus_lambda_functions/cumulus_granules_dapa_ingest_cnm/cumulus_granules_dapa_ingest_cnm.py

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -193,5 +193,5 @@ def start(self):
193193
}
194194
return {
195195
'statusCode': 500,
196-
'body': {'message': f'failed {len(error_list)}/{len(self.__request_body["features"])}', 'details': error_list}
196+
'body': json.dumps({'message': f'failed {len(error_list)}/{len(self.__request_body["features"])}', 'details': error_list})
197197
}

0 commit comments

Comments
 (0)