U-DS Data Bucket in marketplace

[mike-gangl]

U-DS Data Bucket in marketplace

Allow users to create a data bucket in a given venue. Eventually this will be 'multiple' buckets, but for now, a single bucket should be created.

Acceptance Criteria

Acceptance criteria required to implement the epic

• Support for a creating single bucket is in the terraform
• Users can specify a bucket name in the "Variables" section of the MC
• Bucket has appropriate mandatory tags, once created
  • project - DONE (BUT SHOULD BE NAMED Proj)
  • venue - DONE (BUT Should be capital V)
  • ServiceArea
  • Capability
  • CapVersion
  • Component
  • Name
  • Proj
  • CreatedBy
  • Env
  • Stack
• Mechanisms for adding the shared data services read access to the bucket
  • Access is done automatically through bucket policy during bucket creation https://github.com/unity-sds/unity-data-services/blob/develop/tf-module/marketplace/s3_bucket_policy.json.
• Add bucket name to SSM parameter, automatically in the terraform: /unity/${var.project}/${var.venue}/ds/datastore-bucket
• Make this bucket available to other services (e.g. Jupyter for read access, SPS for read/write access)
• New Event/Triggers are setup to do automated data cataloging
• New Gherkin test cases defined for testing the AC above
  • Developer to define these cases.
• New Test Cases added to System Test (see https://github.com/unity-sds/unity-monorepo/blob/main/tests/system-tests/features/sps/cwl.feature#L15-L16 and https://github.com/unity-sds/unity-monorepo/blob/main/tests/system-tests/features/steps/sps/steps.py#L124 for an example and implementation)
  • @brianlee731 and/or @jimmie to probably under here: https://github.com/unity-sds/unity-cs-infra/blob/main/nightly_tests/test_selenium_mc.py

Work Tickets

Link to work tickets required to implement the epic

• Provide terraform for bucket creation unity-data-services#355
• Figure out how to add newly created buckets to Cumulus policy unity-data-services#360
• Remove project and venue variables from the u-ds marketplace json unity-cs#486
• Demo the deployment of U-DS bucket via MC in a Unity Team meeting unity-cs#487
• Implement Gherkin bucket in marketplace test unity-data-services#429
• Update marketplace bucket tags unity-data-services#463
• Add marketplace bucket name to SSM parameter unity-data-services#464

Dependencies

Other epics or outside tickets required for this to work

• Add SSM parameter that specifies shared services account in venue creation process unity-cs#373

Associated Risks

links to risk issues associated with this epic

• TBC

[mike-gangl]

Blocked by unity-sds/unity-cs#373 for now

[mike-gangl]

/unity/ds/data/bucket/primary-data-bucket --> needs project/venue name

[galenatjpl]

@ngachung and/or @mike-gangl we think this can be closed. Is there anything left to do on this in your minds?

[galenatjpl]

@ngachung and/or @mike-gangl ping.