U-DS Data Bucket in marketplace

[mike-gangl]

U-DS Data Bucket in marketplace

Allow users to create a data bucket in a given venue. Eventually this will be 'multiple' buckets, but for now, a single bucket should be created.

Acceptance Criteria

Acceptance criteria required to implement the epic

• Support for a creating single bucket is in the terraform
• Users can specify a bucket name in the "Variables" section of the MC
• Bucket has appropriate mandatory tags, once created
  • project - DONE (BUT SHOULD BE NAMED Proj)
  • venue - DONE (BUT Should be capital V)
  • ServiceArea
  • Capability
  • CapVersion
  • Component
  • Name
  • Proj
  • CreatedBy
  • Env
  • Stack
• Mechanisms for adding the shared data services read access to the bucket
  • Access is done automatically through bucket policy during bucket creation https://github.com/unity-sds/unity-data-services/blob/develop/tf-module/marketplace/s3_bucket_policy.json.
• Add bucket name to SSM parameter, automatically in the terraform: /unity/${var.project}/${var.venue}/ds/datastore-bucket
• Make this bucket available to other services (e.g. Jupyter for read access, SPS for read/write access)
• New Event/Triggers are setup to do automated data cataloging
• New Gherkin test cases defined for testing the AC above
  • Developer to define these cases.
• New Test Cases added to System Test (see https://github.com/unity-sds/unity-monorepo/blob/main/tests/system-tests/features/sps/cwl.feature#L15-L16 and https://github.com/unity-sds/unity-monorepo/blob/main/tests/system-tests/features/steps/sps/steps.py#L124 for an example and implementation)
  • @brianlee731 and/or @jimmie to probably under here: https://github.com/unity-sds/unity-cs-infra/blob/main/nightly_tests/test_selenium_mc.py

Work Tickets

Link to work tickets required to implement the epic

• Provide terraform for bucket creation unity-data-services#355
• Figure out how to add newly created buckets to Cumulus policy unity-data-services#360
• Remove project and venue variables from the u-ds marketplace json unity-cs#486
• Demo the deployment of U-DS bucket via MC in a Unity Team meeting unity-cs#487
• Implement Gherkin bucket in marketplace test unity-data-services#429
• Update marketplace bucket tags unity-data-services#463
• Add marketplace bucket name to SSM parameter unity-data-services#464

Dependencies

Other epics or outside tickets required for this to work

• Add SSM parameter that specifies shared services account in venue creation process unity-cs#373

Associated Risks

links to risk issues associated with this epic

• TBC

[mike-gangl]

Blocked by unity-sds/unity-cs#373 for now

[mike-gangl]

/unity/ds/data/bucket/primary-data-bucket --> needs project/venue name

[galenatjpl]

@ngachung and/or @mike-gangl we think this can be closed. Is there anything left to do on this in your minds?

[galenatjpl]

@ngachung and/or @mike-gangl ping.

[ngachung]

@galenatjpl IIRC we kept this open because using a zip of the terraform doesn't work?

[mike-gangl]

This is probably REALLY close to being closed, but i wanted to ensure that we had the process for adding the venue bucket to the shared-services system documented, as well as the hooks/triggers to support automated cataloging.

[galenatjpl]

@mike-gangl @ngachung I'm going to try this now on a new management console, and just see where we are. It looks like we have marketplace integration, and I'll see what happens:
See:
https://github.com/unity-sds/unity-data-services/blob/develop/tf-module/marketplace/s3_bucket_creation.tf
and
https://github.com/unity-sds/unity-marketplace/blob/main/applications/unity-ds/7.7.0/metadata.json

[galenatjpl]

@mike-gangl @ngachung @brianlee731 @GodwinShen @wphyojpl , I was able to deploy. It seemed to work, and I think this ticket can be closed? Please let me know if you want me to show you what I did to deploy via the MC and Marketplace.

[mike-gangl]

I'd like to make sure, to close out this ticket, we add the Validation/ Integration testing.

gherkin:

Feature: Data-Services Bucket via Management Console
  feature to define the validation for the bucket deployment through management console.

@develop @test
Scenario: Check for bucket creation and configuration

  Given access to the venue AWS account 
  And a market_bucket__notification_prefix of stage_out
  And bucket is deployed via the marketplace with specified bucketname #to be referenced in env files
  When I query the buckets via AWS
  Then i can see the bucket in the bucket listing
  And i can query bucket details for bucketname  #to be referenced in env files
  And i can see an SNS event notifications for .json file in the stage_out prefix

The Given section is where the bucket is actually created- this can be defined as a manual test if it cannot be automated by using the @Manual decorator (currently not used next to develop, test).

[mike-gangl]

I'd like to make sure, to close out this ticket, we add the Validation/ Integration testing.

gherkin:

Feature: Data-Services Bucket via Management Console
  feature to define the validation for the bucket deployment through management console.

@develop @test
Scenario: Check for bucket creation and configuration

  Given access to the venue AWS account 
  And a market_bucket__notification_prefix of stage_out
  And bucket is deployed via the marketplace with specified bucketname #to be referenced in env files
  When I query the buckets via AWS
  Then i can see the bucket in the bucket listing
  And i can query bucket details for bucketname  #to be referenced in env files
  And i can see an SNS event notifications for .json file in the stage_out prefix

The Given section is where the bucket is actually created- this can be defined as a manual test if it cannot be automated by using the @Manual decorator (currently not used next to develop, test).

As I learn more about gherkin and behave (the python framework that runs the tests), there's a chance we use a before_feature and after_feature hook that would do the bucket creation via marketplace and the bucket deletion (via marketplace). This would allow us to create a number of features for checking things like the setup, add a file to see if a .json notification is sent, etc and then delete the bucket when completed.

[brianlee731]

@ngachung Galen provide status that U-CS completed their respective scope for this ticket. Please proceed with U-DS scope of work.

[galenatjpl]

This ticket, I think is basically done from a core U-CS and "integration into the marketplace" perspective. There are some remaining items to do (that mostly fall under the purview of the U-DS Team), that are unrelated to the actual integration into the marketplace. See Acceptance criteria section in the description of this issue.

[galenatjpl]

@ngachung I marked the "Make this bucket available to other services (e.g. Jupyter for read access, SPS for read/write access)" checkbox as complete, since I think the bucket should be accessible to Jupyter and SPS, no problem in this venue. Let me know if you have concerns. I didn't actually test this access out, but I think @LucaCinquini can from SPS somehow? @LucaCinquini let me know if you want to setup such a test.

[brianlee731]

Gherkin Case: unity-monorepo/tests/system-tests/features/data_catalog/managed_service.feature

This test will be executed manually - will not implement via selenium; will require a lot of code change to just test deployment.

[ngachung]

@galenatjpl I am not convinced that Jupyter read access happens "automatically" after the bucket is deployed.

[galenatjpl]

@ngachung What I mean, is that usually read access to buckets are allowed under the base IAM roles that a user would have under the exact same account. Since Jupyter is in the same account and region as the bucket, I assume it would work, but I could be wrong, and it should be tested. I think we should ask @mcduffie to test.

[ngachung]

If it helps, I can confirm it doesn't work. A newly created bucket in unity venue prod is not accessible from Jupyter. Jupyter uses a different IAM role. Somewhat related ticket unity-sds/unity-ads-deployment#177