Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

#269: Using shared disk space for inter-task communication #16

Merged
merged 28 commits into from
Mar 12, 2024
Merged

#269: Using shared disk space for inter-task communication #16

merged 28 commits into from
Mar 12, 2024

Conversation

drewm-swe
Copy link
Contributor

Purpose

  • This PR adds EFS for shared disk space used in inter-task communication within Airflow DAGs.

Proposed Changes

  • [ADD] EFS for shared disk space used in inter-task communication within Airflow DAGs.

Issues

Testing

  • An EKS cluster was manually provisioned in unity-venue-dev using terraform-unity-eks_module. Following the creation of the EKS cluster, an Airflow-based U-SPS was deployed onto the cluster. A hello world DAG utilizing the EFS as a PVC in a KubernetesPodOperator was run successfully.

Drew Meyers and others added 26 commits March 5, 2024 15:48
Volume mount for dag
7cf8389
@LucaCinquini
[CHANGE] Adding additinal parameters to SBG e2e workflow
8f48358
Volume mount testing in dag
abc0ea9
Initial debug pvc mount in dag
4d8b04e
Support of pvc mounting
7b99510
Debug volume mount in dag
faa295c
Rename tf resource
b1662cb
@LucaCinquini
[CHANGE] Using /scratch as working directory, not output directory.
fbbfd95
@LucaCinquini
Merge branch '279-pod-pvc' into 264-execution-of-additional-sbg-workf…
77ddd34 
…lows
@LucaCinquini
[CHANGE] Using the experimental sps-docker-cwl image
1fb7c9d
@LucaCinquini
[CHANGE] Removing the constraint on 1 max concurrent DAG for each type
4e27343
@LucaCinquini
[CHANGE] Switching to max_active_runs=100
45d5545
@LucaCinquini
Using --tmp-outdir-prefix to create the files locally
7e10f63
@LucaCinquini
[FIX] cwl-runner: error: argument --tmp-outdir-prefix: not allowed wi…
696d0dc 
…th argument --cachedir
@LucaCinquini
[CHANGE] Adding volume support to the generic cwl-dag DAG
d0b7c8b
@LucaCinquini
[FIX] Adding shutil
6393397
Update smoke tests github action
0ef6154
Parameterize resource names
b3221cd
Revert pre-commit comment
a8483c7
@LucaCinquini
Merge branch '279-pod-pvc' into 264-execution-of-additional-sbg-workf…
2e5fbe6 
…lows
@LucaCinquini
[CHANGE] Adding parameter for working directory
b62e386
@LucaCinquini
[CHANGE] Creating 2.0.0-alpha.2
2a9e737
@LucaCinquini
[CHANGE] Adding newline at the end of file
ebe5482
@LucaCinquini
[CHANGE] Using ephemeral-storage: "20Gi"
5f4bf0d
@LucaCinquini
[CHANGE] Usig /scratch EFS partition for the DIND Docker directory
fe7ae22
@LucaCinquini
[FIX] Pod calling the latest version of the DIND version
faf17af
@drewm-swe drewm-swe requested a review from LucaCinquini March 11, 2024 23:38
@drewm-swe drewm-swe merged commit 4a4c5da into develop Mar 12, 2024
1 of 2 checks passed
@drewm-swe drewm-swe deleted the 279-pod-pvc branch March 12, 2024 13:34
@drewm-swe drewm-swe mentioned this pull request Mar 12, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@LucaCinquini LucaCinquini Awaiting requested review from LucaCinquini

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@drewm-swe @LucaCinquini