Merge filestore usage feature into develop #319

pawloch00 · 2025-01-10T15:45:51Z

Fixes / Features

Cherry pick filestore related changes from development branch into main branch

Testing / Documentation

Testing details.

[ y/n ] Tests pass
[ y/n ] Appropriate changes to documentation are included in the PR

* Enabling Workload Identity and GCSFuse driver flags added.

* Introduce Storage API * Fix Gh CI/CD * Improve getting storages to mount to workload * Improve handling exception while deleting storage * Fix adding necessary args and cluster credentials * Fix storage class, differentiate storage types in create and delete commands, add --type validation, reduce number of CRD installations

Signed-off-by: Piotr Pawłowski <[email protected]>

* Add Storage to workload creation * Improve documentation

…pawl-cherry-pick

Signed-off-by: Piotr Pawłowski <[email protected]>

* apply pathways fixes Signed-off-by: Piotr Pawłowski <[email protected]> --------- Signed-off-by: Piotr Pawłowski <[email protected]>

Signed-off-by: Piotr Pawłowski <[email protected]>

* Add flags to support GCP Filestore on cluster Signed-off-by: Piotr Pawłowski <[email protected]> --------- Signed-off-by: Piotr Pawłowski <[email protected]>

* enable to use filestore in workload Signed-off-by: Piotr Pawłowski <[email protected]> --------- Signed-off-by: Piotr Pawłowski <[email protected]> Co-authored-by: Patryk Bundyra <[email protected]>

* add flag parser and enable GCPFilestore on cluster Signed-off-by: Piotr Pawłowski <[email protected]> * update cluster with gcp filestore Signed-off-by: Piotr Pawłowski <[email protected]> * fix linter Signed-off-by: Piotr Pawłowski <[email protected]> * filestore -> gcpfilestore Signed-off-by: Piotr Pawłowski <[email protected]> * filetore -> gcpfilestorage Signed-off-by: Piotr Pawłowski <[email protected]> * create filestore instance Signed-off-by: Piotr Pawłowski <[email protected]> * add enable filestore Signed-off-by: Piotr Pawłowski <[email protected]> * enable gcpfilestore on cluster creation Signed-off-by: Piotr Pawłowski <[email protected]> * pyink Signed-off-by: Piotr Pawłowski <[email protected]> * add to workload Signed-off-by: Piotr Pawłowski <[email protected]> * pyink Signed-off-by: Piotr Pawłowski <[email protected]> * fix linter Signed-off-by: Piotr Pawłowski <[email protected]> * fix string formatting Signed-off-by: Piotr Pawłowski <[email protected]> * apply manifest file Signed-off-by: Piotr Pawłowski <[email protected]> * add workloads Signed-off-by: Piotr Pawłowski <[email protected]> * Fix get storage function * pyink; Signed-off-by: Piotr Pawłowski <[email protected]> * remove adding iam Signed-off-by: Piotr Pawłowski <[email protected]> * rm kjob Signed-off-by: Piotr Pawłowski <[email protected]> * rm examples Signed-off-by: Piotr Pawłowski <[email protected]> * rm duplicated Signed-off-by: Piotr Pawłowski <[email protected]> * update filestore iam Signed-off-by: Piotr Pawłowski <[email protected]> * cleanup Signed-off-by: Piotr Pawłowski <[email protected]> * add readme base Signed-off-by: Piotr Pawłowski <[email protected]> * update readme Signed-off-by: Piotr Pawłowski <[email protected]> * README.md update Signed-off-by: Piotr Pawłowski <[email protected]> * update Readme Signed-off-by: Piotr Pawłowski <[email protected]> * fix linter Signed-off-by: Piotr Pawłowski <[email protected]> * add storage filestore pipeline Signed-off-by: Piotr Pawłowski <[email protected]> * fix builds Signed-off-by: Piotr Pawłowski <[email protected]> * fix pipeline Signed-off-by: Piotr Pawłowski <[email protected]> * fix naming Signed-off-by: Piotr Pawłowski <[email protected]> * update pipeline Signed-off-by: Piotr Pawłowski <[email protected]> * fix multiple addons usage Signed-off-by: Piotr Pawłowski <[email protected]> * fix multiple addons usage Signed-off-by: Piotr Pawłowski <[email protected]> * fix list Signed-off-by: Piotr Pawłowski <[email protected]> * += -> append Signed-off-by: Piotr Pawłowski <[email protected]> * comment filestore Signed-off-by: Piotr Pawłowski <[email protected]> * add new storage Signed-off-by: Piotr Pawłowski <[email protected]> * comment filestore Signed-off-by: Piotr Pawłowski <[email protected]> * rm obsolete steps Signed-off-by: Piotr Pawłowski <[email protected]> * create unique cluster and dont rm it Signed-off-by: Piotr Pawłowski <[email protected]> * move filestore to separate job Signed-off-by: Piotr Pawłowski <[email protected]> * fix networks Signed-off-by: Piotr Pawłowski <[email protected]> * add new group Signed-off-by: Piotr Pawłowski <[email protected]> * auth docker Signed-off-by: Piotr Pawłowski <[email protected]> * fix cluster name Signed-off-by: Piotr Pawłowski <[email protected]> * fix Signed-off-by: Piotr Pawłowski <[email protected]> * remove cluster removal Signed-off-by: Piotr Pawłowski <[email protected]> * introduce envs with workload names Signed-off-by: Piotr Pawłowski <[email protected]> * fix newline Signed-off-by: Piotr Pawłowski <[email protected]> * fix envs Signed-off-by: Piotr Pawłowski <[email protected]> * Apply manifest file (#216) * create filestore instance Signed-off-by: Piotr Pawłowski <[email protected]> * add enable filestore Signed-off-by: Piotr Pawłowski <[email protected]> * apply manifest file Signed-off-by: Piotr Pawłowski <[email protected]> * fix builds Signed-off-by: Piotr Pawłowski <[email protected]> --------- Signed-off-by: Piotr Pawłowski <[email protected]> * review fixes Signed-off-by: Piotr Pawłowski <[email protected]> * fix typo Signed-off-by: Piotr Pawłowski <[email protected]> * fix string interpolation Signed-off-by: Piotr Pawłowski <[email protected]> * add gcs fuse manifest example * change example namespace to default * resolve comments * cleanup --------- Signed-off-by: Piotr Pawłowski <[email protected]> Co-authored-by: Piotr Pawłowski <[email protected]> Co-authored-by: Patryk Bundyra <[email protected]>

Signed-off-by: Piotr Pawłowski <[email protected]>

…er/xpk into ppawl-cherry-pick-filestore

Signed-off-by: Piotr Pawłowski <[email protected]>

…pawl-cherry-pick

Signed-off-by: Piotr Pawłowski <[email protected]>

…er/xpk into ppawl-cherry-pick

Signed-off-by: Piotr Pawłowski <[email protected]>

…er/xpk into ppawl-cherry-pick-filestore

Signed-off-by: Piotr Pawłowski <[email protected]>

…o ppawl-cherry-pick-filestore

Signed-off-by: Piotr Pawłowski <[email protected]>

sharabiani and others added 16 commits January 10, 2025 14:36

Enabling Workload Identity and GCSFuse driver flags added. (#177)

718d8de

* Enabling Workload Identity and GCSFuse driver flags added.

fix

b8c419a

Signed-off-by: Piotr Pawłowski <[email protected]>

remove storage from pytype check

a7aaef6

Signed-off-by: Piotr Pawłowski <[email protected]>

Add GCS Fuse Storage to workload creation (#199)

7be30bc

* Add Storage to workload creation * Improve documentation

Fix types (#219)

0b9ef41

Fix get storage function (#221)

704dcad

Add xpk/templates as a package to pyproject (#223)

5f7ff69

Merge branch 'main' of https://github.com/AI-Hypercomputer/xpk into p…

8c73ccb

…pawl-cherry-pick

fix to make xpk run

ea2bbab

Signed-off-by: Piotr Pawłowski <[email protected]>

Fix tests on development branch (#263)

10816ca

* apply pathways fixes Signed-off-by: Piotr Pawłowski <[email protected]> --------- Signed-off-by: Piotr Pawłowski <[email protected]>

remove duplicate env in tests

3f4b4d7

Signed-off-by: Piotr Pawłowski <[email protected]>

Enable GCP Filestore on cluster (#215)

2bd1924

* Add flags to support GCP Filestore on cluster Signed-off-by: Piotr Pawłowski <[email protected]> --------- Signed-off-by: Piotr Pawłowski <[email protected]>

Use filestore in workload (#218)

fae0734

* enable to use filestore in workload Signed-off-by: Piotr Pawłowski <[email protected]> --------- Signed-off-by: Piotr Pawłowski <[email protected]> Co-authored-by: Patryk Bundyra <[email protected]>

Fix storage example typo (#275)

05a6e58

pawloch00 changed the base branch from main to ppawl-cherry-pick January 10, 2025 15:46

pawloch00 added 7 commits January 10, 2025 16:00

fix cluster creation in tests

6449df9

Signed-off-by: Piotr Pawłowski <[email protected]>

debug

523fa8d

Signed-off-by: Piotr Pawłowski <[email protected]>

add subnetwork

e0f8820

Signed-off-by: Piotr Pawłowski <[email protected]>

fix gcs removal

83d9b90

Signed-off-by: Piotr Pawłowski <[email protected]>

Merge branch 'ppawl-cherry-pick' of https://github.com/AI-Hypercomput…

ee61379

…er/xpk into ppawl-cherry-pick-filestore

install with make in filestore test

42fcc6c

Signed-off-by: Piotr Pawłowski <[email protected]>

uncomment integration and unit tests

98424a2

Signed-off-by: Piotr Pawłowski <[email protected]>

pawloch00 marked this pull request as ready for review January 13, 2025 11:47

pawloch00 requested review from Obliviour, 44past4, sharabiani and BluValor as code owners January 13, 2025 11:47

Merge branch 'main' of https://github.com/AI-Hypercomputer/xpk into p…

9a2a8c4

…pawl-cherry-pick

pawloch00 added 8 commits February 4, 2025 14:43

debug

b60e16c

Signed-off-by: Piotr Pawłowski <[email protected]>

debug

87ebbaa

Signed-off-by: Piotr Pawłowski <[email protected]>

restore all tests

fd9028e

Signed-off-by: Piotr Pawłowski <[email protected]>

Merge branch 'develop' into ppawl-cherry-pick

9fa0093

debug

f635be2

Signed-off-by: Piotr Pawłowski <[email protected]>

Merge branch 'ppawl-cherry-pick' of https://github.com/AI-Hypercomput…

0114792

…er/xpk into ppawl-cherry-pick

restore full tests

7d94496

Signed-off-by: Piotr Pawłowski <[email protected]>

Merge branch 'ppawl-cherry-pick' of https://github.com/AI-Hypercomput…

bd9f09f

…er/xpk into ppawl-cherry-pick-filestore

pawloch00 requested a review from gcie as a code owner February 5, 2025 09:16

pawloch00 added 3 commits February 5, 2025 09:18

fix kubectl dependency

21a3980

Signed-off-by: Piotr Pawłowski <[email protected]>

use makefile

6bb7eb0

Signed-off-by: Piotr Pawłowski <[email protected]>

fix

7c7655a

Signed-off-by: Piotr Pawłowski <[email protected]>

pawloch00 added the release-improvements label Feb 5, 2025

Base automatically changed from ppawl-cherry-pick to develop February 7, 2025 10:32

pawloch00 added 14 commits February 7, 2025 10:40

Merge branch 'develop' of https://github.com/AI-Hypercomputer/xpk int…

07cb3bf

…o ppawl-cherry-pick-filestore

fix README

efc9814

Signed-off-by: Piotr Pawłowski <[email protected]>

rename command

79e5191

fix build tests

27b145d

Signed-off-by: Piotr Pawłowski <[email protected]>

remove unused

4bf1999

Signed-off-by: Piotr Pawłowski <[email protected]>

use filestore isntance from network

28efc60

Signed-off-by: Piotr Pawłowski <[email protected]>

fix custom args

d61eda0

Signed-off-by: Piotr Pawłowski <[email protected]>

debug

732b62f

Signed-off-by: Piotr Pawłowski <[email protected]>

fix

1ac34ba

Signed-off-by: Piotr Pawłowski <[email protected]>

fix

f584580

Signed-off-by: Piotr Pawłowski <[email protected]>

fix

4c5564d

Signed-off-by: Piotr Pawłowski <[email protected]>

add network

c02db4b

Signed-off-by: Piotr Pawłowski <[email protected]>

fix typo

ad23cc1

Signed-off-by: Piotr Pawłowski <[email protected]>

fix tests to work with subnetwork

b210e49

Signed-off-by: Piotr Pawłowski <[email protected]>

pawloch00 changed the title ~~Merge filestore usage feature into main~~ Merge filestore usage feature into develop Feb 7, 2025

pawloch00 marked this pull request as draft February 7, 2025 17:21

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Merge filestore usage feature into develop #319

Merge filestore usage feature into develop #319

pawloch00 commented Jan 10, 2025 •

edited

Loading

Merge filestore usage feature into develop #319

Are you sure you want to change the base?

Merge filestore usage feature into develop #319

Conversation

pawloch00 commented Jan 10, 2025 • edited Loading

Fixes / Features

Testing / Documentation

pawloch00 commented Jan 10, 2025 •

edited

Loading