Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move CSI attach detach test and rewrite script in Python #412

Open
wants to merge 30 commits into
base: main
Choose a base branch
from

Conversation

alyssa1303
Copy link
Collaborator

@alyssa1303 alyssa1303 commented Nov 26, 2024

Summary

  • Rewrite CSI attach detach script in Python with some adjustment to schema
  • Move CSI attach detach 300 and 1000 pipelines
  • Adjust storageclass creation in AWS to add all required tags
  • Re-use K8s client module in CL2 for this test, but add a TODO to move it to a separate folder

@alyssa1303 alyssa1303 force-pushed the alyssa/move-csi-test branch 2 times, most recently from 101c9a7 to 10e0296 Compare November 27, 2024 15:39
@alyssa1303 alyssa1303 marked this pull request as ready for review November 27, 2024 18:31
@alyssa1303 alyssa1303 changed the title Move CSI test and rewrite attach detach script Move CSI attach detach test and rewrite script in Python Nov 27, 2024
Copy link
Contributor

@rafael-mendes-pereira rafael-mendes-pereira left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please, add unit tests for the functions at csi.py

modules/python/csi/csi.py Show resolved Hide resolved
p99 = disk_number * 99 // 100
return p50, p90, p99, disk_number

def create_statefulset(namespace, replicas, storage_class):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this could be inside of KubernetesClient. Therefore, we can re-use it other scenarios

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm debating this because lots of spec define here are solely used in this test so not sure if it's worth putting in KubernetesClient. That's why I have a get_app_client() so at least I can re-use the client

Comment on lines +83 to +99
def log_duration(description, start_time, log_file):
"""Log the time duration of an operation."""
end_time = datetime.now()
duration = int((end_time - start_time).total_seconds())
with open(log_file, "a") as f:
f.write(f"{description}: {duration}\n")
print(f"{description}: {duration}s")

def wait_for_condition(check_function, target, comparison="gte", interval=1):
"""Wait for a condition using a given check function."""
while True:
current_list = check_function()
current = len(current_list)
print(f"Current: {current}, Target: {target}")
if (comparison == "gte" and current >= target) or (comparison == "lte" and current <= target):
return current
time.sleep(interval)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. These functions could be in some utils file
  2. Add to the wait_for_condition description what the check_function should return

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean util file under csi folder? I don't think these are useful for other folders like clusterloader2

Copy link
Contributor

@rafael-mendes-pereira rafael-mendes-pereira Nov 28, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They might still be useful for other tests in the future. Note: this is not critical. Feel free to address it in another PR.

modules/python/clusterloader2/kubernetes_client.py Outdated Show resolved Hide resolved
modules/python/csi/csi.py Show resolved Hide resolved
end_time = datetime.now()
duration = int((end_time - start_time).total_seconds())
with open(log_file, "a") as f:
f.write(f"{description}: {duration}\n")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets put quotes ("") in the key and value or add a validation to check if the description does not contains :, what would break the parser

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The description and duration are passed inside the test only though. I don't think that check is necessary

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but someone can change the description in the future and don't notice that it breaks this logic

modules/python/tests/test_kubernetes_client.py Outdated Show resolved Hide resolved
modules/python/tests/test_kubernetes_client.py Outdated Show resolved Hide resolved
modules/python/tests/test_kubernetes_client.py Outdated Show resolved Hide resolved

ns = self.client.create_namespace(name)
self.assertEqual(ns.metadata.name, mock_read_namespace.return_value.metadata.name)
mock_read_namespace.assert_called_once_with(name)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of it, assert that the "kubernetes.client.CoreV1Api.create_namespace" was not called

Comment on lines +138 to +139
self.assertEqual(len(returned_pods), len(expected_pods))
self.assertEqual(returned_pods, expected_pods)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

minor: you can replace those with only self.assertCountEqual(expected_pods, returned_pods)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants