Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improvements to PaaSTA Playground #3727

Merged
merged 4 commits into from
Oct 25, 2023
+34 −6
Merged

Improvements to PaaSTA Playground #3727

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
bbd937f
added paasta-crons make target
cuza Oct 23, 2023
3cd613b
add VSCode debugpy as dev requirement
cuza Oct 23, 2023
05474e8
Making paasta playground not override existing confs
cuza Oct 23, 2023
cd30a4c
Update requirements-dev-minimal.txt
cuza Oct 23, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ general_itests/fake_etc_paasta/clusters.json
pip-wheel-metadata
debian/debhelper-build-stamp
unique-run
.vault-token

# Coverage artifacts
.coverage
Expand Down
20 changes: 20 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -158,13 +158,33 @@ setup-kubernetes-job: k8s_fake_cluster generate_deployments_for_service
export PAASTA_TEST_CLUSTER=kind-${USER}-k8s-test;\
.tox/py38-linux/bin/python -m paasta_tools.list_kubernetes_service_instances -d ./soa_config_playground --shuffle --group-lines 1 | xargs --no-run-if-empty .tox/py38-linux/bin/python -m paasta_tools.setup_kubernetes_job -d ./soa_config_playground -c kind-${USER}-k8s-test

.PHONY: cleanup-kubernetes-jobs
cleanup-kubernetes-jobs:
export KUBECONFIG=./k8s_itests/kubeconfig;\
export PAASTA_SYSTEM_CONFIG_DIR=./etc_paasta_playground/;\
export PAASTA_TEST_CLUSTER=kind-${USER}-k8s-test;\
.tox/py38-linux/bin/python -m paasta_tools.cleanup_kubernetes_jobs -d ./soa_config_playground -c kind-${USER}-k8s-test --force

.PHONY: paasta-secrets-sync
paasta-secrets-sync: setup-kubernetes-job .vault-token
export KUBECONFIG=./k8s_itests/kubeconfig;\
export PAASTA_SYSTEM_CONFIG_DIR=./etc_paasta_playground/;\
export PAASTA_TEST_CLUSTER=kind-${USER}-k8s-test;\
{ .tox/py38-linux/bin/python -m paasta_tools.list_kubernetes_service_instances -d ./soa_config_playground ; echo -n \ _shared; } | cut -f1 -d"." | uniq | shuf | xargs .tox/py38-linux/bin/python -m paasta_tools.kubernetes.bin.paasta_secrets_sync -v -d ./soa_config_playground -t ./.vault-token

define ANNOUNCE_CRONS_BODY
The following PaaSTA cron jobs will run on an infinite loop using the PaaSTA Playground k8s cluster:
- setup-kubernetes-job
- cleanup-kubernetes-job
- paasta-secrets-sync
- generate_deployments_for_service
endef
export ANNOUNCE_CRONS_BODY
.PHONY: paasta-crons
make paasta-cronjobs:
@echo "$$ANNOUNCE_CRONS_BODY"
while true; do make paasta-secrets-sync && make cleanup-kubernetes-jobs; sleep 5; done
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, should we be a little nicer to our devboxes and run these every minute?

and just to verify that my eyes are reading this correctly: we're running this in the foreground so upon ^C we'll stop this loop?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is running every 5s but also is kinda lightweight since is running just using the PaaSTA playground stuff, unless you made your playground super heavy with lots of test svcs it should be fine. I also tried to avoid wait as much as possible, I don't want to wait 1m to see what happened with my yelpsoa-config change when I'm doing some rapid testing


.vault-token:
export VAULT_ADDR=https://vault-devc.yelpcorp.com:8200 ;\
export VAULT_SKIP_VERIFY=true ;\
Expand Down
1 change: 1 addition & 0 deletions paasta_tools/contrib/create_paasta_playground.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ def main():
src="./k8s_itests/deployments/paasta/fake_soa_config",
dst="soa_config_playground",
values=values_path,
overwrite=False,
)


Expand Down
14 changes: 8 additions & 6 deletions paasta_tools/contrib/render_template.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,23 +31,25 @@ def render_file(src, dst, values):
new.write(replace(old.read(), values))


def render(src, dst, values={}, exclude={}):
def render(src, dst, values={}, exclude={}, overwrite=True):
if os.path.isfile(src):
render_file(src, dst, values)
if overwrite:
render_file(src, dst, values)
return
for f in os.scandir(src):
if f.name.startswith(".") or f.path in exclude:
continue
if os.path.isfile(f.path):
render_file(f.path, dst, values)
if overwrite:
render_file(f.path, dst, values)
else:
new_dst = replace(f"{dst}/{f.name}", values)
try:
os.makedirs(new_dst, exist_ok=True)
except OSError as e:
if e.errno != os.errno.EEXIST:
raise
render(f.path, new_dst, values, exclude)
render(f.path, new_dst, values, exclude, overwrite)


def parse_args():
Expand Down Expand Up @@ -82,7 +84,7 @@ def parse_args():
return args


def render_values(src: str, dst: str, values: str) -> None:
def render_values(src: str, dst: str, values: str, overwrite=True) -> None:
if values is not None:
values = os.path.abspath(values)
# Validate src and values. Dst needs to be a directory. src can be either a valid folder of directory. values need to be valid file if provided.
Expand All @@ -108,7 +110,7 @@ def render_values(src: str, dst: str, values: str) -> None:
),
v,
)
render(src, dst, config_dict, {values})
render(src, dst, config_dict, {values}, overwrite)


def main():
Expand Down
3 changes: 3 additions & 0 deletions requirements-dev-minimal.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
astroid
asynctest
coverage
# VSCode debugging requirement
# See https://code.visualstudio.com/docs/python/debugging#_local-script-debugging
debugpy
cuza marked this conversation as resolved.
Show resolved Hide resolved
docutils
flake8
freezegun
Expand Down
1 change: 1 addition & 0 deletions requirements-dev.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ asynctest==0.12.0
Babel==2.9.1
cfgv==2.0.1
coverage==6.5.0
debugpy==1.8.0
distlib==0.3.4
exceptiongroup==1.1.2
filelock==3.0.12
Expand Down