Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Salt upgrade #129

Open
wants to merge 18 commits into
base: integration
Choose a base branch
from
Open

Salt upgrade #129

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
457c3d7
updated lambdas to use python 3.11
sandyhider Oct 28, 2024
bb41230
updated submodules
sandyhider Nov 4, 2024
ff1ae7e
adds updated salt files and packer file
dxenes1 Nov 20, 2024
d82bac0
adds uwsgi as a system library
dxenes1 Nov 21, 2024
ebed9dc
add uwsgi to prereq libs
dxenes1 Nov 21, 2024
61d8d11
adds build packages, uwsgi package install
dxenes1 Nov 21, 2024
67e75e0
updates salt files for activities
dxenes1 Nov 22, 2024
e5c645c
updates pip_env for all pip installs
dxenes1 Dec 4, 2024
75535c3
fixes packer script bug and removes setuptools upgrade
dxenes1 Dec 4, 2024
d934ff0
removes editable install for heaviside
dxenes1 Dec 5, 2024
73a0ac3
updates bootstrap script to latest version
dxenes1 Dec 5, 2024
0aed7a3
updated submodules
sandyhider Dec 10, 2024
9c56ec8
updated lambdas to use python 3.11
sandyhider Oct 28, 2024
0ce3a09
updated submodules
sandyhider Dec 10, 2024
6078c38
Merge branch 'py311' of https://github.com/jhuapl-boss/boss-manage in…
sandyhider Dec 10, 2024
b64fea2
Merge pull request #130 from jhuapl-boss/py311
sandyhider Dec 12, 2024
eb15c58
added changes choose the rds engine version
sandyhider Dec 16, 2024
a906da1
updated api and core so neither run post-init automatically after cre…
sandyhider Jan 30, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion bin/packer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,7 @@ def contains(x, ys):

images = response['Images']
#images = [i for i in images if contains(i['Name'], ('hvm-ssd', '14.04', 'server'))]
images = [i for i in images if contains(i['Name'], ('hvm-ssd', '20.04', 'server'))]
images = [i for i in images if contains(i['Name'], ('hvm-ssd', '24.04', 'server'))]
images = [i for i in images if not contains(i['Name'], ('ubuntu-eks'))]
images = [i for i in images if not contains(i['Name'], ('ubuntu-pro-server'))]
images.sort(key=lambda x: x["CreationDate"], reverse=True)
Expand Down
2 changes: 1 addition & 1 deletion lib/heaviside.git
Open in desktop
Submodule heaviside.git updated 11 files
+20 −2 .circleci/config.yml
+7 −0 CHANGELOG.md
+1 −1 README.md
+0 −207 bin/heaviside
+59 −56 heaviside/__init__.py
+224 −0 heaviside/__main__.py
+9 −2 heaviside/utils.py
+64 −0 pyproject.toml
+1 −1 requirements.txt
+0 −5 setup.cfg
+0 −72 setup.py
4 changes: 2 additions & 2 deletions packer/vm.pkr.hcl
View file
Open in desktop
Copy link
Member

@sandyhider sandyhider Dec 4, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am wondering if we should be using 3006. This is what ChatGPT says about versions:

SaltStack adheres to a [CalVer](https://calver.org/) versioning scheme, where:

Versions are formatted as YYYY.MM (e.g., 3006 corresponds to a release in 2023, June).
Odd-numbered versions (e.g., 3007) are feature releases, introducing new functionality.
Even-numbered versions (e.g., 3006) are LTS (Long-Term Support) releases, focusing on stability and bug fixes.
Key Differences
Purpose of Releases:

Odd Versions (Feature Releases):
Introduce new features and enhancements.
Serve as a preview for the next stable LTS release.
May include experimental features or breaking changes.
Typically have shorter support lifespans.
Even Versions (LTS Releases):
Focus on stability, reliability, and performance.
Incorporate features introduced in the previous odd release, but fully tested and refined.
Receive long-term updates for security and critical bug fixes.
Stability:

LTS versions (even) are better suited for production environments where stability is critical.
Feature releases (odd) are more appropriate for development, testing, or environments where early adoption of new features is desired.
Support Lifecycle:

Odd-numbered versions have a shorter support window, typically ending shortly after the release of the next LTS version.
Even-numbered (LTS) versions are supported for an extended period, ensuring long-term stability for enterprise deployments.```

What do you think?

Original file line number Diff line number Diff line change
Expand Up @@ -114,8 +114,8 @@ build {
provisioner "shell" {
inline = ["sudo tar -C /srv -xf /tmp/salt_files.tar.gz",
"sudo chmod +x /tmp/bootstrap-salt.sh",
"echo sudo /tmp/bootstrap-salt.sh -i ${var.name} stable 3004",
"sudo /tmp/bootstrap-salt.sh -i ${var.name} stable 3004",
"echo sudo /tmp/bootstrap-salt.sh -i ${var.name} stable 3007",
"sudo /tmp/bootstrap-salt.sh -i ${var.name} stable 3007",
"echo sudo salt-call --local state.highstate --file-root=/srv/salt --pillar-root=/srv/pillar --id=${var.name}",
"sudo salt-call --local state.highstate --file-root=/srv/salt --pillar-root=/srv/pillar --id=${var.name}",
"echo deleting temp files from /tmp",
Expand Down
4,877 changes: 2,336 additions & 2,541 deletions salt_stack/salt-bootstrap/bootstrap-salt.sh
View file
Open in desktop

Large diffs are not rendered by default.

2 changes: 1 addition & 1 deletion salt_stack/salt/aws/boto3.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ include:
boto3:
pip.installed:
- name: boto3>=1.0,<2.0
#- bin_env: /usr/local/bin/pip3
- pip_bin: /usr/bin/pip3
- require:
- sls: python.python3
- sls: python.pip3
2 changes: 2 additions & 0 deletions salt_stack/salt/boss-tools/activities.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@ include:

activity-update-pip:
pip.installed:
- pip_bin: /usr/bin/pip3
- name: pip
- upgrade: True

Expand All @@ -24,6 +25,7 @@ activity-files:

activites-lib:
pip.installed:
- pip_bin: /usr/bin/pip3
- name: pymysql
- require:
- sls: python.python3
Expand Down
1 change: 1 addition & 0 deletions salt_stack/salt/boss-tools/bossutils.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ include:

python-lib:
pip.installed:
- pip_bin: /usr/bin/pip3
# DP HACK: Cannot use salt:// with pip.installed, so assume the base directory
- name: /srv/salt/boss-tools/files/boss-tools.git/bossutils/
- require:
Expand Down
3 changes: 3 additions & 0 deletions salt_stack/salt/boss/django.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,16 +13,19 @@ django-update-pip:
pip.installed:
- name: pip
- upgrade: True
- pip_bin: /usr/bin/pip3

django-prerequirements:
pkg.installed:
- pkgs:
- libmysqlclient-dev
- libffi-dev
- awscli
- patchelf

django-requirements:
pip.installed:
- pip_bin: /usr/bin/pip3
- requirements: salt://boss/files/boss.git/requirements.txt
- exists_action: w
- require:
Expand Down
2 changes: 1 addition & 1 deletion salt_stack/salt/boss/files/boss.git
Open in desktop
Submodule boss.git updated 5 files
+31 −0 django/bosstiles/renderers.py
+44 −0 django/bosstiles/test/int_test_tiles_view_uint8.py
+61 −0 django/bosstiles/test/tiles_view_uint8.py
+24 −11 django/bosstiles/views.py
+0 −1 requirements.txt
3 changes: 3 additions & 0 deletions salt_stack/salt/cvdb/init.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,17 +4,20 @@ include:

cvdb-update-pip:
pip.installed:
- pip_bin: /usr/bin/pip3
- name: pip
- upgrade: True

cvdb-lib:
pip.installed:
- pip_bin: /usr/bin/pip3
- name: /srv/salt/cvdb/files/cvdb.git/
- require:
- sls: python.python3

cvdb-test-requirements:
pip.installed:
- pip_bin: /usr/bin/pip3
- requirements: salt://cvdb/files/cvdb.git/requirements-test.txt
- exists_action: w
- require:
Expand Down
1 change: 1 addition & 0 deletions salt_stack/salt/django/rest-framework.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ include:

rest_framework:
pip.installed:
- pip_bin: /usr/bin/pip3
- name: djangorestframework==3.3.1
- bin_env: /usr/local/bin/pip3
- require:
Expand Down
1 change: 1 addition & 0 deletions salt_stack/salt/heaviside/init.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ include:

heaviside:
pip.installed: # pip dependencies not resolving to our version
- pip_bin: /usr/bin/pip3
- editable: git+https://github.com/jhuapl-boss/heaviside.git#egg=heaviside
- exists_action: w

13 changes: 13 additions & 0 deletions salt_stack/salt/ingest-client/ingest.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ ingest-update-pip:
pip.installed:
- name: pip
- upgrade: True
- pip_bin: /usr/bin/pip3

ingest-prerequirements:
pkg.installed:
Expand All @@ -27,14 +28,26 @@ ingest-prerequirements:
# export LC_ALL=en_US.UTF-8
# sudo /usr/local/bin/pip3 install httpretty==0.8.10

ingest-wheel-upgrade:
pip.installed:
- pip_bin: /usr/bin/pip3
- pkgs:
- wheel
- setuptools
- upgrade: True
- require:
- sls: python.python3

# Need to install pyyaml separatly to avoid problems with other requirements
ingest-pyyaml:
pip.installed:
- pip_bin: /usr/bin/pip3
- name: pyyaml
- ignore_installed: True

ingest-client-lib:
pip.installed:
- pip_bin: /usr/bin/pip3
- requirements: salt://ingest-client/files/ingest-client.git/requirements.txt
- require:
- pkg: ingest-prerequirements
Expand Down
2 changes: 1 addition & 1 deletion salt_stack/salt/jenkins-microns/init.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,5 +17,5 @@ include:
## TODO: make pip installation part of the Jenkins job, instead.
intern-sdk:
pip.installed:
- bin_env: /usr/local/bin/pip3
- pip_bin: /usr/bin/pip3
- requirements: salt://jenkins-microns/files/intern-requirements.txt
2 changes: 1 addition & 1 deletion salt_stack/salt/jenkins/django.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ include:

django-jenkins:
pip.installed:
- pip_bin: /usr/bin/pip3
- name: django-jenkins==0.18.1
- bin_env: /usr/local/bin/pip3
- require:
- sls: python.python35
- sls: python.pip
13 changes: 13 additions & 0 deletions salt_stack/salt/ndingest/init.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,23 @@
# Install ndingest into site-packages.
include:
- python.python3
- python.pip3
- python.python3-dev
- spdb

ndingest-wheel-upgrade:
pip.installed:
- pip_bin: /usr/bin/pip3
- pkgs:
- wheel
- setuptools
- upgrade: True
- require:
- sls: python.python3

ndingest-lib:
pip.installed:
- pip_bin: /usr/bin/pip3
# DP HACK: Cannot use salt:// with pip.installed, so assume the base directory
- name: /srv/salt/ndingest/files/ndingest.git/
- require:
Expand Down
2 changes: 1 addition & 1 deletion salt_stack/salt/python/coverage.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ include:
coverage:
pip.installed:
- name: coverage==4.0.3
- bin_env: /usr/local/bin/pip3
- pip_bin: /usr/bin/pip3
- require:
- sls: python.python35
- sls: python.pip
2 changes: 1 addition & 1 deletion salt_stack/salt/python/jsonschema.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ include:

jsonschema:
pip.installed:
- pip_bin: /usr/bin/pip3
- name: jsonschema==2.5.1
- bin_env: /usr/local/bin/pip3
- require:
- sls: python.python35
- sls: python.pip
2 changes: 1 addition & 1 deletion salt_stack/salt/python/nose2-3.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,8 @@ include:

python3-nose2:
pip.installed:
- pip_bin: /usr/bin/pip3
- name: {{ python3.nose2_pkg }}
- bin_env: /usr/local/bin/pip3
- require:
# Currently require pip from Python 2.x for Salt's pip state.
- sls: python.pip
Expand Down
2 changes: 1 addition & 1 deletion salt_stack/salt/python/nose2-cov-3.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ include:
python3-nose2-cov:
pip.installed:
- name: {{ python3.nose2_cov_pkg }}
- bin_env: /usr/local/bin/pip3
- pip_bin: /usr/bin/pip3
- require:
# Currently require pip from Python 2.x for Salt's pip state.
- sls: python.pip
Expand Down
1 change: 1 addition & 0 deletions salt_stack/salt/python/nose2-cov.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,6 @@ include:

pip.installed:
- name: {{ python2.nose2_cov_pkg }}
- pip_bin: /usr/bin/pip3
- require:
- sls: python.pip
1 change: 1 addition & 0 deletions salt_stack/salt/python/nose2.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,5 +6,6 @@ include:
python2-nose2:
pip.installed:
- name: {{ python2.nose2_pkg }}
- pip_bin: /usr/bin/pip3
- require:
- sls: python.pip
2 changes: 1 addition & 1 deletion salt_stack/salt/python/pep8.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ include:
pep8:
pip.installed:
- name: pep8==1.7.0
- bin_env: /usr/local/bin/pip3
- pip_bin: /usr/bin/pip3
- require:
- sls: python.python35
- sls: python.pip
2 changes: 1 addition & 1 deletion salt_stack/salt/python/pylint-django.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ include:
pylint-django:
pip.installed:
- name: pylint-django==0.7.1
- bin_env: /usr/local/bin/pip3
- pip_bin: /usr/bin/pip3
- require:
- sls: python.python35
- sls: python.pip
Expand Down
2 changes: 1 addition & 1 deletion salt_stack/salt/python/pylint.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ include:
pylint:
pip.installed:
- name: pylint==1.5.4
- bin_env: /usr/local/bin/pip3
- pip_bin: /usr/bin/pip3
- require:
- sls: python.python35
- sls: python.pip
2 changes: 1 addition & 1 deletion salt_stack/salt/python/pyminifier.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ include:
pyminifier:
pip.installed:
- name: pyminifier==2.1
- bin_env: /usr/local/bin/pip3
- pip_bin: /usr/bin/pip3
- require:
- sls: python.python35
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These v3.5 requirements aren't causing problems?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah I missed that require, looks like the pyminifier salt file is only used in the "workstation" packer AMI though so it might not be executed when creating the core AMIs for the stack...Not sure what "workstation" is for? A developer environment I'm guessing?

    'workstation*':
        - python.python35
        - git
        - vault.client
        - aws.boto3
        - python.pyminifier

- sls: python.pip
2 changes: 1 addition & 1 deletion salt_stack/salt/python/virtualenvwrapper.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ include:
virtualenvwrapper:
pip.installed:
- name: virtualenvwrapper
- bin_env: /usr/local/bin/pip3
- pip_bin: /usr/bin/pip3
- require:
- sls: python.python35
- sls: python.pip
1 change: 1 addition & 0 deletions salt_stack/salt/scipy/init.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@ scipy-prerequirements:

scipy-lib:
pip.installed:
- pip_bin: /usr/bin/pip3
- pkgs:
- scipy
- require:
Expand Down
19 changes: 11 additions & 8 deletions salt_stack/salt/spdb/init.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,10 +1,13 @@
include:
- python.python3
- python.pip3
- python.python3-dev

spdb-update-pip:
pip.installed:
- name: pip
- upgrade: True
- pip_bin: /usr/bin/pip3

spdb-prerequirements:
pkg.installed:
Expand All @@ -15,6 +18,7 @@ spdb-prerequirements:
- libfreetype6-dev
- liblcms2-dev
- libwebp-dev
- patchelf
#- libopenjpeg-dev

# Install moto dependency separately. Salt sets LC_ALL=C which breaks
Expand All @@ -32,29 +36,28 @@ spdb-prerequirements:

# During testing with the Docker Ubuntu 20.04 image, wheel was too old to
# complete the spdb and blosc installs.
wheel-upgrade:
spdb-wheel-upgrade:
pip.installed:
- name: wheel
- pip_bin: /usr/bin/pip3
- pkgs:
- wheel
- setuptools
- upgrade: True
- require:
- sls: python.python3

spdb-lib:
pip.installed:
- pip_bin: /usr/bin/pip3
# DP HACK: Cannot use salt:// with pip.installed, so assume the base directory
- name: /srv/salt/spdb/files/spdb.git/
- require:
- pkg: spdb-prerequirements
- sls: python.python3

# Need to install pyyaml separatly to avoid problems with other requirements
spdb-pyyaml:
pip.installed:
- name: pyyaml
- ignore_installed: True

spdb-test-requirements:
pip.installed:
- pip_bin: /usr/bin/pip3
- requirements: salt://spdb/files/spdb.git/requirements-test.txt
- exists_action: w
- require:
Expand Down
11 changes: 5 additions & 6 deletions salt_stack/salt/uwsgi/emperor.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,11 @@ include:
- python.python3
- python.pip3

uwsgi:
pip.installed:
- name: uwsgi==2.0.19.1
- require:
- sls: python.python3
- sls: python.pip3
uwsgi-prerequirements:
pkg.installed:
- pkgs:
- uwsgi
- patchelf

uwsgi-init.d:
file.managed:
Expand Down
2 changes: 1 addition & 1 deletion salt_stack/salt/vault/client.sls
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ vault-lib:
pip.installed:
# ToDo: fix needs to be kept in sync with spdb's hvac requirement.
- name: hvac==0.11.2
#- bin_env: /usr/local/bin/pip3
- pip_bin: /usr/bin/pip3
- require:
- sls: python.python3
- sls: python.pip3