Merge pull request #160 from cfs-energy-internal/3.1.5

IDEA Release 3.1.5

CHANGELOG.md

@@ -5,6 +5,49 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
 
+## [3.1.5] - 2023-09-06
+
+### Features
+* Install XDummy driver for non-gpu Linux console eVDI sessions
+* Add command-line options (`--custom-permissions-boundary`, `--cloudformation-execution-policies`, `--public-access-block-configuration`) for `idea-admin.sh bootstrap` sub-command. These are passed to the underlying `cdk bootstrap` command as needed to support IAM permissions boundary and S3 bucket restrictions during the `cdk bootstrap` phase.
+  * Users may also need to customize `cdk.json` for the IDEA-created IAM roles to attach IAM permissions boundaries, depending on their specific AWS Account policy. See [this blog post](https://aws.amazon.com/blogs/devops/secure-cdk-deployments-with-iam-permission-boundaries/) for additional information regarding AWS CDK and IAM permission boundaries.
+* In Active Directory environments - added a cache for discovered Active Directory information (domain controller IP addresses)
+* Support added for [HPC7a](https://aws.amazon.com/ec2/instance-types/hpc7a/) and [P5](https://aws.amazon.com/ec2/instance-types/p5/) instance families
+* Added `delete-backups` sub-command for `idea-admin.sh`
+
+
+### Changes
+* Update AWS EFA Installer from `1.23.1` to `1.25.1`
+* Improve ability for IDEA SDK consumers (such as `scheduler` module) to support newly launched AWS EC2 instances on the same day of launch. This changes the authoritative source of instance type validity from the installed boto3 to the AWS API call `describe_instance_types()` and provides a caching method.
+* Adjust Project budget display (and other displays of currency) to be formatted with commas
+* Change default instance types to be `m6i` family from `m5` family in keeping with current generations of available instance families. Older family instances can still be updated in the generated YAML configuration files before `config update`.
+* Support egress IPv6 traffic in security groups where IPv4-traffic was permitted
+* Split apart VDC Broker Security group generation
+* Update OpenSearch default engine from `2.3` to `2.7`
+* AWS CDK `2.63.0` -> `2.93.0`
+* AWS CDK template updates from `v14` to `v18`
+* Set the FSx/Lustre filesystem version to `2.15` for newly created filesystems
+* Updated default scratch size to `60GiB` in the WebUI (when editing Queue profiles -> EBS Scratch Provider)
+* Misc 3rd party package updates
+
+### Bug Fixes
+* Fixed missing IAM permission for VDC Controller scheduled event transformer lambda when using a customer-managed KMS key for SQS. This would have impacted the ability for eVDI schedules to apply to sessions.
+* Fixed an issue that prevented `Red Hat Enterprise Linux 8.7` from launching VDI sessions on `G4ad` instances
+* Windows eVDI instances were not correctly tagging their underlying EBS volumes and Elastic Network Interfaces. This has been fixed (Linux eVDI was not impacted)
+* HPC Job DryRun functionality was not properly sending EBS Encryption settings. This could cause jobs to fail a DryRun in environments with Service control policies (SCPs) that required Encrypted EBS volumes. The actual job submission would work properly but the DryRun requirement was a gate that was required to be passed first.
+* Restore ability to have spaces in the Project `title`
+* During `GovCloud` installation - display a list of AWS profiles for the commercial profile versus requiring the user to type it in.
+* Correct a defect that was not allowing the selection of Tenancy Choices for a Software stack.
+* Improve eVDI subnet retry logic for both Capacity exceptions abd Unsupported Instance exceptions
+* OpenSearch domains that were not completely deployed were being listed during `idea-admin.sh config generate --existing-resources`
+* When generating a stack from a session - make sure to copy the minimum storage and projects from the session. This should restore the ability to create stacks from existing sessions.
+* Fixed WebUI modal for Session Sharing permissions appearing with a dark blue header no matter what the selected theme is
+* The example configuration displayed in the SSH Access screen was missing the `Hostname` (IP address) when the bastion was deployed in Private subnet scenarios
+* Misc PEP cleanups
+
+
+### Known Caveats
+
 ## [3.1.4] - 2023-07-25
 
 :heavy_exclamation_mark: - *Please note the IDEA ECR Repository location has changed as of `3.1.4`*
@@ -55,7 +98,7 @@ Users of older `idea-admin.sh` and `idea-admin-windows.ps1` may need to manually
 * Default Web/API page size increased from `20` to `50`
 * Update AMI IDs for all supported operating systems
 * Update AWS EFA Installer from `1.22.1` to `1.23.1`
-* Update DCV Server from `2023.0-14852` to `2023.0-15065`, DCV Session Manager from `2023.0-642` to `2023.0-675`, and DCV viewer from `2023.0.5388` to `2023.0.5483`
+* Update DCV Server from `2023.0-14852` to `2023.0-15065`, DCV Session Manager Agent from `2023.0.642` to `2023.0.675`, and DCV viewer from `2023.0.5388` to `2023.0.5483`
 * Reduce the default DCV idle disconnect from 24-hours to 4-hours
 * Update Nvidia drivers from `510.47.03` to `525.105.17`
 
@@ -83,7 +126,7 @@ Users of older `idea-admin.sh` and `idea-admin-windows.ps1` may need to manually
   * `EBS` customer-managed key needs the following service-roles to be added as key users: AWSServiceRoleForAutoScaling, AWSServiceRoleForEC2Fleet, AWSServiceRoleforEC2SpotFleet. Post IDEA cluster deployment, IDEA VDC Controller IAM role also needs to be added as a key user.
   * `SQS` customer-managed key needs customization to grant SNS service-principal access per https://docs.aws.amazon.com/sns/latest/dg/sns-enable-encryption-for-topic-sqs-queue-subscriptions.html
 * New options to control eVDI subnet use/selection can be found under `vdc.dcv_sessions.network`:
-  * Allow for eVDI subnets to differ from HPC/compute subnets in the configuration. By default the same subnets are configured. This can be changed on a running cluster without a restart.
+  * Allow for eVDI subnets to differ from HPC/compute subnets in the configuration. By default, the same subnets are configured. This can be changed on a running cluster without a restart.
   * Allow for `ordered`  or `random`  subnet selection during eVDI launching. Default subnet selection is `ordered` .
   * Allow for automatic retry of eVDI subnets during creating eVDI resources. Default is to `auto-retry` the next subnet. This may be disabled in situations to avoid cross-AZ charges with eVDI resources accessing resources in other AZs.
 * Allow the IDEA Administrator to define NICE DCV USB remotization devices that will apply to the eVDI fleet. USB filter strings can be added to `vdc.server.usb_remotization`  (list) for USB client-side devices to be enabled for USB remotization.
@@ -92,7 +135,7 @@ Users of older `idea-admin.sh` and `idea-admin-windows.ps1` may need to manually
 ### Changes
 
 * AWS EFA Installer updated from `1.22.0`  to `1.22.1`
-* Upgrade to NICE DCV `2023.0` where applicable.
+* Update DCV Server from `2022.1-13300` to `2023.0-14852`, DCV Session Manager Agent from `2022.1-592` to `2023.0-642`, DCV Connection Gateway from `2022.1.377` to `2023.0.531`, DCV Session Manager Broker from `2022.1.355` to `2023.0.392`, and DCV viewer from `2022.1.4251` to `2023.0.5388`
 * Changes to WebUI / notification icon - Password expiration warning will only appear at `<10days`.  Remove the default pip on the icon indicating a waiting notification.
 * eVDI hosts will now populate `/etc/environment`  with two additional environment variables that can be used by bootstrap scripting / post-boot customization. `IDEA_SESSION_OWNER`  and `IDEA_SESSION_ID` .
 * When submitting a job from the WebUI - the job name  will now default to the filename with the `.`  character replaced with `_`  as `.`  is not allowed in job names.
@@ -106,7 +149,7 @@ Users of older `idea-admin.sh` and `idea-admin-windows.ps1` may need to manually
 * The incorrect version number for IDEA was displayed in the Web console
 * eVDI sessions were launched with EBS volume encryption tied to the Hibernation setting
 * The download link for NICE DCV Session manager agent for Windows was incorrect
-* Log files are now encoded in `UTF-8` encoding. This allows for logging of eVDI session names with UTF-8/multi-byte characters. Previously this would cause a traceback.
+* Log files are now encoded in `UTF-8` encoding. This allows for logging of eVDI session names with UTF-8/multibyte characters. Previously this would cause a traceback.
 * A bug was preventing the cluster timezone from properly being detecting in some modules. The timezone would default to `America/Los_Angeles`  for some situations even when the `cluster.timezone`  was properly set. This would cause eVDI schedules to operate in `America/Los_Angeles`  instead of the cluster timezone.
 * Fixed a bug that prevented updates to DCV connection gateway certificate ARNs when private certificates are used
 * Fixed a bug that prevented SSH access to IDEA infrastructure instances when `CentOS7` is used
@@ -227,7 +270,7 @@ Users of older `idea-admin.sh` and `idea-admin-windows.ps1` may need to manually
 * IDEA Python updated to `3.9.16 `
 * Lustre client updated to `2.12`
 * Changes to integration-test infrastructure to leverage `IMDSv2`
-* Changes to integration-tests to cleanup any AWS Backups that are deployed and log more environment variables during the run
+* Changes to integration-tests to clean up any AWS Backups that are deployed and log more environment variables during the run
 * Connect anonymous metrics for telemetry information about the AWS Solution
 * Revised eVDI `allowlist/denylist` functionality to allow fine-grained control of instances. Supports both instance family and specific instance conventions.
 * Build python in the bootstrap directory vs. `/tmp`  - this is more compatible with AMIs that have `noexec`  mount policy for `/tmp` .

IDEA_VERSION.txt

@@ -1 +1 @@
-3.1.4
+3.1.5

idea-admin-windows.ps1

@@ -38,7 +38,7 @@ function Verify-Command($type,$message,$command) {
 $IDEADevMode = if ($Env:IDEA_DEV_MODE) {$Env:IDEA_DEV_MODE} else {""}
 $VirtualEnv = if ($Env:VIRTUAL_ENV) {$Env:VIRTUAL_ENV} else {""}
 $ScriptDir = $PSScriptRoot
-$IDEARevision = if ($Env:IDEA_REVISION) {$Env:IDEA_REVISION} else {"v3.1.4"}
+$IDEARevision = if ($Env:IDEA_REVISION) {$Env:IDEA_REVISION} else {"v3.1.5"}
 $IDEADockerRepo = "public.ecr.aws/h5i3y8y1"
 $DocumentationError = "https://ide-on-aws.com"
 $AWSProfile = if ($Env:AWS_PROFILE) {$Env:AWS_PROFILE} else {"default"}

idea-admin.sh

@@ -28,7 +28,7 @@
 # * IDEA_DEV_MODE - Set to "true" if you are working with IDEA sources
 
 SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
-IDEA_REVISION=${IDEA_REVISION:-"v3.1.4"}
+IDEA_REVISION=${IDEA_REVISION:-"v3.1.5"}
 IDEA_DOCKER_REPO=${IDEA_DOCKER_REPO:-"public.ecr.aws/h5i3y8y1/idea-administrator"}
 IDEA_ECR_CREDS_RESET=${IDEA_ECR_CREDS_RESET:-"true"}
 IDEA_ADMIN_AWS_CREDENTIAL_PROVIDER=${IDEA_ADMIN_AWS_CREDENTIAL_PROVIDER:=""}
@@ -119,6 +119,7 @@ fi
 
 # Launch installer
 ${DOCKER_BIN} run --rm -it -v "${HOME}/.idea/clusters:/root/.idea/clusters" \
+              -e AWS_SESSION_TOKEN -e AWS_ACCESS_KEY_ID -e AWS_SECRET_ACCESS_KEY \
               -e IDEA_ADMIN_AWS_CREDENTIAL_PROVIDER="${IDEA_ADMIN_AWS_CREDENTIAL_PROVIDER}" \
               -e IDEA_ADMIN_ENABLE_CDK_NAG_SCAN="${IDEA_ADMIN_ENABLE_CDK_NAG_SCAN}" \
               -v ~/.aws:/root/.aws "${IDEA_DOCKER_REPO}:${IDEA_REVISION}" \

requirements/dev.txt

@@ -1,129 +1,133 @@
-aiofiles==0.8.0
-alembic==1.7.7
-arrow==1.2.1
-astroid==2.12.11
-attrs==21.4.0
-aws-cdk-asset-awscli-v1==2.2.52
-aws-cdk-asset-kubectl-v20==2.1.1
-aws-cdk-asset-node-proxy-agent-v5==2.0.42
-aws-cdk-lib==2.63.0
+aiofiles==23.1.0
+alembic==1.11.1
+arrow==1.2.3
+astroid==2.15.6
+attrs==23.1.0
+aws-cdk-asset-awscli-v1==2.2.200
+aws-cdk-asset-kubectl-v20==2.1.2
+aws-cdk-asset-node-proxy-agent-v6==2.0.1
+aws-cdk-lib==2.93.0
 banal==1.0.6
-blinker==1.4
-boto3==1.26.138
-botocore==1.29.138
-cacheout==0.13.1
-cachetools==5.1.0
-cattrs==22.1.0
-cdk-nag==2.18.17
-certifi==2022.12.7
-cffi==1.15.0
+blinker==1.6.2
+boto3==1.28.12
+botocore==1.31.12
+build==0.10.0
+cacheout==0.14.1
+cachetools==5.3.1
+cattrs==23.1.2
+cdk-nag==2.27.82
+certifi==2023.7.22
+cffi==1.15.1
 cfn-flip==1.3.0
-charset-normalizer==2.0.12
-click==8.1.3
-colored==1.4.3
-commonmark==0.9.1
-constructs==10.1.10
-coverage[toml]==6.5.0
-cryptography==38.0.4
-dataset==1.5.2
+charset-normalizer==3.2.0
+click==8.1.6
+colorama==0.4.6
+colored==2.2.3
+constructs==10.2.69
+coverage[toml]==7.2.7
+cryptography==41.0.3
+dataset==1.6.2
 decorator==5.1.1
-dill==0.3.5.1
-exceptiongroup==1.1.1
+dill==0.3.7
+exceptiongroup==1.1.2
 fastcounter==1.1.0
 ghp-import==2.1.0
-greenlet==1.1.2
+greenlet==2.0.2
 html5tagger==1.3.0
-httptools==0.4.0
-idna==3.3
-importlib-metadata==4.11.3
-iniconfig==1.1.1
-invoke==1.7.1
+httptools==0.6.0
+idna==3.4
+importlib-metadata==6.8.0
+importlib-resources==6.0.0
+iniconfig==2.0.0
+invoke==2.2.0
 ipaddress==1.0.23
-isort==5.10.1
+isort==5.12.0
 jinja2==3.1.2
-jmespath==1.0.0
-jsii==1.74.0
-lazy-object-proxy==1.7.1
+jmespath==1.0.1
+jsii==1.88.0
+lazy-object-proxy==1.9.0
 ldappool==3.0.0
 mako==1.2.4
-markdown==3.3.7
-markupsafe==2.1.1
+markdown==3.4.4
+markdown-it-py==3.0.0
+markupsafe==2.1.3
 mccabe==0.7.0
-memory-profiler==0.60.0
+mdurl==0.1.2
+memory-profiler==0.61.0
 mergedeep==1.3.4
-mkdocs==1.3.0
-mkdocs-material==8.2.15
-mkdocs-material-extensions==1.0.3
-multidict==6.0.2
-mypy==0.950
-mypy-extensions==0.4.3
+mkdocs==1.5.0
+mkdocs-material==9.1.20
+mkdocs-material-extensions==1.1.1
+multidict==6.0.4
+mypy==1.4.1
+mypy-extensions==1.0.0
 openapi-schema-pydantic==1.2.4
-opensearch-py==2.0.0
-orjson==3.6.5
-packaging==21.3
-pep517==0.12.0
-pip-tools==6.6.1
-platformdirs==2.5.2
-pluggy==1.0.0
-prettytable==3.3.0
-prometheus-client==0.14.1
-prompt-toolkit==3.0.29
-psutil==5.9.0
+opensearch-py==2.2.0
+orjson==3.9.2
+packaging==23.1
+pathspec==0.11.1
+pip-tools==7.1.0
+platformdirs==3.9.1
+pluggy==1.2.0
+prettytable==3.8.0
+prometheus-client==0.17.1
+prompt-toolkit==3.0.39
+psutil==5.9.5
 publication==0.0.3
-pyasn1==0.4.8
-pyasn1-modules==0.2.8
+pyasn1==0.5.0
+pyasn1-modules==0.3.0
 pycparser==2.21
-pydantic==1.9.0
+pydantic==1.10.12
 pyfiglet==0.8.post1
-pygments==2.12.0
-pyhocon==0.3.59
-pyjwt==2.4.0
-pylint==2.15.4
-pymdown-extensions==10.0.1
-pyparsing==2.4.7
-pytest==7.3.2
-pytest-cov==4.0.0
-pytest-mock==3.10.0
+pygments==2.15.1
+pyhocon==0.3.60
+pyjwt==2.8.0
+pylint==2.17.5
+pymdown-extensions==10.1
+pyparsing==3.1.0
+pyproject-hooks==1.0.0
+pytest==7.4.0
+pytest-cov==4.1.0
+pytest-mock==3.11.1
 python-dateutil==2.8.2
 python-dynamodb-lock==0.9.1
-python-ldap==3.4.0
-pytz==2022.1
-pytz-deprecation-shim==0.1.0.post0
-pyyaml==6.0
+python-ldap==3.4.3
+pytz==2023.3
+pyyaml==6.0.1
 pyyaml-env-tag==0.1
 questionary==1.10.0
 random-password-generator==2.2.0
+regex==2023.6.3
 requests==2.31.0
-requests-aws4auth==1.1.2
+requests-aws4auth==1.2.3
 requests-unixsocket==0.3.0
-rich==12.4.1
-s3transfer==0.6.0
+rich==13.4.2
+s3transfer==0.6.1
 sanic==23.3.0
-sanic-routing==22.8.0
-semver==2.13.0
-sh==1.14.2
-shortuuid==1.0.9
+sanic-routing==23.6.0
+semver==3.0.1
+sh==2.0.4
+shortuuid==1.0.11
 six==1.16.0
-sqlalchemy==1.4.36
-supervisor==4.2.4
+sqlalchemy==1.4.49
+supervisor==4.2.5
 tomli==2.0.1
-tomlkit==0.11.5
+tomlkit==0.12.0
 tracerite==1.1.0
-troposphere==4.3.0
+troposphere==4.3.2
 typeguard==2.13.3
-typing-extensions==4.2.0
-tzdata==2022.1
-tzlocal==4.2
-ujson==5.7.0
-urllib3==1.26.9
-uvloop==0.16.0
-validators==0.19.0
-watchdog==2.1.8
-wcwidth==0.2.5
-websockets==10.3
+typing-extensions==4.7.1
+tzlocal==5.0.1
+ujson==5.8.0
+urllib3==1.26.16
+uvloop==0.17.0
+validators==0.20.0
+watchdog==3.0.0
+wcwidth==0.2.6
+websockets==11.0.3
 wheel==0.38.4
-wrapt==1.14.1
-zipp==3.8.0
+wrapt==1.15.0
+zipp==3.16.2
 
 # The following packages are considered to be unsafe in a requirements file:
 # pip

requirements/doc.txt

@@ -1,20 +1,28 @@
-click==8.1.3
+certifi==2023.7.22
+charset-normalizer==3.2.0
+click==8.1.6
+colorama==0.4.6
 ghp-import==2.1.0
-importlib-metadata==4.11.3
+idna==3.4
+importlib-metadata==6.8.0
 jinja2==3.1.2
-markdown==3.3.7
-markupsafe==2.1.1
+markdown==3.4.4
+markupsafe==2.1.3
 mergedeep==1.3.4
-mkdocs==1.3.0
-mkdocs-material==8.2.15
-mkdocs-material-extensions==1.0.3
-packaging==21.3
-pygments==2.12.0
-pymdown-extensions==10.0.1
-pyparsing==3.0.9
+mkdocs==1.5.0
+mkdocs-material==9.1.20
+mkdocs-material-extensions==1.1.1
+packaging==23.1
+pathspec==0.11.1
+platformdirs==3.9.1
+pygments==2.15.1
+pymdown-extensions==10.1
 python-dateutil==2.8.2
-pyyaml==6.0
+pyyaml==6.0.1
 pyyaml-env-tag==0.1
+regex==2023.6.3
+requests==2.31.0
 six==1.16.0
-watchdog==2.1.8
-zipp==3.8.0
+urllib3==2.0.4
+watchdog==3.0.0
+zipp==3.16.2