[develop] Add NOAA Cloud coverage tests and skill-score WE2E to all coverage suites

[EdwardSnyder-NOAA]

DESCRIPTION OF CHANGES:

This PR adds NOAA Cloud coverage test suites for each cloud service providers, and adds the skill-score WE2E test (grid_SUBCONUS_Ind_3km_ics_FV3GFS_lbcs_FV3GFS_suite_WoFS_v0 ) to each coverage suite via the Jenkinsfile. Adding the three cloud service providers reduces the number of the WE2E tests that run on the other T1 platforms. To test these additional machines within the Jenkins pipeline, a 'check nodes' step was added to the Jenkinsfile.

Testing is on-going with the Jenkins pipeline. Results will be posted, and if things pass, the PR will be fully open for review.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

TESTS CONDUCTED:

• derecho.intel
• gaea.intel
• gaea-c6.intel
• hera.gnu
• hera.intel
• hercules.intel
• jet.intel
• orion.intel
• wcoss2.intel
• NOAA Cloud (indicate which platform)
• Jenkins
• fundamental test suite
• comprehensive tests (specify which if a subset was used)

DEPENDENCIES:

DOCUMENTATION:

No documentation is expected.

ISSUE:

CHECKLIST

• My code follows the style guidelines in the Contributor's Guide
• I have performed a self-review of my own code using the Code Reviewer's Guide
• I have commented my code, particularly in hard-to-understand areas
• My changes need updates to the documentation. I have made corresponding changes to the documentation
• My changes do not require updates to the documentation (explain).
• My changes generate no new warnings
• New and existing tests pass with my changes
• Any dependent changes have been merged and published

LABELS (optional):

A Code Manager needs to add the following labels to this PR:

• Work In Progress
• bug
• enhancement
• documentation
• release
• high priority
• run_ci
• run_we2e_fundamental_tests
• run_we2e_comprehensive_tests
• Needs Cheyenne test
• Needs Jet test
• Needs Hera test
• Needs Orion test
• help wanted

CONTRIBUTORS (optional):

@ankimball @BruceKropp-Raytheon @MichaelLueken

[MichaelLueken]

@EdwardSnyder-NOAA -

It looks like the Azure node was looking for azclusternoaav2eus1 rather than azclusternoaav2use1, so the tests aren't running for that CSP. You can track the progress of the tests https://jenkins.epic.oarcloud.noaa.gov/blue/organizations/jenkins/ufs-srweather-app%2Fpipeline/detail/PR-1200/1/pipeline

[MichaelLueken]

The Test stage is failing on all platforms. Unfortunately, I can't tell which line it is failing on, but it looks like a $ symbol is causing issues in this stage.

[MichaelLueken]

@EdwardSnyder-NOAA -

It looks like the Hercules test has successfully completed! Before we run the final test, it looks like the Platform Team is wanting to update the Jenkins node names for Gaea-C5 and Gaea-C6. Since this PR is already updating the Jenkinsfile, I think those changes would fit in well with this PR. Once you are ready to open the PR, I'll let the Platform Team know that we are ready to make the change. Thanks!

[EdwardSnyder-NOAA]

@EdwardSnyder-NOAA -

It looks like the Hercules test has successfully completed! Before we run the final test, it looks like the Platform Team is wanting to update the Jenkins node names for Gaea-C5 and Gaea-C6. Since this PR is already updating the Jenkinsfile, I think those changes would fit in well with this PR. Once you are ready to open the PR, I'll let the Platform Team know that we are ready to make the change. Thanks!

@MichaelLueken - yeah we can update those names here. What will they be renamed to?

[MichaelLueken]

@EdwardSnyder-NOAA -

gaea would be changed to gaeac5 and gaea-c6 would be changed to gaeac6. This will also require minor modifications to other parts of the repository, not just the Jenkinsfile, so I can open a PR into your skill-score-for-all branch with the necessary changes, if you would prefer.

[EdwardSnyder-NOAA]

@MichaelLueken -
If you have time, go ahead and open up a PR into this branch. Otherwise, I could do it. I can update the names in the Jenkinsfile, since I'm already working there, and the testing suites too.

[MichaelLueken]

@EdwardSnyder-NOAA -

I am working on a PR to update the rest of the gaeac5 and gaeac6 changes in your branch.

It looks like the pcluster test failed to append noaacloud.aws to the coverage test name. The Test stage ultimately failed due to there being no machine_suites/coverage test suite.

[EdwardSnyder-NOAA]

yep, troubleshooting it now. Added print statements to catch what isn't working.

[MichaelLueken]

@EdwardSnyder-NOAA -

I have successfully updated the files and have successfully built and ran the coverage tests on Gaea-C6. I'm now testing on Gaea-C5. Once complete, I will open the PR into your branch.

[MichaelLueken]

@EdwardSnyder-NOAA -

I have opened PR #3 in your fork that will update the rest of the repo to use gaeac5 and gaeac6 rather than gaea and gaea-c6. Please let me know if you need any assistance with getting the Jenkins pipeline to work on the CSPs.