Support buildkite CICD and restructure smoke tests

[zpoint]

The PR is based on #4386. Please review and merge that PR first, then review and merge this one.

• Restructure test_smoke.py into multiple files within this directory.

tests/smoke_tests
├── __init__.py
├── test_basic.py
├── test_cluster_job.py
├── test_images.py
├── test_managed_job.py
├── test_mount_and_storage.py
├── test_region_and_zone.py
├── test_required_before_merge.py
├── test_sky_serve.py
└── util.py

• Add a directory .buildkite to contain all the buildkite-related files.
  • Test cases in test_required_before_merge.py will be triggered only by valid users' comment on a PR.
    • Support format: pre-merge-test aws / pre-merge-test aws,azure / pre-merge-test(default aws)
  • Other test cases in different files will be run before release and triggered manually.

.buildkite
├── generate_pipeline.py  -- script to generate following yaml automatically
├── pipeline_smoke_tests_pre_merge.yaml -- pre merge smoke tests, only the minimal requried, generated from tests/smoke_tests/test_required_before_merge.py
└── pipeline_smoke_tests_release.yaml -- release smoke tests, all smoke tests included, generated from tests/smoke_tests/test_*.py

• Support the LOG_TO_STDOUT environment variable when executing pytest, so that the output is visible directly on the CI/CD terminal page

• Support the SKYPILOT_SUPPRESS_SENSITIVE_LOG environment variable when executing pytest to suppress logs from provision.py, ensuring sensitive information is not exposed during pre-merge tests.

• Made a minor change to .pre-commit-config.yaml after discovering that pre-commit wasn't working for files under .buildkite.

Test

Don't have permission for this repo, currently tested under my personal repo

Per PR merged CI

• I configured the pre-PR Buildkite pipeline on my personal forked repository.
• You can see an example PR on my personal repository (GitHub integration is visible in that PR).

Scheduled CI

A build attempt link

Tested (run the relevant ones):

• Code formatting: bash format.sh
• Any manual or new tests for this PR (please specify below)
• All smoke tests: pytest tests/test_smoke.py
• Relevant individual smoke tests: pytest tests/test_smoke.py::test_fill_in_the_name
• Backward compatibility tests: conda deactivate; bash -i tests/backward_compatibility_tests.sh

[zpoint]

If we don't add this flag, pre-commit does nothing for yapf. Now it should be fine.

(sky) ➜  skypilot git:(dev/zeping/restructure_smoke_tests) ✗ git commit -m "fix pre-commit"
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...............................................................Passed
check for added large files..............................................Passed
black (IBM specific).................................(no files to check)Skipped
isort (general)......................................(no files to check)Skipped
isort (IBM specific).................................(no files to check)Skipped
mypy.....................................................................Passed
yapf.....................................................................Failed
- hook id: yapf
- files were modified by this hook
pylint...............................................(no files to check)Skipped

[zpoint]

The format.sh script limits files to this range, but .github/workflows/format.yml does not. Remove the limitation in .pre-commit-config.yaml to ensure consistency with the final result (.github/workflows/format.yml).