Update dry run for atmos vendor pull to support ssh + detailed SCP urls alignment #1076
Conversation
Co-authored-by: Erik Osterman (CEO @ Cloud Posse) <[email protected]>
Co-authored-by: Erik Osterman (CEO @ Cloud Posse) <[email protected]>
Co-authored-by: Erik Osterman (CEO @ Cloud Posse) <[email protected]>
Co-authored-by: Erik Osterman (CEO @ Cloud Posse) <[email protected]>
Co-authored-by: Erik Osterman (CEO @ Cloud Posse) <[email protected]>
Listener430
changed the title
Listener430
changed the title
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
… only in dry mode
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (4)
tests/fixtures/scenarios/vendor-pulls-ssh/vendor.yaml (2)
9-18: Basic HTTPS Source Configuration Review
The first source entry clearly documents a basic HTTPS URL (with token injection expected), and the structure is correct. Note that line 18 has trailing spaces that should be removed to satisfy YAML lint rules.- # Basic HTTPS default (token injection expected)␣␣ + # Basic HTTPS default (token injection expected)🧰 Tools
🪛 YAMLlint (1.35.1)
[error] 18-18: trailing spaces
(trailing-spaces)
19-27: Direct Credentials URL Configuration Caution
The second source entry uses direct credentials in the URL. While this may be intentional for testing or specific scenarios, please verify that exposing credentials in configuration is acceptable for your environment.tests/test-cases/demo-vendoring.yaml (2)
61-79: Environment Variable Formatting Consistency
The "atmos vendor pull ssh component" test case is sound in its intent. However, notice that theATMOS_GITHUB_TOKENvalue is provided without quotes here while quoted in the SSH test case. For maintainability and consistency, consider using the same quoting style for environment variables across test cases.
96-96: Remove Trailing Whitespace
Static analysis has detected trailing spaces on these lines. Removing these extraneous spaces helps maintain clean YAML formatting.- - "Injecting token" + - "Injecting token" - - "!supersecret" + - "!supersecret"Also applies to: 98-98
🧰 Tools
🪛 YAMLlint (1.35.1)
[error] 96-96: trailing spaces
(trailing-spaces)
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (5)
internal/exec/vendor_utils.go(1 hunks)tests/fixtures/scenarios/vendor-pulls-ssh/vendor.yaml(1 hunks)tests/snapshots/TestCLICommands_atmos_vendor_pull_custom_detector_credentials_leakage.stderr.golden(1 hunks)tests/snapshots/TestCLICommands_atmos_vendor_pull_ssh.stderr.golden(1 hunks)tests/test-cases/demo-vendoring.yaml(1 hunks)
✅ Files skipped from review due to trivial changes (1)
- internal/exec/vendor_utils.go
🚧 Files skipped from review as they are similar to previous changes (2)
- tests/snapshots/TestCLICommands_atmos_vendor_pull_custom_detector_credentials_leakage.stderr.golden
- tests/snapshots/TestCLICommands_atmos_vendor_pull_ssh.stderr.golden
🧰 Additional context used
🪛 YAMLlint (1.35.1)
tests/test-cases/demo-vendoring.yaml
[error] 96-96: trailing spaces
(trailing-spaces)
[error] 98-98: trailing spaces
(trailing-spaces)
tests/fixtures/scenarios/vendor-pulls-ssh/vendor.yaml
[error] 18-18: trailing spaces
(trailing-spaces)
⏰ Context from checks skipped due to timeout of 90000ms (4)
- GitHub Check: [localstack] demo-localstack
- GitHub Check: Acceptance Tests (macos-latest, macos)
- GitHub Check: Acceptance Tests (windows-latest, windows)
- GitHub Check: Summary
🔇 Additional comments (4)
tests/fixtures/scenarios/vendor-pulls-ssh/vendor.yaml (2)
1-8: File Header Configuration Looks Good
The top-level keys such asapiVersion,kind, andmetadata(withnameanddescription) are clearly defined and follow the expected schema for an Atmos vendoring manifest.
28-36: SSH Implicit Source Configuration Validated
The third source entry correctly represents an SSH implicit method, aligning with the enhanced SCP and SSH URL support highlighted in the PR objectives. The structure is consistent with the other components, ensuring that the custom detectors in the codebase can correctly process these URLs.tests/test-cases/demo-vendoring.yaml (2)
43-60: SSH Dry-Run Test Case Validation
This test case correctly exercises the dry-run mode for vendor pulls using an SSH-style URL. The use of detailed logging (via--logs-level=Debug) and the dry-run flag ensures that no actual file downloads occur. The work directory and environment variable setup align with the PR objectives.
80-101: Credential Masking Verification
This test case does an excellent job ensuring that injected credentials do not appear in log outputs by checking for a masked output (e.g., "!supersecret") and an indicative log message ("Injecting token"). The expected stdout and stderr values properly cover the security requirement regarding credential leakage.🧰 Tools
🪛 YAMLlint (1.35.1)
[error] 96-96: trailing spaces
(trailing-spaces)
[error] 98-98: trailing spaces
(trailing-spaces)
mergify
bot
added
needs-cloudposse
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (13)
tests/fixtures/scenarios/vendor-creds-sanitize/atmos.yaml (1)
13-13: Remove trailing whitespace.There's trailing whitespace on this line that should be removed.
- +🧰 Tools
🪛 YAMLlint (1.35.1)
[error] 13-13: trailing spaces
(trailing-spaces)
tests/fixtures/scenarios/vendor-pulls-ssh/vendor.yaml (1)
1-25: Minor typo in comment.There's a small typo in line 17's comment.
- ## Explicit ssh vednoring (the schema is explicitly spcified along with a username, no custom detector is invoked) + ## Explicit ssh vendoring (the schema is explicitly specified along with a username, no custom detector is invoked)tests/fixtures/scenarios/vendoring-dry-run/atmos.yaml (1)
1-23: Fix YAML formatting issues.There are a few formatting issues in this YAML file that should be addressed.
- base_path: "./" settings: inject_github_token: true components: terraform: base_path: "components/terraform" apply_auto_approve: false deploy_run_init: true init_run_reconfigure: true auto_generate_backend_file: false - + stacks: base_path: "stacks" included_paths: - "deploy/**/*" excluded_paths: - "**/_defaults.yaml" name_pattern: "{stage}" - - +🧰 Tools
🪛 YAMLlint (1.35.1)
[warning] 1-1: too many blank lines
(1 > 0) (empty-lines)
[error] 13-13: trailing spaces
(trailing-spaces)
[warning] 22-22: too many blank lines
(2 > 0) (empty-lines)
tests/fixtures/scenarios/vendor-pulls-ssh/atmos.yaml (1)
1-20: Add newline at end of file.The YAML file is missing a newline character at the end.
logs: file: "/dev/stderr" level: Info +🧰 Tools
🪛 YAMLlint (1.35.1)
[error] 20-20: no new line character at the end of file
(new-line-at-end-of-file)
tests/fixtures/scenarios/vendor-creds-sanitize/vendor.yaml (2)
18-18: Fix trailing whitespaceRemove the trailing spaces at the end of this line.
- # Direct credentials provided in the URL (token injection should be skipped) + # Direct credentials provided in the URL (token injection should be skipped)🧰 Tools
🪛 YAMLlint (1.35.1)
[error] 18-18: trailing spaces
(trailing-spaces)
28-35: Add newline at end of fileAdd a newline character at the end of the file to comply with YAML standards.
tags: - demo +🧰 Tools
🪛 YAMLlint (1.35.1)
[error] 35-35: no new line character at the end of file
(new-line-at-end-of-file)
website/docs/cli/commands/vendor/vendor-pull.mdx (1)
216-218: Consider refining the wording and punctuation.You might strengthen the verb choice in "When resolving Git sources, Atmos follows these rules" by saying “When processing Git sources, Atmos follows these rules.” Also, using a typographical ellipsis character “…” rather than “...” can be more stylistically consistent.
🧰 Tools
🪛 LanguageTool
[style] ~216-~216: Consider using a different verb to strengthen your wording.
Context: ...lved When resolving Git sources, Atmos follows these rules: 1. If a *full HTTPS URL...(FOLLOW_OBEY)
[style] ~218-~218: Consider using the typographical ellipsis character here instead.
Context: ...1. If a full HTTPS URL is provided (git::https://github.com/...), it is used as-is. No token data is i...(ELLIPSIS)
tests/test-cases/vendoring-ssh-dryrun.yaml (1)
57-59: Remove trailing spaces and add a newline at EOF.YAMLLint flagged trailing spaces on line 57 and requires a newline at the end of the file on line 59. Please ensure both are fixed:
- !not 'supersecret' - - !not 'ATMOS_GITHUB_TOKEN' \ No newline at end of file + - !not 'ATMOS_GITHUB_TOKEN' +🧰 Tools
🪛 YAMLlint (1.35.1)
[error] 57-57: trailing spaces
(trailing-spaces)
[error] 59-59: no new line character at the end of file
(new-line-at-end-of-file)
website/docs/cli/configuration/configuration.mdx (2)
681-683: Minor grammar fixes.
- Lowercase "Requests" for consistency in “Unauthenticated requests”.
- Possibly clarify “avoided” vs. “used” regarding rate limits.
- | ATMOS_BITBUCKET_TOKEN | N/A | App password for Bitbucket API requests is set to avoid rate limits. Unauthenticated Requests are limited to 60 requests per hour across all API resources. | + | ATMOS_BITBUCKET_TOKEN | N/A | App password for Bitbucket API requests, helping avoid rate limits. Unauthenticated requests are limited to 60 requests per hour across all API resources. |
699-699: Typographical correction.Fix the spelling from “comamnds” to “commands”:
- | ATMOS_TERRAFORM_WORKSPACE | The name of the Terraform workspace in which Terraform comamnds should be run | + | ATMOS_TERRAFORM_WORKSPACE | The name of the Terraform workspace in which Terraform commands should be run |internal/exec/go_getter_utils_test.go (3)
118-137: Token injection test could be more robust.While the test correctly verifies the basic token injection, consider adding cases for empty tokens or different host scenarios to ensure comprehensive coverage.
func TestInjectToken(t *testing.T) { os.Setenv("GITHUB_TOKEN", "testtoken") defer os.Unsetenv("GITHUB_TOKEN") config := fakeAtmosConfig(true) detector := &CustomGitDetector{AtmosConfig: &config} uObj, err := url.Parse("https://github.com/user/repo.git") if err != nil { t.Fatalf("Failed to parse URL: %v", err) } detector.injectToken(uObj, hostGitHub) if uObj.User == nil { t.Error("Expected token to be injected into URL") } else { user := uObj.User.Username() if user != getDefaultUsername(hostGitHub) { t.Errorf("Expected username %s, got %s", getDefaultUsername(hostGitHub), user) } } + + // Test with empty token + os.Unsetenv("GITHUB_TOKEN") + uObj2, _ := url.Parse("https://github.com/user/repo.git") + detector.injectToken(uObj2, hostGitHub) + if uObj2.User != nil { + t.Error("Expected no token injection with empty token") + } + + // Test with non-GitHub host + os.Setenv("GITLAB_TOKEN", "gltoken") + defer os.Unsetenv("GITLAB_TOKEN") + uObj3, _ := url.Parse("https://gitlab.com/user/repo.git") + detector.injectToken(uObj3, hostGitLab) + if uObj3.User == nil { + t.Error("Expected token to be injected for GitLab URL") + } else if uObj3.User.Username() != getDefaultUsername(hostGitLab) { + t.Errorf("Expected username %s, got %s", getDefaultUsername(hostGitLab), uObj3.User.Username()) + } }
300-310: Consider consolidating redundant URI validation tests.These test cases duplicate checks already present in the main TestValidateURI function (lines 37-42). Consider consolidating into a single comprehensive test to improve maintainability.
312-319: Consider consolidating redundant SCP URL tests.This test duplicates functionality already tested in TestRewriteSCPURL (lines 98-102). Consolidating these would make the test suite more maintainable.
📜 Review details
Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (25)
.github/workflows/test.yml(3 hunks).golangci.yml(1 hunks)codecov.yml(1 hunks)internal/exec/go_getter_utils.go(4 hunks)internal/exec/go_getter_utils_test.go(1 hunks)internal/exec/vendor_model.go(2 hunks)internal/exec/vendor_utils.go(2 hunks)internal/exec/yaml_func_include.go(1 hunks)pkg/utils/url_utils.go(1 hunks)pkg/utils/url_utils_test.go(1 hunks)tests/fixtures/scenarios/vendor-creds-sanitize/atmos.yaml(1 hunks)tests/fixtures/scenarios/vendor-creds-sanitize/vendor.yaml(1 hunks)tests/fixtures/scenarios/vendor-pulls-ssh/atmos.yaml(1 hunks)tests/fixtures/scenarios/vendor-pulls-ssh/vendor.yaml(1 hunks)tests/fixtures/scenarios/vendoring-dry-run/atmos.yaml(1 hunks)tests/fixtures/scenarios/vendoring-dry-run/components/terraform/ipinfo/component.yaml(1 hunks)tests/snapshots/TestCLICommands_atmos_vendor_pull_component_using_SSH.stderr.golden(1 hunks)tests/snapshots/TestCLICommands_atmos_vendor_pull_custom_detector_credentials_leakage.stderr.golden(1 hunks)tests/snapshots/TestCLICommands_atmos_vendor_pull_ssh.stderr.golden(1 hunks)tests/snapshots/TestCLICommands_atmos_vendor_pull_ssh_component.stderr.golden(1 hunks)tests/snapshots/TestCLICommands_atmos_vendor_pull_using_SSH.stderr.golden(1 hunks)tests/snapshots/TestCLICommands_atmos_vendor_pull_with_custom_detector_and_handling_credentials_leakage.stderr.golden(1 hunks)tests/test-cases/vendoring-ssh-dryrun.yaml(1 hunks)website/docs/cli/commands/vendor/vendor-pull.mdx(2 hunks)website/docs/cli/configuration/configuration.mdx(2 hunks)
🧰 Additional context used
🧠 Learnings (6)
internal/exec/vendor_model.go (1)
Learnt from: samtholiya
PR: cloudposse/atmos#1077
File: pkg/downloader/custom_github_detector.go:0-0
Timestamp: 2025-02-21T20:56:05.539Z
Learning: The `github.com/charmbracelet/log` package should be imported with the alias `log` according to the project's import alias configuration.
tests/snapshots/TestCLICommands_atmos_vendor_pull_component_using_SSH.stderr.golden (1)
Learnt from: Listener430
PR: cloudposse/atmos#1061
File: tests/snapshots/TestCLICommands_atmos_vendor_pull_ssh.stderr.golden:8-8
Timestamp: 2025-03-12T18:52:22.003Z
Learning: Test snapshots in the Atmos project, particularly for dry run scenarios, may be updated during the development process, and temporary inconsistencies in their content should not be flagged as issues.
tests/snapshots/TestCLICommands_atmos_vendor_pull_ssh.stderr.golden (1)
Learnt from: Listener430
PR: cloudposse/atmos#1061
File: tests/snapshots/TestCLICommands_atmos_vendor_pull_ssh.stderr.golden:8-8
Timestamp: 2025-03-12T18:52:22.003Z
Learning: Test snapshots in the Atmos project, particularly for dry run scenarios, may be updated during the development process, and temporary inconsistencies in their content should not be flagged as issues.
tests/snapshots/TestCLICommands_atmos_vendor_pull_ssh_component.stderr.golden (1)
Learnt from: Listener430
PR: cloudposse/atmos#1061
File: tests/snapshots/TestCLICommands_atmos_vendor_pull_ssh.stderr.golden:8-8
Timestamp: 2025-03-12T18:52:22.003Z
Learning: Test snapshots in the Atmos project, particularly for dry run scenarios, may be updated during the development process, and temporary inconsistencies in their content should not be flagged as issues.
tests/snapshots/TestCLICommands_atmos_vendor_pull_using_SSH.stderr.golden (1)
Learnt from: Listener430
PR: cloudposse/atmos#1061
File: tests/snapshots/TestCLICommands_atmos_vendor_pull_ssh.stderr.golden:8-8
Timestamp: 2025-03-12T18:52:22.003Z
Learning: Test snapshots in the Atmos project, particularly for dry run scenarios, may be updated during the development process, and temporary inconsistencies in their content should not be flagged as issues.
internal/exec/go_getter_utils.go (2)
Learnt from: Listener430
PR: cloudposse/atmos#1061
File: internal/exec/go_getter_utils.go:74-75
Timestamp: 2025-03-12T18:52:31.814Z
Learning: In the `CustomGitDetector.Detect` method of `internal/exec/go_getter_utils.go`, verbose debug logging of raw URLs is intentionally kept for debugging purposes, despite potential credential exposure risks.
Learnt from: samtholiya
PR: cloudposse/atmos#1077
File: pkg/downloader/custom_github_detector.go:0-0
Timestamp: 2025-02-21T20:56:05.539Z
Learning: The `github.com/charmbracelet/log` package should be imported with the alias `log` according to the project's import alias configuration.
🪛 YAMLlint (1.35.1)
tests/fixtures/scenarios/vendor-creds-sanitize/atmos.yaml
[warning] 1-1: too many blank lines
(1 > 0) (empty-lines)
[error] 13-13: trailing spaces
(trailing-spaces)
tests/fixtures/scenarios/vendor-pulls-ssh/atmos.yaml
[error] 20-20: no new line character at the end of file
(new-line-at-end-of-file)
tests/fixtures/scenarios/vendoring-dry-run/atmos.yaml
[warning] 1-1: too many blank lines
(1 > 0) (empty-lines)
[error] 13-13: trailing spaces
(trailing-spaces)
[warning] 22-22: too many blank lines
(2 > 0) (empty-lines)
tests/test-cases/vendoring-ssh-dryrun.yaml
[error] 57-57: trailing spaces
(trailing-spaces)
[error] 59-59: no new line character at the end of file
(new-line-at-end-of-file)
tests/fixtures/scenarios/vendor-creds-sanitize/vendor.yaml
[error] 18-18: trailing spaces
(trailing-spaces)
[error] 35-35: no new line character at the end of file
(new-line-at-end-of-file)
🪛 LanguageTool
website/docs/cli/commands/vendor/vendor-pull.mdx
[style] ~216-~216: Consider using a different verb to strengthen your wording.
Context: ...lved When resolving Git sources, Atmos follows these rules: 1. If a *full HTTPS URL...
(FOLLOW_OBEY)
[style] ~218-~218: Consider using the typographical ellipsis character here instead.
Context: ...1. If a full HTTPS URL is provided (git::https://github.com/...), it is used as-is. No token data is i...
(ELLIPSIS)
⏰ Context from checks skipped due to timeout of 90000ms (2)
- GitHub Check: Build (windows-latest, windows)
- GitHub Check: Summary
🔇 Additional comments (42)
.golangci.yml (1)
173-173: Increased nestif complexity thresholdRaising the min-complexity threshold from 4 to 6 will reduce noise in linting output by only flagging more complex nested if statements.
codecov.yml (1)
16-16: Improved boolean type representationChanged from string representation
yesto proper boolean valuetrue. This is a good practice for YAML configuration files.internal/exec/yaml_func_include.go (1)
50-50: Changed parameter passing to use pointerUpdated to pass
atmosConfigby reference instead of by value, which aligns with the function signature ingo_getter_utils.gothat expects a pointer parameter.tests/snapshots/TestCLICommands_atmos_vendor_pull_ssh.stderr.golden (1)
1-10: New test snapshot for SSH vendor pull dry runThe snapshot captures the expected CLI output for the vendor pull command when using SSH, including debug logs, warnings, and status messages. This supports the PR objective of enhancing dry run output with more detailed information.
I note this snapshot shows proper sanitized output including detailed information about vendored components without downloading files, which directly addresses the objectives of this PR.
internal/exec/vendor_model.go (2)
15-15: Good job fixing the import alias.The
charmbracelet/logpackage is now correctly imported with thelogalias, following the project's import alias configuration.
337-337: Improved debugging visibility.Adding this debug log provides better visibility into the package installation process, which is valuable for troubleshooting and aligns with the PR's goal to enhance output information.
tests/fixtures/scenarios/vendor-creds-sanitize/atmos.yaml (1)
1-21: Good test configuration for credential sanitization.This configuration file properly sets up a test scenario with GitHub token injection enabled, which will be useful for testing the sanitization features in the dry run output.
🧰 Tools
🪛 YAMLlint (1.35.1)
[warning] 1-1: too many blank lines
(1 > 0) (empty-lines)
[error] 13-13: trailing spaces
(trailing-spaces)
tests/snapshots/TestCLICommands_atmos_vendor_pull_component_using_SSH.stderr.golden (1)
1-11: Good test snapshot for SSH vendoring.This snapshot correctly captures the expected CLI output for the SSH vendoring dry run scenario, showing proper debug messages, warnings, and confirmation of the dry run completion.
pkg/utils/url_utils.go (2)
8-8: Good constant definition for masking secrets.Using a named constant for the masked value follows good practice and addresses the previous feedback to move the string to a constant.
10-26: Well-implemented URL sanitization function.The
MaskBasicAuthfunction properly handles URL parsing, credential detection, and masking. It correctly differentiates between URLs with username-only and username-password combinations.internal/exec/vendor_utils.go (2)
22-23: LGTM - Adding dedicated stderr logger helps separate detailed messaging.The new StderrLogger ensures that detailed messaging, specifically for files vendoring, doesn't pollute stdout. This aligns well with improving the dry run output clarity.
514-514: Clean implementation of stderr logging.Good replacement of standard log with the dedicated StderrLogger for including paths. This ensures detailed file inclusion messages go to stderr, keeping stdout clean as intended.
tests/fixtures/scenarios/vendor-pulls-ssh/vendor.yaml (2)
9-16: Good implementation of implicit SSH vendoring.The implicit SSH vendoring configuration correctly uses SCP-style Git URL format (
[email protected]:cloudposse/terraform-null-label.git). This addresses the need for SCP URL alignment in the dry run output as mentioned in the PR objectives.
17-24: Explicit SSH vendoring looks correct.The explicit SSH vendoring configuration uses the proper schema (
ssh://) with a username. This provides a good comparison case to the implicit SSH vendoring and ensures comprehensive testing of SSH support.tests/fixtures/scenarios/vendoring-dry-run/atmos.yaml (1)
4-4: Good addition of GitHub token injection setting.The
inject_github_token: truesetting is important for the vendoring functionality being tested, especially for dry run output that needs to show sanitized credentials.tests/fixtures/scenarios/vendor-pulls-ssh/atmos.yaml (1)
18-20: Appropriate log configuration for testing.Setting logs to stderr with Info level is a good choice for this scenario. It ensures that logs don't interfere with test outputs while maintaining visibility of important information during tests.
🧰 Tools
🪛 YAMLlint (1.35.1)
[error] 20-20: no new line character at the end of file
(new-line-at-end-of-file)
tests/snapshots/TestCLICommands_atmos_vendor_pull_using_SSH.stderr.golden (1)
11-13: Dry run output structure looks goodThe output clearly shows what would happen in actual execution (vendoring 2 components) after confirming no actual components were vendored during dry run. This separation helps users understand the expected actions without making changes to the filesystem.
tests/fixtures/scenarios/vendor-creds-sanitize/vendor.yaml (1)
9-35: Well-structured test scenarios for credential handlingGood job defining three distinct scenarios for credential handling:
- Default HTTPS with token injection
- Direct credentials in URL (skipping token injection)
- Pre-existing credentials (skipping token injection)
This comprehensive test file will effectively validate the credential sanitization features mentioned in the PR objectives.
🧰 Tools
🪛 YAMLlint (1.35.1)
[error] 18-18: trailing spaces
(trailing-spaces)
[error] 35-35: no new line character at the end of file
(new-line-at-end-of-file)
tests/snapshots/TestCLICommands_atmos_vendor_pull_with_custom_detector_and_handling_credentials_leakage.stderr.golden (1)
8-12: Excellent credential handling and loggingThe debug logs provide detailed visibility into the URL transformation process while properly masking sensitive credentials ("xxx:xxx"). This achieves the PR objective of showing detailed information about injected tokens and URL transformations without exposing sensitive data.
tests/snapshots/TestCLICommands_atmos_vendor_pull_ssh_component.stderr.golden (2)
8-12: Successfully implements SCP-style SSH URL rewritingThe logs clearly demonstrate the key feature of this PR - rewriting SCP-style SSH URLs (github.com:cloudposse/atmos.git) to standard SSH URLs (ssh://github.com/cloudposse/atmos.git). The credential injection and masking are also correctly handled, fulfilling the PR's objective of supporting SSH in vendor pull commands.
14-16: Dry run output structure is clearSimilar to the first file, the output effectively separates what actually happened (dry run with no actual vendoring) from what would happen in a real execution (vendoring 1 component). This structure helps users understand the expected actions without making filesystem changes.
.github/workflows/test.yml (3)
49-50: Added full Git history checkout to ensure complete repository access.The change to
fetch-depth: 0ensures the entire Git history is available during workflow execution, which is essential for comprehensive Git-related operations like the vendor pull command with SSH support.
88-88: Increased timeout to allow for more complex Git operations.The timeout increase from 15 to 20 minutes provides adequate time for the enhanced vendor pull functionality to complete successfully, particularly with SSH and URL processing.
159-159: Extended acceptance test timeout to accommodate new vendor pull functionality.Increasing the test timeout to 20 minutes ensures tests have sufficient time to execute, especially when testing various Git URL formats and SSH handling.
tests/snapshots/TestCLICommands_atmos_vendor_pull_custom_detector_credentials_leakage.stderr.golden (3)
7-11: Properly masks credentials in Git URLs during dry run.The logs correctly show the custom Git detector in action with appropriate credential masking. The URL transformation process is well-documented in the logs:
- Detection of Git URL
- Default scheme application
- Token injection
- Credential masking (showing "xxx:xxx" instead of actual tokens)
This is exactly what we want to see for secure credential handling in dry run mode.
32-32: Log entry incorrectly identifies directory as file.The debug message refers to "docs" as a file, but it's actually a directory.
87-92: Clear success indicators for vendored components.The output properly indicates successful vendoring of each component with version information, making it easy to verify the operation's results.
pkg/utils/url_utils_test.go (1)
1-49: Comprehensive test coverage for URL credential masking.The tests thoroughly cover all important scenarios for the MaskBasicAuth function:
- URLs with both username and password
- URLs with username only
- URLs without credentials
- Invalid URL handling
Each test verifies the expected output and proper error handling, ensuring the URL masking functionality is reliable.
tests/fixtures/scenarios/vendoring-dry-run/components/terraform/ipinfo/component.yaml (2)
7-9: SSH URL format correctly implemented for Git source.The configuration properly uses the SSH format (
[email protected]) for the Git URI, which aligns with the PR objective of supporting SSH URLs in the vendor pull command.
10-13: Appropriate file inclusion patterns for Terraform components.The included paths cover all essential Terraform files (.tf), variable files (.tfvars), and documentation (*.md), ensuring that all necessary component files are vendored.
website/docs/cli/commands/vendor/vendor-pull.mdx (2)
111-120: Good addition for SSH vendoring!The new "Vendoring from SSH" section is clear and helps users understand how SSH-based repository access is configured.
150-156: Clarity improvement acknowledged.The explanation about invalid URLs and Atmos defaulting to HTTPS is straightforward and addresses common user pitfalls.
internal/exec/go_getter_utils.go (3)
66-70: Struct definition looks good.This new
CustomGitDetectorneatly encapsulates the Atmos config reference for advanced Git URL handling.
74-124: Useful custom detection logic.The rewriting of SCP-style URLs and the fallback to
https://help unify various URL forms. Masking the source URL in debug logs further protects credentials.
345-359: Symlink removal approach is straightforward.Recursively walking the downloaded directory and removing symlinks reduces security risks. This appears correct for ensuring only genuine files remain after the clone.
internal/exec/go_getter_utils_test.go (7)
30-52: Great job with comprehensive URI validation tests.The test cases cover all essential validation paths: empty URIs, excessively long URIs, path traversal sequences, spaces, and various valid URI formats including OCI URIs. This thorough approach will help catch validation issues early.
55-65: Well-structured scheme validation tests.You've properly tested both valid schemes (HTTP, HTTPS, Git, SSH, Git+HTTPS, Git+SSH) and invalid schemes (FTP). This ensures the function handles all expected inputs correctly.
68-86: Good coverage of URL scheme enforcement scenarios.This test effectively verifies three key behaviors:
- Preserving existing schemes
- Rewriting SCP-style URLs to use SSH
- Defaulting to HTTPS when no scheme is present
These align well with the PR objective to support SSH and handle SCP URL alignment.
89-103: Solid test for SCP URL rewriting.The test correctly verifies both positive and negative cases for SCP URL rewriting, which is central to the PR's goals of supporting SSH and SCP URL alignment.
194-221: Good platform-aware symlink testing.The test correctly skips symlink tests on Windows where symlink behavior differs, creates a proper test environment, and verifies both the removal of symlinks and preservation of regular files.
223-258: Comprehensive file download test with proper cleanup.This test thoroughly validates the file download functionality by creating temporary directories, writing test content, performing the download, and verifying the result. The Windows skip and proper cleanup with defer are good practices.
338-342: Good test cleanup practice.Properly restoring original detectors in TestMain ensures tests don't interfere with each other or leave the environment in an altered state.
| @@ -13,7 +13,7 @@ comment: | |||
| layout: "reach,diff,flags,tree" # Display different coverage views | |||
| behavior: default # Default PR comment behavior | |||
| require_changes: true # Only post if coverage changes | |||
There was a problem hiding this comment.
| require_changes: true # Only post if coverage changes | |
| require_changes: true # Only post if coverage changes |
| @@ -13,7 +13,7 @@ comment: | |||
| layout: "reach,diff,flags,tree" # Display different coverage views | |||
| behavior: default # Default PR comment behavior | |||
| require_changes: true # Only post if coverage changes | |||
| require_base: yes # Compare against base branch coverage | |||
| require_base: true # Compare against base branch coverage | |||
There was a problem hiding this comment.
| require_base: true # Compare against base branch coverage | |
| require_base: true # Compare against base branch coverage |
| @@ -19,6 +19,9 @@ import ( | |||
| u "github.com/cloudposse/atmos/pkg/utils" | |||
| ) | |||
|
|
|||
| // Dedicated logger for stderr to keep stdout clean of detailed messaging, e.g. for files vendoring. | |||
| var StderrLogger = log.New(os.Stderr) | |||
There was a problem hiding this comment.
We do not need to introduce a new logger. We use log.* everywhere now without this.
| @@ -508,7 +511,7 @@ func generateSkipFunction(tempDir string, s *schema.AtmosVendorSource) func(os.F | |||
| } | |||
|
|
|||
| // If 'included_paths' is not provided, include all files that were not excluded | |||
| log.Debug("Including", u.TrimBasePathFromPath(tempDir+"/", src)) | |||
| StderrLogger.Debug("Including", "path", u.TrimBasePathFromPath(tempDir+"/", src)) | |||
There was a problem hiding this comment.
| StderrLogger.Debug("Including", "path", u.TrimBasePathFromPath(tempDir+"/", src)) | |
| log.Debug("Including", "path", u.TrimBasePathFromPath(tempDir+"/", src)) |
what
This PR upgrades the output of dry run mode for vendor pull command.
After the change is applied, the dry run mode shows
yet there's no actual files download.
Before ( component vendoring dry-run output)
After ( component vendoring dry-run output)
component.yaml, refer to the respective test case for details
Before ( generic vendoring dry-run output)
After ( generic vendoring dry-run output)
vendor.yaml, refer to the respective test case for detaails
why
Received feedback from in the parent PR
references
Upscopes from #1061 and
Summary by CodeRabbit
New Features
Documentation
Chores