Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update dry run for atmos vendor pull to support ssh + detailed SCP urls alignment #1076

Open
wants to merge 170 commits into
base: main
Choose a base branch
from

Conversation

Listener430
Copy link
Collaborator

@Listener430 Listener430 commented Feb 19, 2025

what

This PR upgrades the output of dry run mode for vendor pull command.
After the change is applied, the dry run mode shows

  1. details on the SCP-style links converion,
  2. Injected tokens
  3. urls being vendored in a santized format
    yet there's no actual files download.

Before ( component vendoring dry-run output)

component_vendoring_dry_run_before

After ( component vendoring dry-run output)

component vendoring dry run after

component.yaml, refer to the respective test case for details

comonent yaml

Before ( generic vendoring dry-run output)

generic_vendoring_dry_run_before

After ( generic vendoring dry-run output)

generic_vendoring_after

vendor.yaml, refer to the respective test case for detaails

generic vendor yaml

why

Received feedback from in the parent PR

references

Upscopes from #1061 and

Summary by CodeRabbit

  • New Features

    • Enhanced vendoring operations now support both SSH and HTTPS sources with secure token injection and improved CLI logging for clearer operation feedback.
    • Introduced a utility for masking basic authentication credentials in URLs.
  • Documentation

    • Added detailed guides on SSH vendoring usage and updated configuration instructions to include new authentication options for Bitbucket and GitLab.
  • Chores

    • Updated CI workflows by performing full repository cloning and extending timeouts, and refined linter configurations to ensure consistency and reliability.
    • Added comprehensive test cases for vendoring operations and utility functions.

@Listener430 Listener430 changed the title Upgrade dry run mode for vendoring Update dry run mode for vendoring to show custom detectors vendoring URL parsing logs Feb 19, 2025
@Listener430 Listener430 changed the title Update dry run mode for vendoring to show custom detectors vendoring URL parsing logs Update dry run for atmos vendor pull to support ssh + detailed SCP urls alignment Feb 19, 2025
coderabbitai[bot]

This comment was marked as outdated.

coderabbitai[bot]
coderabbitai bot previously approved these changes Feb 19, 2025
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 0

🧹 Nitpick comments (4)
tests/fixtures/scenarios/vendor-pulls-ssh/vendor.yaml (2)

9-18: Basic HTTPS Source Configuration Review
The first source entry clearly documents a basic HTTPS URL (with token injection expected), and the structure is correct. Note that line 18 has trailing spaces that should be removed to satisfy YAML lint rules.

-    # Basic HTTPS default (token injection expected)␣␣
+    # Basic HTTPS default (token injection expected)
🧰 Tools
🪛 YAMLlint (1.35.1)

[error] 18-18: trailing spaces

(trailing-spaces)


19-27: Direct Credentials URL Configuration Caution
The second source entry uses direct credentials in the URL. While this may be intentional for testing or specific scenarios, please verify that exposing credentials in configuration is acceptable for your environment.

tests/test-cases/demo-vendoring.yaml (2)

61-79: Environment Variable Formatting Consistency
The "atmos vendor pull ssh component" test case is sound in its intent. However, notice that the ATMOS_GITHUB_TOKEN value is provided without quotes here while quoted in the SSH test case. For maintainability and consistency, consider using the same quoting style for environment variables across test cases.


96-96: Remove Trailing Whitespace
Static analysis has detected trailing spaces on these lines. Removing these extraneous spaces helps maintain clean YAML formatting.

-      - "Injecting token" 
+      - "Injecting token"
-      - "!supersecret"  
+      - "!supersecret"

Also applies to: 98-98

🧰 Tools
🪛 YAMLlint (1.35.1)

[error] 96-96: trailing spaces

(trailing-spaces)

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between be0d38a and 9091ecf.

📒 Files selected for processing (5)
  • internal/exec/vendor_utils.go (1 hunks)
  • tests/fixtures/scenarios/vendor-pulls-ssh/vendor.yaml (1 hunks)
  • tests/snapshots/TestCLICommands_atmos_vendor_pull_custom_detector_credentials_leakage.stderr.golden (1 hunks)
  • tests/snapshots/TestCLICommands_atmos_vendor_pull_ssh.stderr.golden (1 hunks)
  • tests/test-cases/demo-vendoring.yaml (1 hunks)
✅ Files skipped from review due to trivial changes (1)
  • internal/exec/vendor_utils.go
🚧 Files skipped from review as they are similar to previous changes (2)
  • tests/snapshots/TestCLICommands_atmos_vendor_pull_custom_detector_credentials_leakage.stderr.golden
  • tests/snapshots/TestCLICommands_atmos_vendor_pull_ssh.stderr.golden
🧰 Additional context used
🪛 YAMLlint (1.35.1)
tests/test-cases/demo-vendoring.yaml

[error] 96-96: trailing spaces

(trailing-spaces)


[error] 98-98: trailing spaces

(trailing-spaces)

tests/fixtures/scenarios/vendor-pulls-ssh/vendor.yaml

[error] 18-18: trailing spaces

(trailing-spaces)

⏰ Context from checks skipped due to timeout of 90000ms (4)
  • GitHub Check: [localstack] demo-localstack
  • GitHub Check: Acceptance Tests (macos-latest, macos)
  • GitHub Check: Acceptance Tests (windows-latest, windows)
  • GitHub Check: Summary
🔇 Additional comments (4)
tests/fixtures/scenarios/vendor-pulls-ssh/vendor.yaml (2)

1-8: File Header Configuration Looks Good
The top-level keys such as apiVersion, kind, and metadata (with name and description) are clearly defined and follow the expected schema for an Atmos vendoring manifest.


28-36: SSH Implicit Source Configuration Validated
The third source entry correctly represents an SSH implicit method, aligning with the enhanced SCP and SSH URL support highlighted in the PR objectives. The structure is consistent with the other components, ensuring that the custom detectors in the codebase can correctly process these URLs.

tests/test-cases/demo-vendoring.yaml (2)

43-60: SSH Dry-Run Test Case Validation
This test case correctly exercises the dry-run mode for vendor pulls using an SSH-style URL. The use of detailed logging (via --logs-level=Debug) and the dry-run flag ensures that no actual file downloads occur. The work directory and environment variable setup align with the PR objectives.


80-101: Credential Masking Verification
This test case does an excellent job ensuring that injected credentials do not appear in log outputs by checking for a masked output (e.g., "!supersecret") and an indicative log message ("Injecting token"). The expected stdout and stderr values properly cover the security requirement regarding credential leakage.

🧰 Tools
🪛 YAMLlint (1.35.1)

[error] 96-96: trailing spaces

(trailing-spaces)


[error] 98-98: trailing spaces

(trailing-spaces)

coderabbitai[bot]

This comment was marked as outdated.

coderabbitai[bot]
coderabbitai bot previously approved these changes Mar 17, 2025
coderabbitai[bot]
coderabbitai bot previously approved these changes Mar 17, 2025
@cloudposse cloudposse deleted a comment from mergify bot Mar 17, 2025
@cloudposse cloudposse deleted a comment from mergify bot Mar 17, 2025
@cloudposse cloudposse deleted a comment from coderabbitai bot Mar 17, 2025
Copy link
Contributor

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (13)
tests/fixtures/scenarios/vendor-creds-sanitize/atmos.yaml (1)

13-13: Remove trailing whitespace.

There's trailing whitespace on this line that should be removed.

-  
+
🧰 Tools
🪛 YAMLlint (1.35.1)

[error] 13-13: trailing spaces

(trailing-spaces)

tests/fixtures/scenarios/vendor-pulls-ssh/vendor.yaml (1)

1-25: Minor typo in comment.

There's a small typo in line 17's comment.

-    ## Explicit ssh vednoring (the schema is explicitly spcified along with a username, no custom detector is invoked)
+    ## Explicit ssh vendoring (the schema is explicitly specified along with a username, no custom detector is invoked)
tests/fixtures/scenarios/vendoring-dry-run/atmos.yaml (1)

1-23: Fix YAML formatting issues.

There are a few formatting issues in this YAML file that should be addressed.

-
 base_path: "./"
 settings:
   inject_github_token: true

 components:
   terraform:
     base_path: "components/terraform"
     apply_auto_approve: false
     deploy_run_init: true
     init_run_reconfigure: true
     auto_generate_backend_file: false
-  
+
 stacks:
   base_path: "stacks"
   included_paths:
     - "deploy/**/*"
   excluded_paths:
     - "**/_defaults.yaml"
   name_pattern: "{stage}"
-

-
+
🧰 Tools
🪛 YAMLlint (1.35.1)

[warning] 1-1: too many blank lines

(1 > 0) (empty-lines)


[error] 13-13: trailing spaces

(trailing-spaces)


[warning] 22-22: too many blank lines

(2 > 0) (empty-lines)

tests/fixtures/scenarios/vendor-pulls-ssh/atmos.yaml (1)

1-20: Add newline at end of file.

The YAML file is missing a newline character at the end.

 logs:
   file: "/dev/stderr"
   level: Info
+
🧰 Tools
🪛 YAMLlint (1.35.1)

[error] 20-20: no new line character at the end of file

(new-line-at-end-of-file)

tests/fixtures/scenarios/vendor-creds-sanitize/vendor.yaml (2)

18-18: Fix trailing whitespace

Remove the trailing spaces at the end of this line.

-    # Direct credentials provided in the URL (token injection should be skipped)
+    # Direct credentials provided in the URL (token injection should be skipped)
🧰 Tools
🪛 YAMLlint (1.35.1)

[error] 18-18: trailing spaces

(trailing-spaces)


28-35: Add newline at end of file

Add a newline character at the end of the file to comply with YAML standards.

      tags:
        - demo
+
🧰 Tools
🪛 YAMLlint (1.35.1)

[error] 35-35: no new line character at the end of file

(new-line-at-end-of-file)

website/docs/cli/commands/vendor/vendor-pull.mdx (1)

216-218: Consider refining the wording and punctuation.

You might strengthen the verb choice in "When resolving Git sources, Atmos follows these rules" by saying “When processing Git sources, Atmos follows these rules.” Also, using a typographical ellipsis character “…” rather than “...” can be more stylistically consistent.

🧰 Tools
🪛 LanguageTool

[style] ~216-~216: Consider using a different verb to strengthen your wording.
Context: ...lved When resolving Git sources, Atmos follows these rules: 1. If a *full HTTPS URL...

(FOLLOW_OBEY)


[style] ~218-~218: Consider using the typographical ellipsis character here instead.
Context: ...1. If a full HTTPS URL is provided (git::https://github.com/...), it is used as-is. No token data is i...

(ELLIPSIS)

tests/test-cases/vendoring-ssh-dryrun.yaml (1)

57-59: Remove trailing spaces and add a newline at EOF.

YAMLLint flagged trailing spaces on line 57 and requires a newline at the end of the file on line 59. Please ensure both are fixed:

      - !not 'supersecret'
-      - !not 'ATMOS_GITHUB_TOKEN'  
\ No newline at end of file
+      - !not 'ATMOS_GITHUB_TOKEN'
+
🧰 Tools
🪛 YAMLlint (1.35.1)

[error] 57-57: trailing spaces

(trailing-spaces)


[error] 59-59: no new line character at the end of file

(new-line-at-end-of-file)

website/docs/cli/configuration/configuration.mdx (2)

681-683: Minor grammar fixes.

  • Lowercase "Requests" for consistency in “Unauthenticated requests”.
  • Possibly clarify “avoided” vs. “used” regarding rate limits.
- | ATMOS_BITBUCKET_TOKEN                                 | N/A                                             | App password for Bitbucket API requests is set to avoid rate limits. Unauthenticated Requests are limited to 60 requests per hour across all API resources.                                                                  |
+ | ATMOS_BITBUCKET_TOKEN                                 | N/A                                             | App password for Bitbucket API requests, helping avoid rate limits. Unauthenticated requests are limited to 60 requests per hour across all API resources.                                                                  |

699-699: Typographical correction.

Fix the spelling from “comamnds” to “commands”:

- | ATMOS_TERRAFORM_WORKSPACE | The name of the Terraform workspace in which Terraform comamnds should be run                        |
+ | ATMOS_TERRAFORM_WORKSPACE | The name of the Terraform workspace in which Terraform commands should be run                        |
internal/exec/go_getter_utils_test.go (3)

118-137: Token injection test could be more robust.

While the test correctly verifies the basic token injection, consider adding cases for empty tokens or different host scenarios to ensure comprehensive coverage.

 func TestInjectToken(t *testing.T) {
 	os.Setenv("GITHUB_TOKEN", "testtoken")
 	defer os.Unsetenv("GITHUB_TOKEN")
 	config := fakeAtmosConfig(true)
 	detector := &CustomGitDetector{AtmosConfig: &config}
 	uObj, err := url.Parse("https://github.com/user/repo.git")
 	if err != nil {
 		t.Fatalf("Failed to parse URL: %v", err)
 	}
 	detector.injectToken(uObj, hostGitHub)
 	if uObj.User == nil {
 		t.Error("Expected token to be injected into URL")
 	} else {
 		user := uObj.User.Username()
 		if user != getDefaultUsername(hostGitHub) {
 			t.Errorf("Expected username %s, got %s", getDefaultUsername(hostGitHub), user)
 		}
 	}
+
+	// Test with empty token
+	os.Unsetenv("GITHUB_TOKEN")
+	uObj2, _ := url.Parse("https://github.com/user/repo.git")
+	detector.injectToken(uObj2, hostGitHub)
+	if uObj2.User != nil {
+		t.Error("Expected no token injection with empty token")
+	}
+
+	// Test with non-GitHub host
+	os.Setenv("GITLAB_TOKEN", "gltoken")
+	defer os.Unsetenv("GITLAB_TOKEN")
+	uObj3, _ := url.Parse("https://gitlab.com/user/repo.git")
+	detector.injectToken(uObj3, hostGitLab)
+	if uObj3.User == nil {
+		t.Error("Expected token to be injected for GitLab URL")
+	} else if uObj3.User.Username() != getDefaultUsername(hostGitLab) {
+		t.Errorf("Expected username %s, got %s", getDefaultUsername(hostGitLab), uObj3.User.Username())
+	}
 }

300-310: Consider consolidating redundant URI validation tests.

These test cases duplicate checks already present in the main TestValidateURI function (lines 37-42). Consider consolidating into a single comprehensive test to improve maintainability.


312-319: Consider consolidating redundant SCP URL tests.

This test duplicates functionality already tested in TestRewriteSCPURL (lines 98-102). Consolidating these would make the test suite more maintainable.

📜 Review details

Configuration used: .coderabbit.yaml
Review profile: CHILL
Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between 56a325b and 997f5a9.

📒 Files selected for processing (25)
  • .github/workflows/test.yml (3 hunks)
  • .golangci.yml (1 hunks)
  • codecov.yml (1 hunks)
  • internal/exec/go_getter_utils.go (4 hunks)
  • internal/exec/go_getter_utils_test.go (1 hunks)
  • internal/exec/vendor_model.go (2 hunks)
  • internal/exec/vendor_utils.go (2 hunks)
  • internal/exec/yaml_func_include.go (1 hunks)
  • pkg/utils/url_utils.go (1 hunks)
  • pkg/utils/url_utils_test.go (1 hunks)
  • tests/fixtures/scenarios/vendor-creds-sanitize/atmos.yaml (1 hunks)
  • tests/fixtures/scenarios/vendor-creds-sanitize/vendor.yaml (1 hunks)
  • tests/fixtures/scenarios/vendor-pulls-ssh/atmos.yaml (1 hunks)
  • tests/fixtures/scenarios/vendor-pulls-ssh/vendor.yaml (1 hunks)
  • tests/fixtures/scenarios/vendoring-dry-run/atmos.yaml (1 hunks)
  • tests/fixtures/scenarios/vendoring-dry-run/components/terraform/ipinfo/component.yaml (1 hunks)
  • tests/snapshots/TestCLICommands_atmos_vendor_pull_component_using_SSH.stderr.golden (1 hunks)
  • tests/snapshots/TestCLICommands_atmos_vendor_pull_custom_detector_credentials_leakage.stderr.golden (1 hunks)
  • tests/snapshots/TestCLICommands_atmos_vendor_pull_ssh.stderr.golden (1 hunks)
  • tests/snapshots/TestCLICommands_atmos_vendor_pull_ssh_component.stderr.golden (1 hunks)
  • tests/snapshots/TestCLICommands_atmos_vendor_pull_using_SSH.stderr.golden (1 hunks)
  • tests/snapshots/TestCLICommands_atmos_vendor_pull_with_custom_detector_and_handling_credentials_leakage.stderr.golden (1 hunks)
  • tests/test-cases/vendoring-ssh-dryrun.yaml (1 hunks)
  • website/docs/cli/commands/vendor/vendor-pull.mdx (2 hunks)
  • website/docs/cli/configuration/configuration.mdx (2 hunks)
🧰 Additional context used
🧠 Learnings (6)
internal/exec/vendor_model.go (1)
Learnt from: samtholiya
PR: cloudposse/atmos#1077
File: pkg/downloader/custom_github_detector.go:0-0
Timestamp: 2025-02-21T20:56:05.539Z
Learning: The `github.com/charmbracelet/log` package should be imported with the alias `log` according to the project's import alias configuration.
tests/snapshots/TestCLICommands_atmos_vendor_pull_component_using_SSH.stderr.golden (1)
Learnt from: Listener430
PR: cloudposse/atmos#1061
File: tests/snapshots/TestCLICommands_atmos_vendor_pull_ssh.stderr.golden:8-8
Timestamp: 2025-03-12T18:52:22.003Z
Learning: Test snapshots in the Atmos project, particularly for dry run scenarios, may be updated during the development process, and temporary inconsistencies in their content should not be flagged as issues.
tests/snapshots/TestCLICommands_atmos_vendor_pull_ssh.stderr.golden (1)
Learnt from: Listener430
PR: cloudposse/atmos#1061
File: tests/snapshots/TestCLICommands_atmos_vendor_pull_ssh.stderr.golden:8-8
Timestamp: 2025-03-12T18:52:22.003Z
Learning: Test snapshots in the Atmos project, particularly for dry run scenarios, may be updated during the development process, and temporary inconsistencies in their content should not be flagged as issues.
tests/snapshots/TestCLICommands_atmos_vendor_pull_ssh_component.stderr.golden (1)
Learnt from: Listener430
PR: cloudposse/atmos#1061
File: tests/snapshots/TestCLICommands_atmos_vendor_pull_ssh.stderr.golden:8-8
Timestamp: 2025-03-12T18:52:22.003Z
Learning: Test snapshots in the Atmos project, particularly for dry run scenarios, may be updated during the development process, and temporary inconsistencies in their content should not be flagged as issues.
tests/snapshots/TestCLICommands_atmos_vendor_pull_using_SSH.stderr.golden (1)
Learnt from: Listener430
PR: cloudposse/atmos#1061
File: tests/snapshots/TestCLICommands_atmos_vendor_pull_ssh.stderr.golden:8-8
Timestamp: 2025-03-12T18:52:22.003Z
Learning: Test snapshots in the Atmos project, particularly for dry run scenarios, may be updated during the development process, and temporary inconsistencies in their content should not be flagged as issues.
internal/exec/go_getter_utils.go (2)
Learnt from: Listener430
PR: cloudposse/atmos#1061
File: internal/exec/go_getter_utils.go:74-75
Timestamp: 2025-03-12T18:52:31.814Z
Learning: In the `CustomGitDetector.Detect` method of `internal/exec/go_getter_utils.go`, verbose debug logging of raw URLs is intentionally kept for debugging purposes, despite potential credential exposure risks.
Learnt from: samtholiya
PR: cloudposse/atmos#1077
File: pkg/downloader/custom_github_detector.go:0-0
Timestamp: 2025-02-21T20:56:05.539Z
Learning: The `github.com/charmbracelet/log` package should be imported with the alias `log` according to the project's import alias configuration.
🪛 YAMLlint (1.35.1)
tests/fixtures/scenarios/vendor-creds-sanitize/atmos.yaml

[warning] 1-1: too many blank lines

(1 > 0) (empty-lines)


[error] 13-13: trailing spaces

(trailing-spaces)

tests/fixtures/scenarios/vendor-pulls-ssh/atmos.yaml

[error] 20-20: no new line character at the end of file

(new-line-at-end-of-file)

tests/fixtures/scenarios/vendoring-dry-run/atmos.yaml

[warning] 1-1: too many blank lines

(1 > 0) (empty-lines)


[error] 13-13: trailing spaces

(trailing-spaces)


[warning] 22-22: too many blank lines

(2 > 0) (empty-lines)

tests/test-cases/vendoring-ssh-dryrun.yaml

[error] 57-57: trailing spaces

(trailing-spaces)


[error] 59-59: no new line character at the end of file

(new-line-at-end-of-file)

tests/fixtures/scenarios/vendor-creds-sanitize/vendor.yaml

[error] 18-18: trailing spaces

(trailing-spaces)


[error] 35-35: no new line character at the end of file

(new-line-at-end-of-file)

🪛 LanguageTool
website/docs/cli/commands/vendor/vendor-pull.mdx

[style] ~216-~216: Consider using a different verb to strengthen your wording.
Context: ...lved When resolving Git sources, Atmos follows these rules: 1. If a *full HTTPS URL...

(FOLLOW_OBEY)


[style] ~218-~218: Consider using the typographical ellipsis character here instead.
Context: ...1. If a full HTTPS URL is provided (git::https://github.com/...), it is used as-is. No token data is i...

(ELLIPSIS)

⏰ Context from checks skipped due to timeout of 90000ms (2)
  • GitHub Check: Build (windows-latest, windows)
  • GitHub Check: Summary
🔇 Additional comments (42)
.golangci.yml (1)

173-173: Increased nestif complexity threshold

Raising the min-complexity threshold from 4 to 6 will reduce noise in linting output by only flagging more complex nested if statements.

codecov.yml (1)

16-16: Improved boolean type representation

Changed from string representation yes to proper boolean value true. This is a good practice for YAML configuration files.

internal/exec/yaml_func_include.go (1)

50-50: Changed parameter passing to use pointer

Updated to pass atmosConfig by reference instead of by value, which aligns with the function signature in go_getter_utils.go that expects a pointer parameter.

tests/snapshots/TestCLICommands_atmos_vendor_pull_ssh.stderr.golden (1)

1-10: New test snapshot for SSH vendor pull dry run

The snapshot captures the expected CLI output for the vendor pull command when using SSH, including debug logs, warnings, and status messages. This supports the PR objective of enhancing dry run output with more detailed information.

I note this snapshot shows proper sanitized output including detailed information about vendored components without downloading files, which directly addresses the objectives of this PR.

internal/exec/vendor_model.go (2)

15-15: Good job fixing the import alias.

The charmbracelet/log package is now correctly imported with the log alias, following the project's import alias configuration.


337-337: Improved debugging visibility.

Adding this debug log provides better visibility into the package installation process, which is valuable for troubleshooting and aligns with the PR's goal to enhance output information.

tests/fixtures/scenarios/vendor-creds-sanitize/atmos.yaml (1)

1-21: Good test configuration for credential sanitization.

This configuration file properly sets up a test scenario with GitHub token injection enabled, which will be useful for testing the sanitization features in the dry run output.

🧰 Tools
🪛 YAMLlint (1.35.1)

[warning] 1-1: too many blank lines

(1 > 0) (empty-lines)


[error] 13-13: trailing spaces

(trailing-spaces)

tests/snapshots/TestCLICommands_atmos_vendor_pull_component_using_SSH.stderr.golden (1)

1-11: Good test snapshot for SSH vendoring.

This snapshot correctly captures the expected CLI output for the SSH vendoring dry run scenario, showing proper debug messages, warnings, and confirmation of the dry run completion.

pkg/utils/url_utils.go (2)

8-8: Good constant definition for masking secrets.

Using a named constant for the masked value follows good practice and addresses the previous feedback to move the string to a constant.


10-26: Well-implemented URL sanitization function.

The MaskBasicAuth function properly handles URL parsing, credential detection, and masking. It correctly differentiates between URLs with username-only and username-password combinations.

internal/exec/vendor_utils.go (2)

22-23: LGTM - Adding dedicated stderr logger helps separate detailed messaging.

The new StderrLogger ensures that detailed messaging, specifically for files vendoring, doesn't pollute stdout. This aligns well with improving the dry run output clarity.


514-514: Clean implementation of stderr logging.

Good replacement of standard log with the dedicated StderrLogger for including paths. This ensures detailed file inclusion messages go to stderr, keeping stdout clean as intended.

tests/fixtures/scenarios/vendor-pulls-ssh/vendor.yaml (2)

9-16: Good implementation of implicit SSH vendoring.

The implicit SSH vendoring configuration correctly uses SCP-style Git URL format ([email protected]:cloudposse/terraform-null-label.git). This addresses the need for SCP URL alignment in the dry run output as mentioned in the PR objectives.


17-24: Explicit SSH vendoring looks correct.

The explicit SSH vendoring configuration uses the proper schema (ssh://) with a username. This provides a good comparison case to the implicit SSH vendoring and ensures comprehensive testing of SSH support.

tests/fixtures/scenarios/vendoring-dry-run/atmos.yaml (1)

4-4: Good addition of GitHub token injection setting.

The inject_github_token: true setting is important for the vendoring functionality being tested, especially for dry run output that needs to show sanitized credentials.

tests/fixtures/scenarios/vendor-pulls-ssh/atmos.yaml (1)

18-20: Appropriate log configuration for testing.

Setting logs to stderr with Info level is a good choice for this scenario. It ensures that logs don't interfere with test outputs while maintaining visibility of important information during tests.

🧰 Tools
🪛 YAMLlint (1.35.1)

[error] 20-20: no new line character at the end of file

(new-line-at-end-of-file)

tests/snapshots/TestCLICommands_atmos_vendor_pull_using_SSH.stderr.golden (1)

11-13: Dry run output structure looks good

The output clearly shows what would happen in actual execution (vendoring 2 components) after confirming no actual components were vendored during dry run. This separation helps users understand the expected actions without making changes to the filesystem.

tests/fixtures/scenarios/vendor-creds-sanitize/vendor.yaml (1)

9-35: Well-structured test scenarios for credential handling

Good job defining three distinct scenarios for credential handling:

  1. Default HTTPS with token injection
  2. Direct credentials in URL (skipping token injection)
  3. Pre-existing credentials (skipping token injection)

This comprehensive test file will effectively validate the credential sanitization features mentioned in the PR objectives.

🧰 Tools
🪛 YAMLlint (1.35.1)

[error] 18-18: trailing spaces

(trailing-spaces)


[error] 35-35: no new line character at the end of file

(new-line-at-end-of-file)

tests/snapshots/TestCLICommands_atmos_vendor_pull_with_custom_detector_and_handling_credentials_leakage.stderr.golden (1)

8-12: Excellent credential handling and logging

The debug logs provide detailed visibility into the URL transformation process while properly masking sensitive credentials ("xxx:xxx"). This achieves the PR objective of showing detailed information about injected tokens and URL transformations without exposing sensitive data.

tests/snapshots/TestCLICommands_atmos_vendor_pull_ssh_component.stderr.golden (2)

8-12: Successfully implements SCP-style SSH URL rewriting

The logs clearly demonstrate the key feature of this PR - rewriting SCP-style SSH URLs (github.com:cloudposse/atmos.git) to standard SSH URLs (ssh://github.com/cloudposse/atmos.git). The credential injection and masking are also correctly handled, fulfilling the PR's objective of supporting SSH in vendor pull commands.


14-16: Dry run output structure is clear

Similar to the first file, the output effectively separates what actually happened (dry run with no actual vendoring) from what would happen in a real execution (vendoring 1 component). This structure helps users understand the expected actions without making filesystem changes.

.github/workflows/test.yml (3)

49-50: Added full Git history checkout to ensure complete repository access.

The change to fetch-depth: 0 ensures the entire Git history is available during workflow execution, which is essential for comprehensive Git-related operations like the vendor pull command with SSH support.


88-88: Increased timeout to allow for more complex Git operations.

The timeout increase from 15 to 20 minutes provides adequate time for the enhanced vendor pull functionality to complete successfully, particularly with SSH and URL processing.


159-159: Extended acceptance test timeout to accommodate new vendor pull functionality.

Increasing the test timeout to 20 minutes ensures tests have sufficient time to execute, especially when testing various Git URL formats and SSH handling.

tests/snapshots/TestCLICommands_atmos_vendor_pull_custom_detector_credentials_leakage.stderr.golden (3)

7-11: Properly masks credentials in Git URLs during dry run.

The logs correctly show the custom Git detector in action with appropriate credential masking. The URL transformation process is well-documented in the logs:

  1. Detection of Git URL
  2. Default scheme application
  3. Token injection
  4. Credential masking (showing "xxx:xxx" instead of actual tokens)

This is exactly what we want to see for secure credential handling in dry run mode.


32-32: Log entry incorrectly identifies directory as file.

The debug message refers to "docs" as a file, but it's actually a directory.


87-92: Clear success indicators for vendored components.

The output properly indicates successful vendoring of each component with version information, making it easy to verify the operation's results.

pkg/utils/url_utils_test.go (1)

1-49: Comprehensive test coverage for URL credential masking.

The tests thoroughly cover all important scenarios for the MaskBasicAuth function:

  • URLs with both username and password
  • URLs with username only
  • URLs without credentials
  • Invalid URL handling

Each test verifies the expected output and proper error handling, ensuring the URL masking functionality is reliable.

tests/fixtures/scenarios/vendoring-dry-run/components/terraform/ipinfo/component.yaml (2)

7-9: SSH URL format correctly implemented for Git source.

The configuration properly uses the SSH format ([email protected]) for the Git URI, which aligns with the PR objective of supporting SSH URLs in the vendor pull command.


10-13: Appropriate file inclusion patterns for Terraform components.

The included paths cover all essential Terraform files (.tf), variable files (.tfvars), and documentation (*.md), ensuring that all necessary component files are vendored.

website/docs/cli/commands/vendor/vendor-pull.mdx (2)

111-120: Good addition for SSH vendoring!

The new "Vendoring from SSH" section is clear and helps users understand how SSH-based repository access is configured.


150-156: Clarity improvement acknowledged.

The explanation about invalid URLs and Atmos defaulting to HTTPS is straightforward and addresses common user pitfalls.

internal/exec/go_getter_utils.go (3)

66-70: Struct definition looks good.

This new CustomGitDetector neatly encapsulates the Atmos config reference for advanced Git URL handling.


74-124: Useful custom detection logic.

The rewriting of SCP-style URLs and the fallback to https:// help unify various URL forms. Masking the source URL in debug logs further protects credentials.


345-359: Symlink removal approach is straightforward.

Recursively walking the downloaded directory and removing symlinks reduces security risks. This appears correct for ensuring only genuine files remain after the clone.

internal/exec/go_getter_utils_test.go (7)

30-52: Great job with comprehensive URI validation tests.

The test cases cover all essential validation paths: empty URIs, excessively long URIs, path traversal sequences, spaces, and various valid URI formats including OCI URIs. This thorough approach will help catch validation issues early.


55-65: Well-structured scheme validation tests.

You've properly tested both valid schemes (HTTP, HTTPS, Git, SSH, Git+HTTPS, Git+SSH) and invalid schemes (FTP). This ensures the function handles all expected inputs correctly.


68-86: Good coverage of URL scheme enforcement scenarios.

This test effectively verifies three key behaviors:

  1. Preserving existing schemes
  2. Rewriting SCP-style URLs to use SSH
  3. Defaulting to HTTPS when no scheme is present

These align well with the PR objective to support SSH and handle SCP URL alignment.


89-103: Solid test for SCP URL rewriting.

The test correctly verifies both positive and negative cases for SCP URL rewriting, which is central to the PR's goals of supporting SSH and SCP URL alignment.


194-221: Good platform-aware symlink testing.

The test correctly skips symlink tests on Windows where symlink behavior differs, creates a proper test environment, and verifies both the removal of symlinks and preservation of regular files.


223-258: Comprehensive file download test with proper cleanup.

This test thoroughly validates the file download functionality by creating temporary directories, writing test content, performing the download, and verifying the result. The Windows skip and proper cleanup with defer are good practices.


338-342: Good test cleanup practice.

Properly restoring original detectors in TestMain ensures tests don't interfere with each other or leave the environment in an altered state.

@cloudposse cloudposse deleted a comment from coderabbitai bot Mar 17, 2025
@@ -13,7 +13,7 @@ comment:
layout: "reach,diff,flags,tree" # Display different coverage views
behavior: default # Default PR comment behavior
require_changes: true # Only post if coverage changes
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
require_changes: true # Only post if coverage changes
require_changes: true # Only post if coverage changes

@@ -13,7 +13,7 @@ comment:
layout: "reach,diff,flags,tree" # Display different coverage views
behavior: default # Default PR comment behavior
require_changes: true # Only post if coverage changes
require_base: yes # Compare against base branch coverage
require_base: true # Compare against base branch coverage
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
require_base: true # Compare against base branch coverage
require_base: true # Compare against base branch coverage

@@ -19,6 +19,9 @@ import (
u "github.com/cloudposse/atmos/pkg/utils"
)

// Dedicated logger for stderr to keep stdout clean of detailed messaging, e.g. for files vendoring.
var StderrLogger = log.New(os.Stderr)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We do not need to introduce a new logger. We use log.* everywhere now without this.

@@ -508,7 +511,7 @@ func generateSkipFunction(tempDir string, s *schema.AtmosVendorSource) func(os.F
}

// If 'included_paths' is not provided, include all files that were not excluded
log.Debug("Including", u.TrimBasePathFromPath(tempDir+"/", src))
StderrLogger.Debug("Including", "path", u.TrimBasePathFromPath(tempDir+"/", src))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
StderrLogger.Debug("Including", "path", u.TrimBasePathFromPath(tempDir+"/", src))
log.Debug("Including", "path", u.TrimBasePathFromPath(tempDir+"/", src))

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
needs-cloudposse Needs Cloud Posse assistance no-release Do not create a new release (wait for additional code changes) size/xl
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants