Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow users to provide tags on upload #378

Merged
merged 2 commits into from
Mar 20, 2025
Merged

Conversation

WrightJosh
Copy link

Closes: #172

Description of changes:

This change adds the ability for users to pass in any number of tags during upload. The tags are passed as given to EBS start-snapshot. Tags are expected to be inputted as "Key=k,Value=v" where k and v are users desired tags. The key and value delimiters were chosen to adhere as much as possible to other AWS cli tag inputs. Any restrictions on tags are expected to be handled by the underlining EBS apis.

Testing

local testing performed with the following commands and validation of upload and metadata when expected using aws ec2 describe-snapshots --snapshot-id <<ID>>. I have removed fields from the describe commands that aren't useful for the validation of tags.

No tags provided

$ cargo run upload /tmp/coldsnapTestAMI.bin 
{
    "Snapshots": [
        {
            "SnapshotId": "snap-0d6c225070eded508",
            "State": "completed",
            "Progress": "100%",
            "Description": "coldsnapTestAMI.bin",
        }
    ]
}

$ cargo run upload /tmp/coldsnapTestAMI.bin --description "Test coldsnap tags: no tags"
{
    "Snapshots": [
        {
            "SnapshotId": "snap-033a94351b0f601c1",
            "VolumeId": "vol-ffffffff",
            "State": "completed",
            "Progress": "100%",
            "Description": "Test coldsnap tags: no tags",
        }
    ]
}

Single tag provided:

$ cargo run upload /tmp/coldsnapTestAMI.bin --description "Test coldsnap tags: single happy tag" --tag "Key=Test,Value=Hello"
{
    "Snapshots": [
        {
            "Tags": [
                {
                    "Key": "Test",
                    "Value": "Hello"
                }
            ],
            "SnapshotId": "snap-00b163873d33fd695",
            "VolumeId": "vol-ffffffff",
            "State": "completed",
            "Progress": "100%",
            "Description": "Test coldsnap tags: single happy tag",
        }
    ]
}

$ cargo run upload /tmp/coldsnapTestAMI.bin --description "Test coldsnap tags: special characters" --tag "Key=Test1+-=._:/@,,Value=Hello"
{
    "Snapshots": [
        {
            "Tags": [
                {
                    "Key": "Test1+-=._:/@,",
                    "Value": "Hello"
                }
            ],
            "SnapshotId": "snap-0ebd0652b188d5ad0",
            "VolumeId": "vol-ffffffff",
            "State": "completed",
            "Progress": "100%",
            "Description": "Test coldsnap tags: special characters",
        }
    ]
}

$ cargo run upload /tmp/coldsnapTestAMI.bin --description "Test coldsnap tags: empty value" --tag "Key=Test,Value= "
{
    "Snapshots": [
        {
            "Tags": [
                {
                    "Key": "Test",
                    "Value": " "
                }
            ],
            "SnapshotId": "snap-04acbc91193f94484",
            "VolumeId": "vol-ffffffff",
            "State": "completed",
            "Progress": "100%",
            "Description": "Test coldsnap tags: empty value",
        }
    ]
}

$ cargo run upload /tmp/coldsnapTestAMI.bin --description "Test coldsnap tags: key with space" --tag "Key=Test with space,Value=Hello"
{
    "Snapshots": [
        {
            "Tags": [
                {
                    "Key": "Test with space",
                    "Value": "Hello"
                }
            ],
            "SnapshotId": "snap-0a4b1d6557c23eb93",
            "VolumeId": "vol-ffffffff",
            "State": "completed",
            "Progress": "100%",
            "Description": "Test coldsnap tags: key with space",
        }
    ]
}

$ cargo run upload /tmp/coldsnapTestAMI.bin --description "Test coldsnap tags: Key= and Value= parsing allows themselves for input" --tag "Key=Key=,Value=,Value="
{
    "Snapshots": [
        {
            "Tags": [
                {
                    "Key": "Key=",
                    "Value": ",Value="
                }
            ],
            "SnapshotId": "snap-0f8312c736d941007",
            "VolumeId": "vol-ffffffff",
            "State": "completed",
            "Progress": "100%",
            "Description": "Test coldsnap tags: Key= and Value= parsing allows themselves for input",
        }
    ]
}

Multiple tags provided:

$ cargo run upload /tmp/coldsnapTestAMI.bin --description "Test coldsnap tags: multiple happy tags" --tag "Key=Test,Value=Hello" --tag "Key=Test2,Value=Hello2"
{
    "Snapshots": [
        {
            "Tags": [
                {
                    "Key": "Test",
                    "Value": "Hello"
                },
                {
                    "Key": "Test2",
                    "Value": "Hello2"
                }
            ],
            "SnapshotId": "snap-037581676b4e505a6",
            "VolumeId": "vol-ffffffff",
            "State": "completed",
            "Progress": "100%",
            "Description": "Test coldsnap tags: multiple happy tags",
        }
    ]
}

$ cargo run upload /tmp/coldsnapTestAMI.bin --description "Test coldsnap tags: case sensitive keys are unique" --tag "Key=Test,Value=Hello" --tag "Key=test,Value=Hello2"
{
    "Snapshots": [
        {
            "Tags": [
                {
                    "Key": "test",
                    "Value": "Hello2"
                },
                {
                    "Key": "Test",
                    "Value": "Hello"
                }
            ],
            "SnapshotId": "snap-01a853dec4ae8e16f",
            "VolumeId": "vol-ffffffff",
            "State": "completed",
            "Progress": "100%",
            "Description": "Test coldsnap tags: case sensitive keys are unique",
        }
    ]
}

Expected failure modes:

$ cargo run upload /tmp/coldsnapTestAMI.bin --description "Test coldsnap tags: tag key over 127 characters" --tag "Key=.A_,C/,=-=_1BB@ c,,bcC ,+a--3_.c._. -1C-.c3@@aa./C C-/A:B_B+13A= :a@@bbCA .-BC =C2b_-@,3@@3aC.A+ C=a,/ a/b1c+11/a+21C@c._AA133a-.A_,C/,=-=_1BB@ c,,bcC ,+a--3_.c._. -1C-.c3@@aa./C C-/A:B_B+13A= :a@@bbCA .-BC =C2b_-@,3@@3aC.A+ C=a,/ a/b1c+11/a+21C@c._AA133a-,Value=Hello"
Failed to upload snapshot: Failed to start snapshot: service error: ValidationException: 1 validation error detected: Value '.A_,C/,=-=_1BB@ c,,bcC ,+a--3_.c._. -1C-.c3@@aa./C C-/A:B_B+13A= :a@@bbCA .-BC =C2b_-@,3@@3aC.A+ C=a,/ a/b1c+11/a+21C@c._AA133a-.A_,C/,=-=_1BB@ c,,bcC ,+a--3_.c._. -1C-.c3@@aa./C C-/A:B_B+13A= :a@@bbCA .-BC =C2b_-@,3@@3aC.A+ C=a,/ a/b1c+11/a+21C@c._AA133a-' at 'tags.1.member.key' failed to satisfy constraint: Member must have length less than or equal to 127: ValidationException: 1 validation error detected: Value '.A_,C/,=-=_1BB@ c,,bcC ,+a--3_.c._. -1C-.c3@@aa./C C-/A:B_B+13A= :a@@bbCA .-BC =C2b_-@,3@@3aC.A+ C=a,/ a/b1c+11/a+21C@c._AA133a-.A_,C/,=-=_1BB@ c,,bcC ,+a--3_.c._. -1C-.c3@@aa./C C-/A:B_B+13A= :a@@bbCA .-BC =C2b_-@,3@@3aC.A+ C=a,/ a/b1c+11/a+21C@c._AA133a-' at 'tags.1.member.key' failed to satisfy constraint: Member must have length less than or equal to 127

$ cargo run upload /tmp/coldsnapTestAMI.bin --description "Test coldsnap tags: tag value over 255 characters" --tag "Key=Test,Value=.A_,C/,=-=_1BB@ c,,bcC ,+a--3_.c._. -1C-.c3@@aa./C C-/A:B_B+13A= :a@@bbCA .-BC =C2b_-@,3@@3aC.A+ C=a,/ a/b1c+11/a+21C@c._AA133a-.A_,C/,=-=_1BB@ c,,bcC ,+a--3_.c._. -1C-.c3@@aa./C C-/A:B_B+13A= :a@@bbCA .-BC =C2b_-@,3@@3aC.A+ C=a,/ a/b1c+11/a+21C@c._AA133a-.A_,C/,=-=_1BB@ c,,bcC ,+a--3_.c._. -1C-.c3@@aa./C C-/A:B_B+13A= :a@@bbCA .-BC =C2b_-@,3@@3aC.A+ C=a,/ a/b1c+11/a+21C@c._AA133a-"
Failed to upload snapshot: Failed to start snapshot: service error: ValidationException: 1 validation error detected: Value '.A_,C/,=-=_1BB@ c,,bcC ,+a--3_.c._. -1C-.c3@@aa./C C-/A:B_B+13A= :a@@bbCA .-BC =C2b_-@,3@@3aC.A+ C=a,/ a/b1c+11/a+21C@c._AA133a-.A_,C/,=-=_1BB@ c,,bcC ,+a--3_.c._. -1C-.c3@@aa./C C-/A:B_B+13A= :a@@bbCA .-BC =C2b_-@,3@@3aC.A+ C=a,/ a/b1c+11/a+21C@c._AA133a-.A_,C/,=-=_1BB@ c,,bcC ,+a--3_.c._. -1C-.c3@@aa./C C-/A:B_B+13A= :a@@bbCA .-BC =C2b_-@,3@@3aC.A+ C=a,/ a/b1c+11/a+21C@c._AA133a-' at 'tags.1.member.value' failed to satisfy constraint: Member must have length less than or equal to 255: ValidationException: 1 validation error detected: Value '.A_,C/,=-=_1BB@ c,,bcC ,+a--3_.c._. -1C-.c3@@aa./C C-/A:B_B+13A= :a@@bbCA .-BC =C2b_-@,3@@3aC.A+ C=a,/ a/b1c+11/a+21C@c._AA133a-.A_,C/,=-=_1BB@ c,,bcC ,+a--3_.c._. -1C-.c3@@aa./C C-/A:B_B+13A= :a@@bbCA .-BC =C2b_-@,3@@3aC.A+ C=a,/ a/b1c+11/a+21C@c._AA133a-.A_,C/,=-=_1BB@ c,,bcC ,+a--3_.c._. -1C-.c3@@aa./C C-/A:B_B+13A= :a@@bbCA .-BC =C2b_-@,3@@3aC.A+ C=a,/ a/b1c+11/a+21C@c._AA133a-' at 'tags.1.member.value' failed to satisfy constraint: Member must have length less than or equal to 255

$ cargo run upload /tmp/coldsnapTestAMI.bin --description "Test coldsnap tags: duplicate key " --tag "Key=Test,Value=Hello" --tag "Key=Test,Value=Hello"    
Failed to upload snapshot: Failed to start snapshot: service error: ValidationException: Duplicate tag key 'Test' specified.: ValidationException: Duplicate tag key 'Test' specified.

$ cargo run upload /tmp/coldsnapTestAMI.bin --description "Test coldsnap tags: tag is empty string" --tag ""
Error parsing option '--tag' with value '': Tag inputs must start with 'Key='

Run coldsnap --help for more information.

$ cargo run upload /tmp/coldsnapTestAMI.bin --description "Test coldsnap tags: tag doesn't contain ',Value='" --tag "Key=TestValue=Hello"
Error parsing option '--tag' with value 'Key=TestValue=Hello': Tag inputs must contain value entry with ',Value='

Run coldsnap --help for more information.

$ cargo run upload /tmp/coldsnapTestAMI.bin --description "Test coldsnap tags: tag doesn't start with 'Key='" --tag "FailKey=Test,Value=Hello"
Error parsing option '--tag' with value 'FailKey=Test,Value=Hello': Tag inputs must start with 'Key='

Run coldsnap --help for more information.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

This change adds the ability for users to pass in any number of
tags during upload. The tags are passed as given to EBS start-snapshot.
Tags are expected to be inputted as "Key=k,Value=v" where k and v
are users desired tags. The key and value delimiters were chosen
to adhere as much as possible to other AWS cli tag inputs. Any
restrictions on tags are expected to be handled by the underlining
EBS apis.
Copy link
Contributor

@bcressey bcressey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM with a couple of minor notes.

Please run cargo fmt or else the CI job will fail when I kick that off.

Comment on lines 287 to 289
let k_v: (&str, &str) = input.split_once(KEY_DELIMITER).unwrap().1.split_once(VALUE_DELIMITER).unwrap();

Ok(Tag::builder().key(k_v.0).value(k_v.1).build())
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should reject empty keys, since they lead to an error otherwise:

❯ cargo run -- upload --tag Key=,Value= --tag Key=A,Value=B Cargo.toml
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.10s
     Running `target/debug/coldsnap upload --tag Key=,Value= --tag Key=A,Value=B Cargo.toml`
Failed to upload snapshot: Failed to start snapshot: service error: ValidationException: Tag key cannot be null or empty.: ValidationException: Tag key cannot be null or empty.
Suggested change
let k_v: (&str, &str) = input.split_once(KEY_DELIMITER).unwrap().1.split_once(VALUE_DELIMITER).unwrap();
Ok(Tag::builder().key(k_v.0).value(k_v.1).build())
let (key, val) = input.split_once(KEY_DELIMITER).unwrap().1.split_once(VALUE_DELIMITER).unwrap();
if key.is_empty() {
return Err("Tag inputs must contain a non-empty key entry".to_string());
}
Ok(Tag::builder().key(key).value(val).build())

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice catch, I did miss this test above. I'm fine to make the change and do we want to do any of the other validations ourselves? The error spat was from letting EBS reject tag inputs so we don't have to lock step any changes to valid inputs over time. I'm letting the duplicate key and character limit fall through as well. The duplicate key would have to be done outside of the parsing function though.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we want to do any of the other validations ourselves?

I'm fine with it like this - seems unnecessary to repeat all the validations that the AWS SDK is going to perform. Even the non-empty key check is a tad excessive, but it's cheap to do and provides a better error message.

Updates tag parsing to validate non-empty keys
and tests for valid/invalid tags. Additionally
ran cargo fmt across previous tag changes.
@cbgbt cbgbt merged commit bab439b into awslabs:develop Mar 20, 2025
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support for Tags
6 participants