Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws_sagemaker: Stack level tags get added to SageMaker CfnModelPackage causing deployment failure #33699

Open
1 task
TeSheng93 opened this issue Mar 6, 2025 · 2 comments
Labels
@aws-cdk/aws-sagemaker Related to AWS SageMaker bug This issue is a bug. effort/medium Medium work item – several days of effort p2

Comments

@TeSheng93
Copy link

Describe the bug

When deploying a stack with tags set at the stack level, cdk tries to apply the tag to sagemaker model package which results in deployment error Tags are not supported in Model Package versions. Please add them to the Model Package Group.

The same code deployed without errors before this; the last successful deployment tested was in February 2025.

Regression Issue

  • Select this option if this issue appears to be a regression.

Last Known Working CDK Version

2.175.1 (same version no longer works now)

Expected Behavior

Tags should not be applied to untaggable resources as stated in the Stack class docstring:

Image

Current Behavior

CDK attempts to apply stack-level tags to sagemaker model package.

Reproduction Steps

from aws_cdk import App, Aws, Stack
from aws_cdk.aws_sagemaker import CfnModelPackage, CfnModelPackageGroup


class CustomStack(Stack):
    def __init__(self, scope: App, construct_id: str, tags: dict):
        super().__init__(scope, construct_id, tags=tags)

        model_package_group = CfnModelPackageGroup(self, "ModelPackageGroup", model_package_group_name="name")

        model_package = CfnModelPackage(
            self,
            "ModelPackage",
            model_package_group_name=model_package_group.model_package_group_name,
            model_package_version=1,
            inference_specification=CfnModelPackage.InferenceSpecificationProperty(
                containers=[
                    CfnModelPackage.ModelPackageContainerDefinitionProperty(
                        image=f"{Aws.ACCOUNT_ID}.dkr.ecr.{Aws.REGION}.amazonaws.com/test:latest",
                    )
                ],
                supported_content_types=["application/x-recordio"],
                supported_response_mime_types=["application/json"],
            ),
        )
        model_package.node.add_dependency(model_package_group)

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.175.1

Framework Version

No response

Node.js Version

2.175.1

OS

Linux

Language

Python

Language Version

No response

Other information

No response

@TeSheng93 TeSheng93 added bug This issue is a bug. needs-triage This issue or PR still needs to be triaged. labels Mar 6, 2025
@github-actions github-actions bot added the @aws-cdk/aws-sagemaker Related to AWS SageMaker label Mar 6, 2025
@pahud pahud self-assigned this Mar 6, 2025
@pahud
Copy link
Contributor

pahud commented Mar 6, 2025

When deploying a stack with tags set at the stack level, cdk tries to apply the tag to sagemaker model package which results in deployment error

Hi

I check the synthesized template and I'm not seeing any Tag property defined

  ModelPackageGroup:
    Type: AWS::SageMaker::ModelPackageGroup
    Properties:
      ModelPackageGroupName: name
    Metadata:
      aws:cdk:path: CustomStack/ModelPackageGroup
  ModelPackage:
    Type: AWS::SageMaker::ModelPackage
    Properties:
      InferenceSpecification:
        Containers:
          - Image:
              Fn::Join:
                - ""
                - - Ref: AWS::AccountId
                  - .dkr.ecr.
                  - Ref: AWS::Region
                  - .amazonaws.com/test:latest
        SupportedContentTypes:
          - application/x-recordio
        SupportedResponseMIMETypes:
          - application/json
      ModelPackageGroupName: name
      ModelPackageVersion: 1
    DependsOn:
      - ModelPackageGroup
    Metadata:
      aws:cdk:path: CustomStack/ModelPackage

But yes I do see this error

CustomStack | 2/4 | 3:27:42 PM | CREATE_FAILED | AWS::SageMaker::ModelPackage | ModelPackage Resource handler returned message: "Tags are not supported in Model Package versions. Please add them to the Model Package Group. (Service: SageMaker, Status Code: 400, Request ID: af60ccb4-96ce-41ed-b418-8428aac5ccb1) (SDK Attempt Count: 1)" (RequestToken: 16be2965-6b1e-28ac-d4dd-aa8cdb8a8744, HandlerErrorCode: GeneralServiceException)

Investigating

@pahud
Copy link
Contributor

pahud commented Mar 6, 2025

OK looks like stack-level tagging by default will propagate to all resources. To disable that and apply to stack only, you'll need to use the Tags API like this instead.

class CustomStack(Stack):
    def __init__(self, scope: App, construct_id: str):
        super().__init__(scope, construct_id, )
        
        Tags.of(self).add(
            key="my-key",
            value="my-value",
            include_resource_types=[]  # Empty list prevents propagation
        )

I am bringing this up to the engineering team for further inputs.

We need to figure out:

  • why CFN is trying to add Tag on AWS::SageMaker::ModelPackage when stack-level tags are defined and we can't see that in the synthesized CFN template.
  • obviously AWS::SageMaker::ModelPackage is taggable(see Tags prop) but why it's complaining the error.

@pahud pahud added p2 and removed needs-triage This issue or PR still needs to be triaged. labels Mar 6, 2025
@pahud pahud removed their assignment Mar 6, 2025
@pahud pahud added the effort/medium Medium work item – several days of effort label Mar 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
@aws-cdk/aws-sagemaker Related to AWS SageMaker bug This issue is a bug. effort/medium Medium work item – several days of effort p2
Projects
None yet
Development

No branches or pull requests

2 participants