Error "InvalidParameterException: Invalid identifier: Identifier is for cluster..." when running docker compose up for AWS ECS

[jared-hess]

Description

Using AWS ECS, when running "docker compose up", creation of the Service fails with error "InvalidParameterException: Invalid identifier: Identifier is for cluster Test1. Your cluster is default". The cluster defined in the docker compose file is Test1.

Here's the command used to deploy the stack:

> docker -c aws-context compose -f test.yml up
[+] Running 13/13
 ⠿ jaredhess                 DeleteComplete                                                                                                                                                                                                         210.0s
 ⠿ WebTaskRole               DeleteComplete                                                                                                                                                                                                         161.0s
 ⠿ WebTaskExecutionRole      DeleteComplete                                                                                                                                                                                                         161.0s
 ⠿ CloudMap                  DeleteComplete                                                                                                                                                                                                         205.0s
 ⠿ WebTCP80TargetGroup       DeleteComplete                                                                                                                                                                                                         160.0s
 ⠿ DefaultNetwork            DeleteComplete                                                                                                                                                                                                         159.0s
 ⠿ LogGroup                  DeleteComplete                                                                                                                                                                                                         161.0s
 ⠿ WebTCP80Listener          DeleteComplete                                                                                                                                                                                                         157.0s
 ⠿ Default80Ingress          DeleteComplete                                                                                                                                                                                                          53.0s
 ⠿ DefaultNetworkIngress     DeleteComplete                                                                                                                                                                                                          53.0s
 ⠿ WebTaskDefinition         DeleteComplete                                                                                                                                                                                                         144.0s
 ⠿ WebServiceDiscoveryEntry  DeleteComplete                                                                                                                                                                                                         109.0s
 ⠿ WebService                DeleteComplete                                                                                                                                                                                                         104.0s
InvalidParameterException: Invalid identifier: Identifier is for cluster Test1. Your cluster is default

Here's the content of the docker compose file (we have to add the permissions boundary to the roles due to organization policy):

version: "3.8"

x-aws-vpc: "vpc-<redacted>"
x-aws-cluster: "Test1"
x-aws-loadbalancer: "arn:aws:elasticloadbalancing:us-west-2:<redacted>"

services:
  web:
    image: nginx
    ports:
      - "80:80"
    x-aws-policies:
      - "arn:aws:iam::aws:policy/service-role/<redacted>"

x-aws-cloudformation:
   Resources:
     WebTaskExecutionRole:
       Properties:
         PermissionsBoundary: "arn:aws:iam::<redacted>"
     WebTaskRole:
       Properties:
         PermissionsBoundary: "arn:aws:iam::<redacted>"

Output of docker-compose --version:

docker-compose version 1.29.2, build 5becea4c

Output of docker version:

Client:
 Cloud integration: 1.0.17
 Version:           20.10.8
 API version:       1.41
 Go version:        go1.16.6
 Git commit:        3967b7d
 Built:             Fri Jul 30 19:55:20 2021
 OS/Arch:           darwin/amd64
 Context:           aws-context
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.8
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.6
  Git commit:       75249d8
  Built:            Fri Jul 30 19:52:10 2021
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.4.9
  GitCommit:        e25210fe30a0a703442421b0f60afac609f950a3
 runc:
  Version:          1.0.1
  GitCommit:        v1.0.1-0-g4144b63
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker context show:

> docker context show
aws-context

> docker context inspect aws-context
[
    {
        "Name": "aws-context",
        "Metadata": {
            "Type": "ecs"
        },
        "Endpoints": {
            "docker": {
                "SkipTLSVerify": false
            },
            "ecs": {
                "Profile": "default"
            }
        },
        "TLSMaterial": {},
        "Storage": {
            "MetadataPath": "<redacted>",
            "TLSPath": "<redacted>"
        }
    }
]

Output of docker info (had to change context back to default):

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., v0.6.1-docker)
  compose: Docker Compose (Docker Inc., v2.0.0-rc.3)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 34
  Running: 13
  Paused: 0
  Stopped: 21
 Images: 86
 Server Version: 20.10.8
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e25210fe30a0a703442421b0f60afac609f950a3
 runc version: v1.0.1-0-g4144b63
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.10.47-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 1.938GiB
 Name: docker-desktop
 ID: QW3A:OLNX:63GE:36EZ:V6D5:ME4Q:JVWO:JIH7:I5BK:QRBT:7SFR:RHGP
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS ECS, Azure ACI, local, etc.):
AWS ECS

[jared-hess]

I'll also note that the service will deploy successfully by using "docker compose convert" to get a cloudformation template, then deploying the template with "aws cloudformation createstack".

[andiradulescu]

Same issue on our side, also.

But, if using docker compose up -d then it works.

[vinters768]

having the same issue

[ezpuzz]

Same issue here, only using the x-aws-cluster parameter in our config.

[christopher-haueter]

I'm having the same issue.

[mociarain]

Same here too

[CptLolliPants]

Same problem here. Using both x-aws-cluster and x-aws-vpc, or just use either result in the same problem

[imphatic]

Also having this issue.

[pbenefice]

Same error here, setting x-aws-cluster gives me the InvalidParameterException: Invalid identifier: Identifier is for cluster Demo-Apps. Your cluster is default error
What's weired is that it works when I run the same command with the -d option.

so docker --context "ecs" compose --file ecs.docker-compose.yaml up doesnt work
whereas docker --context "ecs" compose --file ecs.docker-compose.yaml up -d work (daemonized)

[joshua-davis1]

I'm also having this problem. Detaching seems to be the solution for now.

[adv4000]

Same here:
docker compose up error InvalidParameterException: Invalid identifier: Identifier is for cluster MyECS. Your cluster is default
docker compose up -d works fine.
Looks very buggy and basic alfa version, far away from ready to prod due to lack of customization.

[JohnPreston]

Hello all. For those of you still into using docker-compose to define your services and deploy to ECS, I'd suggest having a look at ECS Compose-X.

Specifically around this issue, the way Compose-X deals with x-cluster to define the ECS Cluster to use if very versatile. You can create a brand new one, or use an existing one.
When using Lookup to use an existing one (which supports cross-accounts lookup for those who want different accounts in CICD), it will also collect settings about the cluster configuration, for example, capacity providers.

If your cluster did not have the FARGATE capacity providers for example, it would autocorrect/override the deployment settings to be compatible with the available runtime. Equally, compose-x allows you to define the compute platform to use for each services individually (see these for reference).

It also supports to set services to run on ECS Anywhere for those of you who would be running ECS instances outside of AWS regions :)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[KMontag42]

I'm also having this issue. Don't really want to add yet another compose plugin into my project (sorry Compose-X). Any idea of if this will get fixed?

[JohnPreston]

I'm also having this issue. Don't really want to add yet another compose plugin into my project (sorry Compose-X). Any idea of if this will get fixed?

No worries :) Good luck with your ECS Journey

[stale]

This issue has been automatically marked as not stale anymore due to the recent activity.

[rubbo898]

Same here!!

[KMontag42]

bumping again. @gtardif @ndeloof @rumpl any chance of this getting looked at?

[besil]

same problem here

[fsodre]

Same

[Dasky14]

Same issue here. Cannot run detached either since it is part of a CI/CD pipeline.

[austindrenski]

FYI I've opened a patch for this via #2269, but given the imminent EOL (#2258, docker/compose-ecs#7), it will (most likely) not be merged in this repository, so I've opened docker/compose-ecs#19 to track this issue, and docker/compose-ecs#20 to patch it in the new repo.