pulumi up fails with "error: could not get AWS account ID: operation error STS: GetCallerIdentity" even though there are valid credential

[eriklz]

What happened?

I created a project with pulumi new aws-typescript, pnpm package manager.
After that I changed it to use the AWS Cloud Control provider.

Credentials activated via assume -x awsprofilename (tool https://granted.dev), an AWS SSO profile.

Running pulumi up failed:

❯ pulumi up
Previewing update (dev)

View in Browser (Ctrl+O): https://app.pulumi.com/elz/cred-validate/dev/previews/5aa196eb-3581-42e2-8aa9-7f166239a160

     Type                     Name               Plan       Info
 +   pulumi:pulumi:Stack      cred-validate-dev  create
     └─ aws-native:s3:Bucket  my-bucket                     1 error

Diagnostics:
  aws-native:s3:Bucket (my-bucket):
    error: could not get AWS account ID: operation error STS: GetCallerIdentity, get identity: get credentials: failed to refresh cached credentials, the SSO session has expired or is invalid

Running the original project with the "classic" AWS provider works fine in the same shell session and with the same credentials activated as used above.

Example

index.ts:

import * as aws from "@pulumi/aws-native";

const bucket = new aws.s3.Bucket("my-bucket");

export const bucketName = bucket.id;

package.json:

{
    "name": "cred-validate",
    "main": "index.ts",
    "devDependencies": {
        "@types/node": "^18",
        "typescript": "^5.0.0"
    },
    "dependencies": {
        "@pulumi/aws": "^6.0.0",
        "@pulumi/aws-native": "^1.17.0",
        "@pulumi/awsx": "^2.0.2",
        "@pulumi/pulumi": "^3.113.0"
    }
}

Environment variables starting with AWS that are set:

AWS_SESSION_TOKEN
AWS_DEFAULT_REGION
AWS_SESSION_EXPIRATION
AWS_ACCESS_KEY_ID
AWS_SECRET_ACCESS_KEY
AWS_CREDENTIAL_EXPIRATION
AWS_REGION
AWS_PROFILE

Output of pulumi about

❯ pulumi about
CLI
Version 3.143.0
Go Version go1.23.4
Go Compiler gc

Plugins
KIND NAME VERSION
resource aws 6.65.0
resource aws-native 1.17.0
resource awsx 2.19.0
resource docker 4.5.8
resource docker 3.6.1
language nodejs 3.143.0-dev.0

Host
OS darwin
Version 14.6.1
Arch arm64

This project is written in nodejs: executable='/Users/eriklz/.local/state/fnm_multishells/6144_1734648945463/bin/node' version='v22.11.0'

Current Stack: elz/cred-validate/dev

Found no resources associated with dev

Found no pending operations associated with dev

Backend
Name pulumi.com
URL https://app.pulumi.com/elz
User elz
Organizations elz
Token type personal

Pulumi locates its logs in /var/folders/ms/t731ppx52m56f1qgfhmt0tmm0000gn/T/ by default
warning: Failed to get information about the Pulumi program's dependencies: no package-lock.json or yarn.lock file found (searching upwards from /Users/eriklz/Documents/Dev/elz_repos/pulumi-cdk-tests/cred-validate)

Additional context

I first discovered this issue when trying pulumi-cdk, and then tested this above to see if the issue was with pulumi-cdk, or with the AWS Cloud Control provider. It seems to be with the AWS Cloud Control provider, hopefully addressing this will also resolve this issue for pulumi-cdk.

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[t0yv0]

Cannot seem to reproduce this easily unfortunately, I use aws sso login to activate an SSO session and that seems to work correctly for me in the current provider version. Does the error persist after you retry? Or refresh credentials?

I've checked the Go SDK verisons but it seems both pulumi/pulumi-aws-native and pulumi/pulumi-aws use the same version of Go SDK v2.

[eriklz]

If I use aws sso login --profile profilename it does work, but it does not work for the AWS cloud control provider if I use assume -x and select the profile name, only for the classic provider.

If I clear the credential environment variables using assume --un, after having used aws sso login successfully and then run assume -x again, it works for both providers. This makes sense, since the cached data from aws sso login is still in the SSO cache.

Looking at the cache data in ~/.aws/sso/cache, I notice that there is a difference in the cached data. The JSON object for the assume -xcase contains the fields "expiresAt", "clientId", "clientSecret".
In the aws sso logincase, the JSON object contains the fields "startUrl", "region", "accessToken", "expiresAt", "clientId", "clientSecret", "registrationExpiresAt".

So the cached SSO credential data and what is in the environment variables is sufficient for classic provider, but not the cloud control provider, so there is some difference.

[flostadler]

Hey @eriklz, I'm not familiar with granted, but the error message failed to refresh cached credentials, the SSO session has expired or is invalid makes me think that the AWS SDK assumes you're using a certain AWS profile. After looking into the granted code base on GitHub it looks like they're assuming a role via STS and then exporting the credentials via env variables (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN). So no profile should be in the mix here.

The explicitly defined credentials take precedence over the profile configuration (AWS_PROFILE) for AWS SDKs and the CLI. I tried to reproduce this by setting creds for an assumed role (AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_SESSION_TOKEN) and pointing AWS_PROFILE to a totally different role.
The Cloud Control provider indeed wrongly prefers AWS_PROFILE over explicit credentials.

The problem lies here:

pulumi-aws-native/provider/pkg/provider/provider.go

Lines 303 to 306 in 09d081f

	if profile, ok := varsOrEnv(vars, "aws-native:config:profile", "AWS_PROFILE"); ok {
	glog.V(4).Infof("using AWS profile: %q", profile)
	loadOptions = append(loadOptions, config.WithSharedConfigProfile(profile))
	} else {

If a profile is set, we're replacing the default credential chain by one hard coded to use the profile. I'll need to do some thinking around the implications of changing this (i.e. breaking changes).

@eriklz for now you could work around that by un-setting AWS_PROFILE. Would this work for you?

[flostadler]

After thinking a bit about it, I do not think we can correct this in a minor version and need to wait for the next major version. There's most definitely some users that depend on this behavior and making the change now would break their workflows.

For now we should emit a warning if both explicit creds and a profile are specified and hint at the behavior being changed in the next major version. Additionally we can introduce a provider configuration option (e.g. credentialOrder or preferExplicitCredentials) that could help users align the behavior between this provider and the rest of the AWS provider ecosystem.

[eriklz]

I am fine with the workaround, the AWS profile environment variable set by granted also, and is set to the same profile as the temporary credentials are set for.

Knowing why it happens and what to do about it is good enough for now, I think.