tfe ignores token = parameter, only accepts TFE_TOKEN in HCP Terraform workspace variables

[jeffhuenemann]

Terraform Enterprise version

HCP Terraform

Terraform version

1.9.8

Terraform Configuration Files

terraform.tf:

terraform {
  required_providers {
    tfe = {
      source  = "hashicorp/tfe"
      version = "~> 0.60.1"
    }
  }
  cloud {
    organization = "our-org"
    workspaces {
      project = "our-project"
      name    = "the-workspace-for-this-codebase"
    }
  }
}

provider "tfe" {
  organization = "our-org"
  token        = var.TFE_TOKEN
}

main.tf:

# Here's a sample for the sake of having one, but the details of the code are not what's important here
data "tfe_project" "<name>" {
  name = "<name>"
}

resource "tfe_workspace" "workspaces_setup" {
  name               = "name-of-the-workspace" # Static
  description        = "Houses resources and state for setting up the other Workspaces in this Project"
  # <other properties>
}

Debug Output

2024-11-21T14:10:57.328Z [DEBUG] ReferenceTransformer: "data.tfe_project.<name>" references: []
2024-11-21T14:10:57.328Z [DEBUG] expandResourceImports: skipping import address tfe_workspace.workspaces_setup already in state
2024-11-21T14:10:57.328Z [DEBUG] ReferenceTransformer: "tfe_workspace.workspaces_setup" references: []
2024-11-21T14:10:57.335Z [DEBUG] provider.terraform-provider-tfe_v0.60.1_x5: [DEBUG] Read configuration of workspace: ws-1234567890abcdef
2024-11-21T14:10:57.441Z [ERROR] provider.terraform-provider-tfe_v0.60.1_x5: Response contains error diagnostic: @module=sdk.proto diagnostic_severity=ERROR diagnostic_summary="Error reading configuration of workspace ws-1234567890abcdef: unauthorized" tf_provider_addr=registry.terraform.io/hashicorp/tfe tf_req_id=ba97f581-c858-05cc-b2d9-7955bd134a61 tf_resource_type=tfe_workspace @caller=github.com/hashicorp/[email protected]/tfprotov5/internal/diag/diagnostics.go:58 diagnostic_detail="" tf_proto_version=5.4 tf_rpc=ReadResource timestamp=2024-11-21T14:10:57.441Z
2024-11-21T14:10:57.442Z [ERROR] vertex "tfe_workspace.workspaces_setup" error: Error reading configuration of workspace ws-1234567890abcdef: unauthorized
2024-11-21T14:10:57.442Z [ERROR] vertex "tfe_workspace.workspaces_setup (expand)" error: Error reading configuration of workspace ws-1234567890abcdef: unauthorized
2024-11-21T14:10:57.444Z [ERROR] provider.terraform-provider-tfe_v0.60.1_x5: Response contains error diagnostic: diagnostic_severity=ERROR tf_data_source_type=tfe_project tf_proto_version=5.4 @caller=github.com/hashicorp/[email protected]/tfprotov5/internal/diag/diagnostics.go:58 @module=sdk.proto diagnostic_detail="" diagnostic_summary="Error retrieving projects: unauthorized" tf_provider_addr=registry.terraform.io/hashicorp/tfe tf_req_id=fa696e03-4312-b51a-b4aa-936ee34fa065 tf_rpc=ReadDataSource timestamp=2024-11-21T14:10:57.443Z
2024-11-21T14:10:57.444Z [ERROR] vertex "data.tfe_project.<name>" error: Error retrieving projects: unauthorized
2024-11-21T14:10:57.444Z [ERROR] vertex "data.tfe_project.<name> (expand)" error: Error retrieving projects: unauthorized
2024-11-21T14:10:57.445Z [WARN]  Planning encountered errors, so plan is not applyable

Expected Behavior

In normal "Remote" execution on HCP Terraform, the provider should have used the token being explicitly passed in the provider config:

provider "tfe" {
  organization = "our-org"
  token        = var.TFE_TOKEN
}

According to the tfe provider docs, Authentication section, the provider should accept the token either by explicitly configuring as above, or by having it in the TFE_TOKEN environment variable in the run.

There are several ways to provide the required token:

• Set the token argument in the provider configuration. You can set the token argument in the provider configuration. Use an input variable for the token.
• Set the TFE_TOKEN environment variable: The provider can read the TFE_TOKEN environment variable and the token stored there to authenticate.
  When configuring the input variable for either of these options, mark them as sensitive.

Note:
If you are using this provider in HCP Terraform or Terraform Enterprise, you will need to use one of the two options above, even if you're using the remote backend with remote operations and the CLI-driven Run workflow.

If passed in to the provider config as a variable, it should accept a variable value in normal ways, including passing in a .tfvars file, Using a TF_VAR_*** environment variable, or -var at the CLI that kicks off the run. When this codebase is used without the cloud { } block during local execution, this works as the docs say it should.

Actual Behavior

When the run kicks off inside the remote runner, tfe provider is either failing to use the variable-provided token at all, or a token provided for the run is "winning" and being used, but doesn't have access to read other resources from HCP Terraform. Whatever token the provider has picked up is resulting in unauthorized errors from the API when trying to refresh state at the beginning of a terraform plan operation:

data.tfe_project.<name>: Reading...
tfe_workspace.workspaces_setup: Refreshing state... [id=ws-1234567890abcdef]

Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: Error retrieving projects: unauthorized
│ 
│   with data.tfe_project.<name>,
│   on main.tf line 1, in data "tfe_project" "<name>":
│    1: data "tfe_project" "<name>" {
│ 
╵
╷
│ Error: Error reading configuration of workspace ws-1234567890abcdef: unauthorized
│ 
│   with tfe_workspace.workspaces_setup,
│   on main.tf line 21, in resource "tfe_workspace" "workspaces_setup":
│   21: resource "tfe_workspace" "workspaces_setup" {
│ 
╵

If I pass the exact same token to the provider by placing a TFE_TOKEN variable, of type env, into the HCP Terraform workspace variables, then the provider uses the correct token and the terraform plan run succeeds.

Additional Context

As mentioned above, this code (including the ability to pass a token explicity in the provider config) works successfully before using HCP Terraform via cloud { } configuration.

I believe either of two paths forward would work for resolving this issue:

• Restore the documented behavior of the tfe provider, wherein it should accept a token via explicit definition, including by variable passing into that explicit config, or
• If this is designed/expected behavior, update the docs to reflect that during Remote execution, only the TFE_TOKEN environment variable option is honored.