Kubeconfig does not apply

[matterai]

Hi!
There is a Github Action with such step:

      - id: get-credentials
        uses: google-github-actions/get-gke-credentials@v2
        with:
          cluster_name: ${{ vars.GKE_CLUSTER_NAME }}
          location: ${{ vars.GOOGLE_LOCATION }}-a

According the doc it setting $KUBECONFIG env variable.
Then I use pulumi config to set up variables and pulumi up to create a namespace in my GKE cluster:

      - id: set_pulumi_config
        name: set pulumi config
        working-directory: ./project/pulumi
        run: |
          pulumi stack select --create gke.monitoring
          pulumi config set gcp:project ${{ vars.GOOGLE_PROJECT_ID }}
          pulumi config set gcp:region ${{ vars.GOOGLE_LOCATION }}
          pulumi config set project:pulumiOrg ${{ vars.PULUMI_ORG }}
          pulumi config set project:githubSha ${{ github.sha }}
          pulumi config set gke_monitoring:grafanaAdminPassword --secret ${{ secrets.GRAFANA_ADMIN_PASSWORD }}
          pulumi config set gke_monitoring:grafanaDomain ${{ vars.GRAFANA_DOMAIN }}
          pulumi config set gke_monitoring:githubOrg ${{ github.repository_owner }}
          pulumi config set gke_monitoring:githubOauthClientId ${{ vars.GH_OAUTH_CLIENT_ID }}
          pulumi config set gke_monitoring:githubOauthSecret --secret ${{ secrets.GH_OAUTH_SECRET }}
        env:
          PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}

      - id: pulumi_up
        name: pulumi up
        uses: pulumi/actions@v4
        with:
          command: up
          stack-name: gke.monitoring
          work-dir: ./project/pulumi
          refresh: true
          upsert: true
        env:
          PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}

Pulumi stack running on pulumi up:

export class GkeMonitoringStack extends pulumi.ComponentResource {
  public static readonly stackName = 'gke.monitoring';

  constructor(
    name: string,
    args: {
      gcpProjectId: string;
      gcpRegion: string;
      grafanaPassword: string;
      grafanaDomain: string;
      githubOrg: string;
      githubOauthClientId: string;
      githubOauthSecret: string;
      devDomain: pulumi.Output<any>;
      githubSha: string;
    },
    opts?: pulumi.ComponentResourceOptions,
  ) {
    super(
      'stacks:GkeMonitoringStack',
      GkeMonitoringStack.stackName,
      args,
      opts,
    );

    const grafanaIp = new gcp.compute.GlobalAddress(`${name}-grafana-ip`, {
      name: `${name}-grafana-ip`,
    });

    const provider = new k8s.Provider(`${name}-monitoring-provider`, {});

    const namespace = new k8s.core.v1.Namespace(
      `${name}-monitoring-namespace`,
      { metadata: { name: `monitoring` } },
      { dependsOn: [provider], provider: provider },
    );
...

According the doc it can source config from KUBECONFIG.
But when I run my action, I got an error:

    kubernetes:core/v1:Namespace (monitoring-namespace):
      warning: configured Kubernetes cluster is unreachable: failed to parse kubeconfig data in `kubernetes:config:kubeconfig`- couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct *** APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" ***
      error: failed to read resource state due to unreachable cluster. If the cluster was deleted, you can remove this resource from Pulumi state by rerunning the operation with the PULUMI_K8S_DELETE_UNREACHABLE environment variable set to "true"

What am I doing wrong?

I also tried to configure kubernetes:config:kubeconfig and pass kubeconfig JSON to it directly, but nothing helps.

[lunaris]

Hi @matterai, I'm not 100% sure of all the interactions going on here, but at first glance I'm wondering if setting KUBECONFIG in the env of the Pulumi action works for you? E.g.:

      - id: pulumi_up
        name: pulumi up
        uses: pulumi/actions@v4
        with:
          command: up
          stack-name: gke.monitoring
          work-dir: ./project/pulumi
          refresh: true
          upsert: true
        env:
          KUBECONFIG: ${{ env.KUBECONFIG }}
          PULUMI_ACCESS_TOKEN: ${{ secrets.PULUMI_ACCESS_TOKEN }}

[matterai]

No, setting $KUBECONFIG doesn’t work. It looks like the Provider class doesn’t recognize what’s passed in the variable when it’s created.

I’m really not sure what exactly the Provider is doing when initializing an object. When I explicitly pass it a YAML kubeconfig generated on my local machine using the command below, I get an error:

gcloud container clusters get-credentials NAME \
  --zone $ZONE \
  --project $PROJECT_ID

The error makes it seem like Provider expects the kubeconfig in JSON format:

kubernetes:core/v1:Namespace (monitoring-namespace):
  warning: configured Kubernetes cluster is unreachable: failed to parse kubeconfig data in `kubernetes:config:kubeconfig` - couldn't get version/kind; json parse error: json: cannot unmarshal string into Go value of type struct { APIVersion string "json:\"apiVersion,omitempty\""; Kind string "json:\"kind,omitempty\"" }
  error: failed to read resource state due to unreachable cluster. If the cluster was deleted, you can remove this resource from Pulumi state by rerunning the operation with the PULUMI_K8S_DELETE_UNREACHABLE environment variable set to "true"

So, I tried running kubectl config view --flatten -o json, saved the result to kubeconfig.json, and explicitly passed it to the Provider constructor. But when I run the script, I still get an error:

   ~  kubernetes:core/v1:Namespace monitoring-namespace refreshing (0s) error: failed to read resource state due to unreachable cluster. If the cluster was deleted, you can remove this resource from Pulumi state by rerunning the operation with the PULUMI_K8S_DELETE_UNREACHABLE environment variable set to "true"

At this point, I have no idea how this is supposed to work.

[rquitales]

@matterai would you be able to let us know the following:

• What version of Pulumi Kubernetes provider you're using here
• Are you able to deploy your Pulumi stack/program locally outside of GHA? This will help us identify if the issue only occurs in GHA, or a bug in the provider.

Note: the Kubernetes provider is able to parse kubeconfigs formatted as YAML. You do not need to convert to json. The reason for json parse error: json: occurring in the error message is that we use upstream Kubernetes library packages to help parse kubeconfig values. The upstream implementation of doing this is to convert yaml into json first before deserializing it into Go structs.

[matterai]

Hey! I am sorry for a delay.

• pulumi version is v3.149.0.
• No, I got that same issue on my local MacOS when I pass /.kube/config.